Über
Duration: 6Months Contract to Hire
Data PlatformInfrastructure & DevOps
Administer, optimize, and scale our Databricks Lakehouseenvironment, ensuring high performance, cost efficiency, and operationalexcellence. Develop, maintain, and enhance our data platforminfrastructure and security configurations using
Terraform
. Thisincludes provisioning Databricks workspaces, SQL Endpoints, Unity Catalogobjects, and network components. Manage and enforce
Unity Catalog
for data governance, access control, andmetadata management. Implement and manage
CI/CDpipelines
for data pipelines, dbt projects, andinfrastructure deployments using
GitHub Actions
. Automate operational tasks, monitoring, and alerting for thedata platform. Implement and enforce DevSecOps principles, working closelywith security teams to ensure compliance and manage/rotate credentialssecurely. Design and implement data ingestion patterns into Databricks using
Delta Lake, optimizing for large-scale data processing and storage. Develop, optimize, andtroubleshoot complex
Spark jobs (PySpark/Scala)
for data processingand transformation within Databricks. Manage and extend dataingestion pipelines using
Airbyte
(or similar modern tools like Fivetran, Stitch), including configuringconnectors, monitoring syncs, and ensuring data quality and reliabilityfrom diverse source systems (e.g., ERP, CRM, marketing, supply chain). Orchestrate and automate data pipelines and dbt models usingDatabricks Workflows and potentially integrating with other orchestrationtools. Collaborate with Analytics Engineers to translate businessrequirements into efficient and scalable data models using
dbt
(Data Build Tool). Implement dbt best practices for modularity, testing,documentation, and version control, ensuring seamless integration withDatabricks. Partner effectively with Analytics Engineers, DataScientists, and business stakeholders to deliver high-quality datasolutions. Provide technical guidance and mentorship to junior teammembers, and champion data engineering best practices, code quality, anddocumentation standards. Qualifications& Skills
Education:
Bachelor's degree in ComputerScience, Data Engineering, or a related technical field required. Experience:
5+ years of progressiveexperience as a Data Engineer, with a strong focus on cloud-based dataplatforms. Deep Databricks Expertise:
Extensive experience with Spark(PySpark/Scala), Delta Lake, Unity Catalog, Databricks SQL, and platformadministration. Data Modeling:
Proven experience with
dbt
for data modeling, transformation, and testing. Infrastructure as Code (IaC):
Strong proficiency with
Terraform
for defining,provisioning, and managing cloud infrastructure and Databricks resourcesas code. DevOps & CI/CD:
Expertise in Git and
GitHubActions
for version control and implementing robust CI/CD pipelines. Programming:
Proficiency in
SQL
and atleast one programming language (Pythonstrongly preferred, Scalais a plus). Data Architecture:
Solid understanding of datawarehousing, data lake, and lakehouse architectures.
#J-18808-Ljbffr
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klicken Sie auf „Jetzt Bewerben“, um Ihre Bewerbung direkt auf deren Website einzureichen.