This job offer is no longer available
About
This is a remote position. We are looking for skilled Data Engineers to join our team. As a Data Engineer, you will be responsible for understanding business and technological challenges, develop data pipelines tackling those challenges, and ensuring their smooth deployment. You will also be responsible for the applications of standard industry and within-the-company good practices, and the application and evolution of our various patterns. Key Responsibilities
Design, build, and maintain scalable batch pipelines in Databricks using PySpark. Develop and support real-time data streams. Implement robust data transformations following the Medallion Architecture (Bronze, Silver, Gold). Optimize and troubleshoot performance of large-scale data jobs. Platform & Infrastructure
Manage and deploy infrastructure using Terraform and GitOps principles. Work within our Airflow orchestrator running on Azure Kubernetes Service (AKS). Apply best practices for CI/CD, monitoring, and error handling in production. Collaboration & Communication
Work closely with Data Architects to understand and implement proposed architectures. Coordinate with Project Managers and stakeholders to align on requirements and milestones. Participate in code reviews and share knowledge across the engineering team. Testing & Documentation
Define and perform tests to ensure data quality and pipeline reliability. Document technical specifications, workflows, and deployment processes. Requirements
Proven experience with Databricks and PySpark. Strong understanding of batch and streaming data processing. Experience with Azure Data Lake Storage Gen2 (ADLS). Solid experience with Airflow, preferably deployed in Kubernetes (AKS). Understanding of Medallion Architecture. Familiarity with data privacy, governance, and security standards. Working knowledge of Terraform and infrastructure-as-code. If it sounds like you, share your CV with us and let's talk!
Languages
- English
Notice for Users
This job was posted by one of our partners. You can view the original job source here.