This job offer is no longer available
About
Job Description • Design and build scalable, cloud-native data platforms using PySpark, Python, and modern data engineering practices. • Mentor and guide other engineers, sharing knowledge, reviewing code, and fostering a culture of curiosity, growth, and continuous improvement. • Create robust, maintainable ETL/ELT pipelines that integrate with diverse systems and serve business-critical use cases. • Lead by example—write high-quality, testable code and participate in architecture and design discussions with a long-term view in mind. • Decompose complex problems into modular, efficient, and scalable components that align with platform and product goals. • Champion best practices in data engineering, including testing, version control, documentation, and performance tuning. • Drive collaboration across teams, working closely with product managers, data scientists, and other engineers to deliver high-impact solutions. • Support data governance and quality efforts, ensuring data lineage, cataloging, and access management are built into the platform. • Continuously learn and apply new technologies, frameworks, and tools to improve team productivity and platform reliability. • Own and optimize cloud infrastructure components related to data engineering workflows, storage, processing, and orchestration. • Participate in architectural discussions, iteration planning, and feature sizing meetings • Adhere to Agile processes and participate actively in agile ceremonies • Stakeholder management skills
All About You Solid understanding of EMR Serverless, Apache Airflow, DAGs, Iceberg S3, SQS, DLQ, Lambda, AWS Lake Formation, Glue, Athena, Data Pipelines, Databricks Unity catalog, DBX S3
Any experience in YAML Data Contracts
Experience with containerization and infrastructure automation (e.g., Docker, Terraform) will be a good addition.
Experience integrating heterogeneous systems and building resilient data pipelines across cloud environments.
Familiarity with orchestration tools (e.g., Airflow, dbt, Step Functions, etc.).
Exposure to data governance tools and practices (e.g., Lake Formation, Purview, or Atlan).
AWS Certified Data Analytics, Azure Data Engineer), or demonstrable contributions to open source/data engineering communities will be a bonus. Exposure to machine learning data pipelines or MLOps is a plus.
Languages
- English
Notice for Users
This job was posted by one of our partners. You can view the original job source here.