Data Engineer

Data Engineer – Enterprise Lakehouse (Azure Databricks & ADF)

6-Month Contract-to-hire

100% Remote

40 hrs/week

Job Description

Our client operates a mission-critical Azure Databricks Lakehouse supporting clinical, financial, quality, safety, and enterprise analytics workloads. We are seeking a Data Engineer to join a small, high-impact core Lakehouse team focused on stability, reliability, and data quality for existing data products.

This role partners closely with a Senior Lakehouse Solution Architect / Platform Lead and collaborates with distributed domain data engineers across the organization. The initial emphasis is operational excellence—monitoring pipelines, improving ingestion robustness, automating validation, and reducing incidents—rather than onboarding large volumes of new data sources.

This position is well suited to a mid-level engineer with strong SQL and Python skills, foundational Azure Data Factory experience, and a willingness to learn and grow in a modern lakehouse environment.

Responsibilities

• Monitor and support daily ingestion pipelines across Azure Data Factory, Databricks Jobs, and dbt
• Investigate and resolve pipeline failures, data delays, and anomalies
• Improve pipeline reliability through:
• Standardized incremental and watermark logic
• Retry, idempotency, and recovery patterns
• Enhanced logging and error diagnostics
• Implement and maintain data quality and validation checks, including:
• dbt tests and SQL/Python validation logic
• Freshness, volume, schema, and anomaly detection
• Build and maintain pipeline health and data quality reporting views
• Support controlled backfills, reprocessing, and reloads of existing datasets
• Contribute to ingestion enhancements and limited new source onboarding under architectural guidance
• Participate in CI/CD workflows, including code reviews, testing, and promotion of Databricks and dbt assets
• Collaborate closely with the Lakehouse Architect and distributed engineering teams to improve platform reliability and standards

Required Qualifications

• 3–6 years of experience in data engineering or analytics engineering
• Strong SQL skills for transformation, analysis, and validation
• Solid working knowledge of Python for data processing and automation
• Experience with Azure Data Factory (pipelines, triggers, basic orchestration)
• Familiarity with Databricks or Spark-based platforms (notebooks, jobs, SQL)
• Experience using Git in a collaborative development environment
• Ability to learn new tools quickly and leverage AI-assisted tooling productively

Preferred Qualifications

• Experience with dbt-core (models, tests, environments)
• Exposure to healthcare or regulated data environments
• Familiarity with incremental ingestion and watermarking patterns
• Experience building monitoring, validation, or data quality reporting
• Interest in platform engineering, reliability, or data quality disciplines