Data Engineering Lead

Data Engineering Lead (Hands‑on | Azure Lakehouse | LLM Analytics)

Location: Plano, TX (Hybrid)

Reports to: Chief Architect

Team: Data Engineering (DE + Python Engineers)

This role will require you to be in office, will need you to be hands on with the skills mentioned below and we we are unable to sponsor a work visa. The total compensation range for this role is $145,000-$160,000. Please apply only if you meet these requirements.

About the Role

We're building a modern Azure lakehouse platform that powers a chat‑based natural language analytics interface for operational leaders. The goal: move beyond static dashboards to an experience where business leaders can ask complex questions in plain English, get trusted answers with provenance, and receive proactive alerts when trends go off track.

You will be the hands‑on technical leader who architects and builds this platform while mentoring a team of DE & Python engineers.

What You'll Do

● Own the data platform architecture (Azure Data Lake Gen2, Delta Lake, Lakehouse) and build production‑grade ELT/ETL pipelines (PySpark, SQL, Python).

● Implement a semantic layer/metrics store to enable natural language → SQL/metric translation and consistent KPI definitions across the business.

● Design and operate real‑time and batch pipelines using ADF/Synapse/Databricks Workflows; implement medallion architecture, schema evolution, and data contracts.

● Build the retrieval layer for LLMs (embeddings, metadata, grounding context) using Azure OpenAI + Azure AI Search (or vectorized Delta tables) to support chat‑based analytics.

● Implement data quality, lineage, and observability (e.g., Great Expectations, Unity Catalog/Purview), plus cost governance (partitioning, Z‑order, compaction).

● Deliver automated anomaly detection and alerting (time‑series baselines, isolation forests, Azure ML pipelines, Event Grid/Functions).

● Partner with product/ops leaders to translate vague analytical questions into robust data models, metrics, and queries with clear SLAs.

● Lead, mentor, and uplevel a team of data & Python engineers; establish patterns, reviews, and documentation; own CI/CD and IaC (Bicep/Terraform).

● Drive security, privacy, and compliance by design (RBAC, least privilege, PII handling, encryption, auditability).

Must‑Have Qualifications

● 7–10+ years in data engineering; 2–4+ years leading small teams while staying hands‑on (50–70%).

● Expert in Azure Data Lake Gen2, Delta Lake, Unity Catalog (or Fabric equivalent), PySpark, SQL, and Python.

● Proven experience designing Lakehouse/medallion architectures, incremental loads, MERGE/UPSERT patterns, schema evolution.

● Strong command of Databricks (or Fabric Lakehouse), ADF/Synapse/Databricks Workflows, and monitoring/observability.

● Built or contributed to a semantic/metrics layer and query optimization for complex, multi‑join analytics.

● Practical experience with Azure OpenAI integrations, retrieval/RAG, embeddings, vector search, and grounding structured data.

● CI/CD for data (GitHub Actions/Azure DevOps), IaC (Bicep/Terraform), testing frameworks for pipelines, data contracts.

● Excellent communication; able to translate business questions into data models and mentor engineers.

Nice‑to‑Have

● Azure ML pipelines; time‑series forecasting; root cause analysis frameworks.

● Great Expectations/Monte Carlo; OpenLineage; Purview; Fabric Semantic Models.

● Event‑driven patterns (Event Grid/Service Bus), streaming (Kafka/Event Hubs).

● Experience in operations/financial services/valuations domains.