Lead Data Engineer

Duration: 5 months with strong likelihood of extension

Our client is looking for Lead Data Engineer to support a large Teradata-to-Databricks migration, including SQL code conversion, historical data loading, and supporting Ab Initio job repointing/rewrite efforts.

RESPONSIBILITIES

• Convert tables, views, stored procedures, and functions from Teradata SQL to Databricks SQL.

• Load, test, and validate historical datasets using established migration patterns.

• Develop, optimize, and troubleshoot ETL/ELT pipelines in Databricks (SQL + PySpark).

• Support Premier teams as they repoint and rewrite Ab Initio jobs (no Abi expertise required; ability to read/navigate is a plus).

• Work directly with internal and client teams to deliver high-quality data engineering outcomes.

Lead Data Engineer

• Lead technical workstreams and small engagements.

• Architect scalable solutions and translate objectives into technical direction.

• Review work, guide junior engineers, and ensure delivery quality.

REQUIRED SKILLS

• Min 5 years as a Data Engineer

• Min 4 years in Databricks (SQL, PySpark)

• Experience with data migration, ETL/ELT pipelines, and data validation.

• Solution Architecture in Databricks migration (only for Lead Data Engineer)

• Ability to operate in a client-facing consulting environment.

PREFERRED SKILLS

• Teradata migration experience.

• Ability to read or navigate Ab Initio workflows.