Azure data engineer

Job Title - DataBricks Data Engineer

Location - Pittsburg, PA

**Must have 8+ years of real hands on experience**

We are specifically seeking a Data Engineer-Lead with strong expertise in Databricks development.

The role involves:

• Building and testing data pipelines using Python/Scala on Databricks
• Hands on experience to develop and lead the offshore team to perform development/testing work in Azure data bricks
• Architect data platforms using Azure services such as Azure Data Factory (ADF), Azure Databricks (ADB), Azure SQL Database, and PySpark.
• Collaborate with stakeholders to understand business needs and translate them into technical solutions.
• Provide technical leadership and guidance to the data engineering team and need to perform development.
• Familiar with Safe Agile concepts and good to have working experience in agile model.
• Develop and maintain data pipelines for efficient data movement and transformation.
• Onsite and offshore team communication and co-ordination.
• Create and update the documentation to facilitate cross-training and troubleshooting
• Hands on experience in scheduling tools like BMC control-M and setup jobs and test the schedules.
• Understand the data models and schemas to support the development work and help in creation of tables in databricks
• Proficiency in Azure Data Factory (ADF), Azure Databricks (ADB), SQL, NoSQL, PySpark, Power BI and other Azure data tools.
• Implementing automated data validation frameworks such as Great Expectations or Deequ
• Reconciling large-scale datasets
• Ensuring data reliability across both batch and streaming processes

The ideal candidate will have hands-on experience with:

• PySpark, Scala, Delta Lake, and Unity Catalog
• Devops CI/CD automation
• Cloud-native data services
• Azure databricks/Oracle
• BMC Control-M

Location: Pittsburgh, PA