Data Engineer

We need Azure Data Factory and Databricks Python notebook developers. They need to have practical experience as data engineers, not simply being data analysts or dbas or similar. They need to be able to discuss (Synapse) pipelines creation, scheduling, and configuration. As well as execution of python notebooks in a data lake/delta lake architecture.

The client is seeking a
Data Engineer
. This role will report to the Enterprise Data Architect. If you want to help public agencies, non-profits, and companies get their data right, we look forward to hearing from you.

• What you will do:

The Data Engineer must have considerable experience in providing highly specialized applications and operational analysis, supporting network and computing infrastructure and has knowledge in networking technologies. In this role you will provide data engineering expertise to our team as we develop a custom solution for our client, a large healthcare organization. You will help with the business development activities and will be responsible for describing data engineering solutions in our proposals.

• This role will be responsible for designing and maintaining data pipelines, data warehouses, and data integration solutions
• You will be responsible for providing data integration and analysis services working across several technologies and other disciplines including data modeling and data science, working with data lake-based solutions
• You will be responsible for testing your code and your team's code with manual and automated test scripts
• All code must be managed in GitHub repositories for effective version and deployment control
• You will be responsible for monitoring and maintaining code execution, data quality, and supporting data defects as they arise
• Able to lead an architecture or client call with other engineers explaining the architecture, development, and testing approach that you are implementing
• Provide data engineering support by mapping data, including Electronic Health Record data, between source systems and target data models
• Use your experience in healthcare to understand and transform data appropriately including evaluating quality or formatting issues
• You will work closely with cross-functional teams to ensure data quality, optimize performance, and implement scalable data infrastructure

Required Education:

• Bachelor's Degree in a field such as Computer Science, Statistics, Mathematics, Database Engineering, or Management Information Systems

Required Experience:

• 5+ years of recent professional experience in database architecture, database engineering, data analysis, data mining, and/or data science

Required Skills:

• You will develop ETL/ELT pipelines that centralize data into a cloud-based infrastructure using Azure Data Factory (ADF), Azure Databricks using Python notebooks, and SQL
• All code must be managed in Github repositories for effective version and deployment control
• Experience in creating, optimizing, and running Transact-SQL queries in Microsoft SQL Server
• Experience in creating, optimizing, and running Azure SQL Dedicated Pools
• Experience working with Apache Parquet and/or Delta Lake formatted data
• Experience with Synapse, reviewing GitHub scripts, JSON, Spark Notebooks, and Python
• Experience in data migration to include data mapping and data profiling
• Experience with ETL and ETL pipelines
• Experience integrating data and data marts for consumption by visualization and predictive AI/ML tools
• Ability to communicate concisely and persuasively with software engineers and clients
• Ability to work with the federal government and be able to obtain a Public Trust clearance

Preferred:

• Experience provisioning, using, and optimizing Microsoft Azure and Cloud
• Experience with data modelling using data modelling tools such as Erwin
• Experience working for the Department of Veterans Affairs
• Experience large-scale data analysis systems, such as Databricks, Hadoop, Pig, Scala, Spark or MPP databases
• Experience working with healthcare data, preferably Electronic Health Record data
• An in-depth understanding of the terminologies, code sets, and standards of healthcare data
• Experience supporting MLOps pipelines
• Experience with CI/CD