Data Engineer

Job Summary:

We are seeking an experienced Data Engineer to collaborate with business stakeholders and technical teams to acquire and migrate data sources critical to business objectives. This role involves designing, developing, and managing robust ETL pipelines, working with both structured and unstructured data, and ensuring efficient data management across various platforms. The candidate will also be expected to build cross-platform data strategies, automate processes, and optimize data delivery.

Key Responsibilities:

Data Acquisition and Migration:

Collaborate with business stakeholders and technical teams to acquire and migrate relevant data sources for business needs and goals.

Database Expertise:

Demonstrate deep technical knowledge of relational and non-relational databases, Data Warehouses, and Data Lakes, including structured and unstructured storage options.

ETL Pipeline Development:

Develop data flow pipelines to extract, transform, and load (ETL) data from various sources. This includes custom ETL pipelines that enable model and product development.

Data Extraction:

Write custom scripts to extract data from unstructured or semi-structured data sources.

Documentation and Communication:

Provide clear documentation for all delivered solutions and processes, and ensure integration with relevant corporate stakeholders.

Process Improvement:

Identify and implement internal process improvements for data management, including automating manual processes, optimizing data delivery, and re-designing infrastructure for better scalability.

Adoption of New Tools:

Stay current with emerging tools and applications to ensure the development of high-quality and efficient solutions.

Cross-Platform Data Strategy:

Build cross-platform data strategies to aggregate multiple data sources and streamline development datasets.

Required Qualifications:

Education: Bachelor's degree in Computer Engineering, Computer Science, or a related discipline (Master's Degree preferred). Experience:

7+ years of experience in ETL design, development, and performance tuning using tools such as SSIS or Azure Data Factory (ADF) in a multi-dimensional Data Warehousing environment. 3+ years of experience with Python or SQL to set up and operate data pipelines. 7+ years of advanced SQL programming experience (PL/SQL, T-SQL). 5+ years of experience in Enterprise Data & Analytics solution architecture. 3+ years of extensive hands-on experience with Azure, especially in data-heavy/analytics applications involving relational and NoSQL databases, Data Warehouses, and Big Data technologies. 3+ years of experience with Azure Data Factory, Azure Synapse Analytics, Azure Analysis Services, Azure Databricks, Blob Storage, Azure SQL DW/Synapse, and Azure Functions. 2+ years of experience in defining and enabling data quality standards for auditing and monitoring. Strong analytical abilities and intellectual curiosity. In-depth knowledge of relational database design, data warehousing, and dimensional data modeling concepts. Deep understanding of REST and API design principles. Excellent collaboration, communication, and teamwork skills. Self-starter with the ability to thrive in a fast-paced development environment. Agile experience is highly desirable. Proficiency in the development environment, including IDEs, database servers, GIT, Continuous Integration, unit testing, and defect management tools. Strong experience with Python, Spark, and Pyspark. Strong leadership capabilities. Preferred Qualifications:

2+ years of experience with Big Data Management (BDM) for relational and non-relational data formats such as JSON, XML, Avro, Parquet, and Copybook. Knowledge of DevOps processes (CI/CD) and infrastructure as code. Experience with Master Data Management (MDM) and Data Quality tools. Experience developing REST APIs. Familiarity with Kafka. Knowledge of key machine learning concepts and MLOps.

Technologies We Use:

Databricks, Azure SQL DW/Synapse, Azure Tabular, Azure Data Factory, Azure Functions, Azure Containers, Docker, DevOps, Python, PySpark, Scripting (PowerShell, Bash), Git, Terraform, Power BI, Snowflake.

Education:

Bachelors Degree