About
Location Chicago, IL (Hybrid)
Job Summary We are looking for a Senior Data Engineer to take a central role in building and operating the data infrastructure that powers our analytics, business intelligence, and machine learning capabilities. This is a hands‑on engineering role for someone who takes genuine pride in well‑architected data systems—pipelines that are reliable, scalable, and built to last. You will work closely with business, product, and analytics stakeholders to understand complex requirements and translate them into elegant, production‑ready data solutions. The ideal candidate brings deep expertise in Azure Databricks and cloud‑native data platforms, a strong command of modern lakehouse architecture patterns, and the collaborative instincts to partner effectively across technical and non‑technical teams. This role reports to the Senior Manager of Data Engineering.
Key Responsibilities
Architect, develop, and maintain end‑to‑end data solutions on Azure Databricks, including scalable ETL and streaming pipelines built on Apache Spark, Delta Lake, and Azure Data Lake Storage (ADLS Gen2) to support reliable lakehouse architectures.
Design and optimize data models and schemas that serve a variety of consumers from analytics and reporting to operational data stores with a focus on performance and long‑term maintainability.
Implement and refine Delta Lake / Lakehouse patterns across Bronze, Silver, and Gold layers, including schema evolution strategies and time travel capabilities.
Write high‑quality PySpark and Spark SQL transformations, with a sharp eye for optimization across joins, partitioning, caching, and shuffle behavior.
Build and maintain data quality frameworks that include validation rules, monitoring coverage, and alerting mechanisms to catch issues before they reach downstream consumers.
Partner with data architects, analysts, data scientists, and product teams to ensure data engineering work is tightly aligned with business objectives and user needs.
Leverage cloud platforms: primarily Azure, with exposure to AWS or GCP to architect and optimize data storage solutions spanning warehouses, lakehouses, and real‑time processing systems.
Build out automation frameworks and CI/CD processes underpinned by version control, linting, automated testing, security scanning, and observability tooling.
Contribute meaningfully to data governance practices, helping ensure that data assets are accurate, accessible, well‑documented, and compliant with regulations including GDPR.
Mentor junior engineers, sharing expertise in software and data engineering best practices, agile delivery, and the craft of building systems that hold up at scale.
Diagnose and resolve complex platform and pipeline issues in Azure Databricks environments, maintaining high availability and optimal performance across the data infrastructure.
Requirements
Education: Bachelor's degree in Computer Science, Engineering, Data Science, or a related technical field.
Experience: At least 5 years of hands‑on data engineering experience, with a strong track record designing and building scalable data pipelines, ETL/ELT processes, and production‑grade data solutions.
Skills: Deep knowledge of Databricks architecture and its core components including Lakehouse, Delta Lake, Databricks SQL, Apache Spark Clusters, Unity Catalog, Workflows/Jobs, MLflow, and Notebooks; strong proficiency in Python, SQL, and Apache Spark; proven experience building reusable, metadata‑driven ingestion frameworks in Python and Scala; solid grounding in data modeling, schema design, and large‑scale performance tuning.
Cloud: Extensive experience with at least one major cloud data platform (Azure strongly preferred; AWS or GCP acceptable); hands‑on experience with modern data platform components including object storage, lakehouse engines, orchestration tools, columnar warehouses, and streaming services.
Engineering Practices: Deep familiarity with data engineering best practices including code repositories, CI/CD pipelines, test automation, monitoring, and alerting.
Other: Skilled at communicating data insights through tables, reports, dashboards, and visualizations; excellent interpersonal and communication skills with the ability to engage both technical and business audiences effectively.
Preferred Qualifications
Master's degree in Computer Science, Engineering, or a related discipline.
Hands‑on experience integrating Azure Databricks with Azure DevOps, ADLS Gen2, Azure Key Vault, and Azure Data Factory.
Familiarity with enterprise data modeling tools such as ERwin, including the ability to interpret and apply logical and physical models to analytical and lakehouse architectures.
Experience either migrating legacy platforms to modern data infrastructure or building greenfield data platforms from scratch.
Proficiency with Infrastructure as Code (IaC) and Governance as Code practices.
Exposure to machine learning workloads and experience collaborating on feature engineering efforts.
Experience delivering within an Agile development model.
#J-18808-Ljbffr
Languages
- English
Notice for Users
This job comes from a TieTalent partner platform. Click "Apply Now" to submit your application directly on their site.