Cette offre d'emploi n'est plus disponible
Senior Lead Data Engineer
Brillio
- St Louis, Missouri, United States
- St Louis, Missouri, United States
À propos
Brillio takes pride in its status as an employer of choice, consistently attracting the most exceptional and talented individuals due to its unwavering emphasis on contemporary, groundbreaking technologies, and exclusive digital projects. Brillio's relentless commitment to providing an exceptional experience to its Brillians and nurturing their full potential consistently garners them the Great Place to Work® certification year after year.
Senior Lead Data Engineer Salary: $125,000 - $130,000 a year
Key Responsibilities
Design, develop, and maintain ETL/ELT pipelines using PySpark and Python in Databricks (notebooks, jobs, Delta Lake tables, Unity Catalog for governance).
Implement medallion architecture (bronze/silver/gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability.
Write efficient SQL queries for data transformation, validation, and analytics within Databricks.
Provision and manage cloud infrastructure using Terraform (IaC) for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles/permissions, and related resources on Azure and/or AWS.
Implement and maintain CI/CD pipelines using Jenkins, GitHub (Actions/Repositories), and branching strategies for automated testing, deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations.
Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers.
Ensure data quality, lineage, security, and compliance (Delta Lake ACID transactions, schema evolution, time travel, and access controls).
Monitor pipeline performance, troubleshoot failures, and implement alerting/observability (Databricks tools, cloud monitoring services, or third‑party solutions).
Optimize cloud costs through auto‑scaling clusters, spot instances, job scheduling, and efficient resource usage.
Collaborate in agile teams, participate in code reviews, and contribute to best practices for data engineering.
Required Skills & Experience
Strong proficiency in Python and PySpark for distributed data processing and ETL.
Advanced SQL skills with experience in complex querying, window functions, and optimization.
Hands‑on experience with Databricks (clusters, notebooks, Delta Lake, Unity Catalog, Delta Live Tables, workflows/jobs).
Proficiency in Terraform for infrastructure provisioning and management (Databricks resources, cloud storage, IAM, networking).
Experience with GitHub for version control and collaboration (branching, pull requests, code reviews).
Solid knowledge of CI/CD practices and tools, particularly Jenkins (pipelines, plugins for Databricks/GitHub/Terraform).
Working experience on Azure (Data Lake, Data Factory, Synapse, Key Vault, etc.) and/or AWS (S3, Glue, EMR, IAM, Lambda, etc.).
Understanding of big data concepts, data modeling (star/snowflake, dimensional), and lakehouse principles.
Familiarity with performance tuning in Spark/Databricks environments.
Equal Employment Opportunity Declaration Brillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.
#J-18808-Ljbffr
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre a été publiée par l’un de nos partenaires. Vous pouvez consulter l’offre originale ici.