Cette offre d'emploi n'est plus disponible
À propos
St. Louis, MO
Title:
Data Engineer
Experience:
10+ Years (Software/Data Engineering)
Job Description
Core Responsibilities
Drive adoption and implementation of tools and platforms that support both internal and external customers. Serve as a key contributor across the full data/software/platform development lifecycle:
design, development, documentation, testing, deployment, and support . Operate efficiently within highly secure environments adhering to
PII
and
PCI-DSS
standards. Support architects and engineers in designing and building scalable, secure, and agile data applications and platforms. Design, develop, and optimize
batch and real-time data pipelines
using
Medallion Architecture , preferably on
Snowflake
or
Databricks . Build modular, test-driven data transformation workflows using
dbt , following strict
TDD
practices. Implement CI/CD pipelines via
GitLab
and
Jenkins
for automated testing, deployment, and monitoring. Embed
DataOps principles
across all phases of the pipeline lifecycle-testing, monitoring, versioning, collaboration, and automation. Create scalable, reusable
data models
to support analytics and reporting, including
Power BI
dashboards. Develop, optimize, and support
real-time streaming pipelines
using technologies such as
Kafka
and
Spark Structured Streaming . Establish data observability frameworks for monitoring
data quality, freshness, lineage, and anomaly detection . Lead deployments, migrations, and upgrades for data platforms ensuring minimal downtime and strong mitigation planning. Collaborate with cross-functional stakeholders to translate requirements into reliable, high-impact data solutions. Maintain comprehensive documentation covering pipeline architecture, processes, standards, and operating procedures. Troubleshoot complex data and system issues using advanced analytical and problem-solving skills. Communicate clearly and effectively with both technical and non-technical stakeholders. Required Qualifications
Bachelor's degree in
Computer Science ,
Engineering , or related technical discipline. 15+ years
of proven experience in software development, data engineering, or platform engineering. Deep hands-on expertise with
Databricks ,
Python ,
PySpark , and
Spark SQL . Strong experience building high-performance transformations involving joins, window functions, aggregations, partitioning, and caching strategies. Skilled in developing and managing
real-time streaming pipelines . Experience with
Delta Live Tables (DLT)
and
Databricks Workflows ; knowledge of
Lakeflow Declarative Pipelines
is a plus. Strong DevOps background including:
Databricks Asset Bundles (DABs) Git/GitHub/Bitbucket Jenkins automation
Proven experience with
dbt
for modular, testable, and scalable transformation workflows. Solid understanding of
cloud database ecosystems
(AWS, Azure, or GCP). Expertise in designing scalable data models and dashboards using
Power BI . Advanced SQL development and query optimization skills. Demonstrated capability in building and managing
data observability
frameworks. Strong track record in planning and executing large-scale
deployments, upgrades, and migrations
with minimal operational impact.
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre a été publiée par l’un de nos partenaires. Vous pouvez consulter l’offre originale ici.