Cette offre d'emploi n'est plus disponible
À propos
We are seeking an experienced Data Engineer to collaborate on modernizing our 12-year-old on-premises SQL Server data warehouse by migrating it to Databricks Lakehouse architecture. This includes evaluating & redesigning our ETL processes, establishing sustainable data pipelines, and transitioning our existing Power BI reports (paginated reports and dashboards) to use the new Databricks environment as the primary data source.
Our current setup relies on a legacy SQL Server environment feeding a collection of Power BI reports. We have begun ingesting new data sources into Databricks and have a few migrated sources running in parallel across both platforms during the transition phase. The goal is to fully shift to Databricks for better scalability, flexibility, performance, and maintainability while ensuring data integrity and minimal disruption to business reporting.
Project Goals & Key Deliverables:
• Adopt a modern, flexible platform using Medallion architecture (Bronze → Silver → Gold layers) for progressive data refinement.
• Evaluate both current environments (SQL Server and existing Databricks setup) and design an optimal, sustainable ETL/ELT structure with clear guidelines and best practices.
• Define standardized processes and guidelines for bronze-to-gold transformations, including data cleansing, enrichment, validation, and business logic application.
• Migrate appropriate data ingestion jobs and pipelines to Databricks
• Rebuild/refactor Power BI reports and datasets to connect directly to Databricks Gold-layer objects (e.g., via Databricks SQL endpoints, DirectQuery, or optimized semantic models) while preserving or improving report functionality.
• Implement robust data quality controls throughout the pipelines to prevent incomplete, inaccurate, or inconsistent data from reaching reports or downstream consumers.
• Design a structured process and framework for identifying, resolving, and preventing data quality issues (including monitoring, alerting, and remediation workflows).
This is a hands-on, collaborative project with our internal team. You'll provide architectural recommendations, implement pipelines (using Databricks tools), validate migrations, and support the Power BI transition.
Expected Outcomes:
• A fully modernized, governed data platform in Databricks that replaces our legacy SQL Server dependencies.
• Reliable, high-quality Gold-layer assets serving as the single source of truth for reporting.
• Reduced technical debt, improved data freshness, and easier future scalability.
If you've led similar SQL Server → Databricks migrations with Medallion architecture and Power BI integration, and can deliver structured, maintainable solutions, please apply with examples of your relevant work
Contract duration of more than 6 months. with 40 hours per week.
Mandatory skills: ETL Pipeline, Data Warehousing, Data Modeling, Microsoft SQL Server, Databricks Platform, Data Engineering
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre a été publiée par l’un de nos partenaires. Vous pouvez consulter l’offre originale ici.