Data Engineer

BGTS LTD

London, England, United Kingdom

London, England, United Kingdom

Find similar jobs

About

As a Data Engineer, you will be responsible for: Data Engineering & Development * Design, build, and maintain high-quality, scalable, and tested data pipelines. * Develop and manage Databricks structured streaming pipelines. * Build and optimize event-driven and real-time data processing solutions. * Implement and maintain Unity Catalog-based Lakehouse architecture. * Develop analytics-ready datasets to support business insights and reporting. Platform & Automation * Build and manage CI/CD pipelines using Azure DevOps. * Identify and implement automation opportunities across workflows. * Ensure reliable and stable data platform operations. * Apply governance, security, and documentation standards. Data Quality & Reliability * Establish the Data Lakehouse as a trusted and reliable source of truth. * Monitor, troubleshoot, and resolve data incidents. * Support business users and technical teams with data-related queries. * Continuously improve platform performance and reliability. Collaboration & Support * Work closely with data science, analytics, platform, and business teams. * Champion data engineering best practices. * Provide technical guidance and mentorship where required. * Contribute to a culture of learning, quality, and continuous improvement. Essential Skills * Strong experience with Azure Databricks and cloud data platforms. * Advanced proficiency in Python, PySpark, and SQL. * Experience developing Spark/Databricks pipelines. * Hands-on experience with structured streaming and event-driven systems. * Strong understanding of Lakehouse architecture and best practices. * Experience with Unity Catalog. * Expertise in Azure DevOps and CI/CD pipelines. * Knowledge of data modelling (dimensional/star schemas). * Experience working in Agile environments. Desirable Skills * Exposure to multiple data technology stacks. * Experience in large-scale enterprise environments. * Knowledge of security, governance, and compliance frameworks

London, England, United Kingdom

Languages

English

Notice for Users

This job was posted by one of our partners. You can view the original job source here.

Find similar jobs