Data Engineer – Databricks SpecialistAutomobile Association • London, England, United Kingdom
Data Engineer – Databricks Specialist
Automobile Association
- London, England, United Kingdom
- London, England, United Kingdom
Über
At The AA, our purpose is to create confidence for drivers now and for the future. Data plays a critical role in delivering that purpose, and we are investing heavily in a modern, Databricks-centric data platform to unlock the full value of our connected car and insurance data. This role is for an experienced Databricks Data Engineer. You will be working day-to-day designing, building and operating production-grade Databricks Lakehouse solutions, including structured streaming pipelines, Unity Catalog governance, and Spark-based data engineering at scale. This is not a role for someone who has “touched” Databricks occasionally or supported it at the edges. We are looking for someone who has built, owned and evolved Databricks solutions, and who can confidently articulate design decisions, trade-offs, and best practice. If Databricks is central to how you work and can demonstrate this natural, we want to hear from you. What will I be doing?
Designing, building and operating
production-grade Databricks Lakehouse solutions , including
structured streaming pipelines
using
Python and PySpark Owning and evolving
Unity Catalog–based governance , ensuring secure, discoverable and well-managed data assets Developing and maintaining
event-driven data pipelines , integrating closely with backend engineering teams Implementing and supporting
CI/CD pipelines in Azure DevOps
to enable reliable, automated Databricks deployments Creating high-quality, analytics-ready datasets that deliver
actionable business insight at scale Proactively improving
performance, reliability, automation and observability
across the Databricks data platform What do I need?
We are intentionally setting a high bar for Databricks experience. You should be able to demonstrate deep, hands-on capability, not just theoretical knowledge. Significant, hands-on experience with Azure Databricks in a production environment Proven experience building Spark / Databricks pipelines using Python and PySpark Strong experience with structured streaming and event-driven architectures Practical experience implementing and operating Unity Catalog Solid understanding of Lakehouse design principles, including dimensional and analytical modelling Experience building and maintaining CI/CD pipelines, ideally using Azure DevOps Confidence working with large-scale data, performance tuning, and troubleshooting complex pipelines What we mean by “Databricks experience” You have: Designed and built Databricks pipelines end-to-end Made architectural decisions within Databricks environments Worked with Spark internals, optimisation techniques and cluster configuration Operated Databricks solutions in live, business-critical contexts If your Databricks exposure has been limited to minor contributions, proof-of-concepts, or occasional usage alongside other tools, this role is unlikely to be the right fit. Additional information
We’re always looking to recognise and reward our employees for the work they do. As a valued member of The AA team, you’ll have access to a range of benefits including: 25 days annual leave plus bank holidays + holiday buying scheme Worksave pension scheme with up to 7% employer contribution Free AA breakdown membership from Day 1 plus 50% discount for family and friends Discounts on AA products including car and home insurance Employee discount scheme that gives you access to a car salary sacrifice scheme plus great discounts on healthcare, shopping, holidays and more Company funded life assurance Diverse learning and development opportunities to support you to progress in your career Dedicated Employee Assistance Programme and a 24/7 remote GP service for you and your family Plus, so much more! We’re an equal opportunities employer and welcome applications from everyone. The AA values diversity and the difference this brings to our culture and our customers. We actively seek people from diverse backgrounds to join us and become part of an inclusive company where you can be yourself, be empowered to be your best and feel like you truly belong. We have five communities to bring together people with shared characteristics and backgrounds and drive positive change.
#J-18808-Ljbffr
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klicken Sie auf „Jetzt Bewerben“, um Ihre Bewerbung direkt auf deren Website einzureichen.