This job offer is no longer available
Sr. Data Engineer
Bubble Insurance (joined The Baldwin Group)
- United States
- United States
About
Must be authorized to work in the U.S. without the need for employment-based visa sponsorship now or in the future. We are unable to sponsor applicants for U.S. work visa status for this opportunity (no sponsorship is available for H-1B, L-1, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT or any other employment-based visa). About the Role We are seeking a highly skilled Senior Data Engineer to join our Data Engineering & Platform organization. This role is ideal for someone passionate about building scalable data solutions, enabling cross-functional data delivery, and raising engineering excellence across the organization. You will work on an advanced Databricks-on-AWS data platform and play a key role in transforming our data ecosystemdesigning and building resilient pipelines, implementing CI/CD, and contributing to platform modernization efforts. As a senior member of the team, you will partner with data engineers, analytics teams, product teams, and platform engineering to build high-quality, production-grade data solutions that power enterprise analytics, data science, and operational workloads. Key Responsibilities: Data Engineering & Pipeline Development Design, build, maintain, and optimize scalable data pipelines and ELT/ETL processes to ingest, process, and store large volumes of data from various sources. Architect robust batch and streaming solutions using PySpark, Spark SQL, and Databricks Jobs. Monitor and troubleshoot data pipelines and infrastructure to ensure high availability and performance. Ensure reliability, observability, and SLAs through monitoring, alerting, and automated recovery patterns. Implement and maintain data integration solutions, including APIs and data connectors, to facilitate data exchange between systems. Cloud Platform Engineering Build and manage data solutions on AWS, leveraging services such as S3, Lambda, Glue, IAM, EC2, EKS, Step Functions, and EventBridge. Apply cloud engineering best practices for cost optimization, reliability, and security. CI/CD, DevOps, and Infrastructure-as-Code Implement and maintain CI/CD pipelines using: Databricks Asset Bundles Terraform (IaC) GitHub Actions / Azure DevOps / Jenkins (depending on internal stack) Own deployment automation and environment management for data workloads and platform components. Contribute to reusable Terraform modules and engineering standards. Data Governance and Data Quality Maintain data quality through validation frameworks (e.g., DQX, Great Expectations, Databricks expectations, custom frameworks). Ensure data quality, integrity, and security by implementing best practices for data governance and management. Implement governance and lineage through Unity Catalog, cataloging standards, and metadata best practices. Platform Modernization Contribute to initiatives such as: Migration of pipelines from legacy integration tools to Lakeflow / Lakehouse-native ingestion Improving infrastructure automation Enhancing data observability, testing, and deployment workflows Collaboration & Technical Leadership Serve as a technical mentor to junior engineers. Partner closely with data analysts, data scientists, and business stakeholders to deliver enterprise data assets. Contribute to architecture decisions, design reviews, and platform roadmap conversations. Required Qualifications 7+ years of data engineering experience building production-grade data pipelines. Deep expertise in Databricks, Spark, and Delta Lake. Hands-on experience with AWS cloud services. Strong proficiency in Python, SQL, and distributed computing patterns. Experience with Terraform and cloud-native CI/CD. Strong understanding of data modeling, orchestration, and data quality frameworks. Ability to operate autonomously while collaborating in a team environment. Additional Qualifications Experience with Lakeflow, Databricks Workflows, or ingestion modernization initiatives. Knowledge of software engineering best practices: test automation versioning code reviews Git workflows Familiarity with event-driven architecture and streaming technologies. Experience supporting Data Science and ML platform teams. P&C Insurance domain knowledge preferred
Languages
- English
Notice for Users
This job was posted by one of our partners. You can view the original job source here.