Data Engineer Databricks (Performance & Security)99x Europe • United States
Cette offre d'emploi n'est plus disponible
Data Engineer Databricks (Performance & Security)
99x Europe
- United States
- United States
À propos
This is a remote position. We are looking for a Data Engineer focused on Databricks platform excellence mastering features, security, performance optimization, and operational best practices. You will play a key role in improving how we leverage Databricks across our data platform, ensuring scalable, secure, and efficient workloads. Work can be done remotely and the communication with the team is in Portuguese and English. Key Responsibilities
Optimize Databricks performance across jobs, clusters, and data workloads. Implement and maintain governance, security, and permissions via Unity Catalog and Terraform. Contribute to Databricks feature enablement Delta Live Tables, Databricks Asset Bundles (DABs), and cost efficiency improvements. Monitor cluster and job performance using Grafana and internal observability tools. Support data engineers in troubleshooting and fine-tuning jobs and SQL queries. Explore and progressively implement streaming features (Structured Streaming, Autoloader, Kafka integration). Requirements
Minimum of 5 years of experience with similar responsibilities. Deep knowledge of Databricks configuration, optimization, and administration. Experience with PySpark, SQL, and data performance tuning. Familiarity with Unity Catalog, DABs, and Terraform integration. Strong understanding of data governance, cost optimization, and cluster policies. Basic to intermediate experience with Kafka or interest in learning streaming processing. Platform & Infrastructure
Manage Databricks environments integrated with Airflow on AKS, Azure ADLS Gen2, and Terraform. Collaborate with DataOps to ensure high availability, logging, and monitoring coverage. Implement performance and security benchmarks for production workloads. Our stack includes Talend, Azure ADLS Gen2, Azure Databricks, Airflow (AKS), Terraform, SQL, Python, PySpark, Grafana, Kafka (Aiven), GitLab, Databricks Asset Bundles (DABs), and DQX (Data Quality Framework). If it sounds like you, share your CV with us and let's talk!
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre a été publiée par l’un de nos partenaires. Vous pouvez consulter l’offre originale ici.