Über
5+ years' experience in SQL (Expert) 4+ years of experience in Azure Databricks with PySpark 4+ years of experience in Azure Cloud platform 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL 2+ years of experience in Databricks workflow & Unity catalog 2+ years of experience in Python programming & package builds Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing. Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows). Ability to design modular, reusable workflows using tasks, triggers, and dependencies. Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers. Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies. Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures. Good to have Azure Entra/AD skills and GitHub Actions Good to have experience working on event-driven architectures using Kafka, Azure Event Hub Responsibilities:
Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms Implement and maintain ETL workflows for scalability cost effective and operational efficiency Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klicken Sie auf „Jetzt Bewerben“, um Ihre Bewerbung direkt auf deren Website einzureichen.