Airflow Data Engineer

Job Summary:

We are looking for a strong Data Engineer with expertise in Apache Airflow, specifically with professional experience using Astronomer. The ideal candidate will also have a solid background in Azure Cloud and experience working with DataBricks. This role will focus on managing data workflows, automating processes, and working in a cloud-based environment to support data-driven solutions.

Key Responsibilities: Data Workflow Automation: Design, build, and maintain data pipelines using Apache Airflow and Astronomer, ensuring efficient and scalable automation. Azure Cloud Integration: Leverage Azure Cloud services for cloud-based data processing, storage, and analytics solutions. Data Management: Work with DataBricks to manage and process large datasets, ensuring seamless integration with data lakes and other cloud-based data repositories. Collaboration: Work with data teams, software engineers, and other stakeholders to understand data requirements and design solutions that meet the organization's needs. Optimization: Optimize data workflows and processes for performance, scalability, and reliability within cloud environments. Troubleshooting and Monitoring: Monitor and troubleshoot data pipelines to ensure smooth operation and address any issues that arise. Required Qualifications:

Proficiency with Apache Airflow: Strong experience in managing and automating data workflows using Apache Airflow. Astronomer Expertise: Professional experience with Astronomer for managing Airflow deployments and orchestrating data pipelines. Azure Cloud: Experience working with Azure Cloud services, including storage, processing, and analytics. DataBricks: Solid experience working with DataBricks for big data processing and management. Strong knowledge of data integration, ETL processes, and data pipeline architecture. Excellent troubleshooting and problem-solving skills. Ability to collaborate with teams and stakeholders to design effective data solutions. Preferred Qualifications:

Experience with other cloud platforms (e.g., AWS, Google Cloud) could be beneficial. Familiarity with containerization technologies like Docker and Kubernetes. Exposure to Apache Spark or other big data frameworks is a plus. Background in data engineering best practices for performance optimization.

Certifications (if any):

Astronomer or Apache Airflow related certifications preferred.

Azure Data Engineer or similar cloud certifications are a plus.

Education:

Bachelors Degree