Cette offre d'emploi n'est plus disponible
Data Engineer (Python with Spark)
Diverse Lynx
- United States
- United States
À propos
10+ Years of experience.
Strong hands-on experience in Python programming for data engineering and data processing.
Extensive experience with Apache Spark (PySpark) for large-scale data processing and distributed computing.
Strong knowledge of SQL and experience working with relational and non-relational databases.
Experience in building and maintaining ETL/ELT pipelines for data ingestion and transformation.
Good understanding of data warehousing concepts, data modeling, and data architecture.
Experience working with big data technologies such as Hadoop ecosystem, Hive, or similar platforms.
Familiarity with cloud platforms (AWS, Azure, or GCP) and related data services.
Hands-on experience with data pipeline orchestration tools such as Airflow or similar.
Knowledge of data formats such as Parquet, Avro, JSON, and CSV.
Experience with performance tuning and optimization of Spark jobs and data pipelines.
Strong problem-solving skills and ability to work with cross-functional teams.
Responsibilities:
Design, develop, and maintain scalable data pipelines using Python and Spark.
Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
Work closely with data analysts, data scientists, and business stakeholders to understand data requirements.
Develop efficient and reusable data processing frameworks and components.
Perform data validation, cleansing, and transformation to ensure quality and consistency.
Optimize and tune Spark jobs and SQL queries for performance and scalability.
Collaborate with DevOps and platform teams to deploy and manage data solutions in cloud environments.
Ensure data security, governance, and compliance standards are met.
Troubleshoot production issues, perform root cause analysis, and implement fixes.
Participate in design discussions, contribute to data architecture decisions, and promote best practices.
Work in an Agile environment, supporting sprint activities and continuous improvement initiatives.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre a été publiée par l’un de nos partenaires. Vous pouvez consulter l’offre originale ici.