Senior Data Engineer _ Python with SparkTata Consultancy Services • Jersey City, New Jersey, United States
Senior Data Engineer _ Python with Spark
Tata Consultancy Services
- Jersey City, New Jersey, United States
- Jersey City, New Jersey, United States
Über
We are seeking a Senior Data Engineer with 10-15 years of experience to design, develop, and maintain scalable data pipelines using Python and Spark in a cloud environment.
Responsibilities
Strong hands‑on experience in Python programming for data engineering and processing.
Extensive experience with Apache Spark (PySpark) for large‑scale data processing and distributed computing.
Proficiency in SQL and working with relational and non‑relational databases.
Experience building and maintaining ETL/ELT pipelines for data ingestion and transformation.
Good understanding of data warehousing concepts, data modeling, and data architecture.
Experience working with big data technologies such as Hadoop ecosystem, Hive, or similar platforms.
Familiarity with cloud platforms (AWS, Azure, or GCP) and related data services.
Hands‑on experience with data pipeline orchestration tools such as Airflow or similar.
Knowledge of data formats such as Parquet, Avro, JSON, and CSV.
Experience with performance tuning and optimization of Spark jobs and data pipelines.
Strong problem‑solving skills and ability to collaborate with cross‑functional teams.
Design, develop, and maintain scalable data pipelines using Python and Spark.
Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
Work closely with data analysts, data scientists, and business stakeholders to understand data requirements.
Develop efficient and reusable data processing frameworks and components.
Perform data validation, cleansing, and transformation to ensure quality and consistency.
Optimize and tune Spark jobs and SQL queries for performance and scalability.
Collaborate with DevOps and platform teams to deploy and manage data solutions in cloud environments.
Ensure data security, governance, and compliance standards are met.
Troubleshoot production issues, perform root cause analysis, and implement fixes.
Participate in design discussions, contribute to data architecture decisions, and promote best practices.
Work in an Agile environment, supporting sprint activities and continuous improvement initiatives.
Qualifications Bachelor's degree in Computer Science or related field; 10‑15 years of experience in data engineering.
Benefits
Discretionary Annual Incentive
Comprehensive Medical, Dental, Vision, Disability, and Pet Insurance plans
Family support: Maternal and Parental Leaves
Insurance options: Auto, Home and Identity Theft Protection
Convenience and Professional Growth: Commuter Benefits, Certification and Training Reimbursement
Time Off: Vacation, Personal and Sick leave, Holidays
Legal and Financial Assistance: Legal Assistance, 401(k), Performance Bonus, College Fund, Student Loan Refinancing
#J-18808-Ljbffr
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klick auf „Jetzt Bewerben”, um deine Bewerbung direkt auf deren Website einzureichen.