Data EngineerTata Consultancy Services • O'Fallon, Missouri, United States
Cette offre d'emploi n'est plus disponible
Data Engineer
Tata Consultancy Services
- O'Fallon, Missouri, United States
- O'Fallon, Missouri, United States
À propos
Big Data/Storage: Apache Spark, MinIO, AWS S3.
Key Responsibilities
Pipeline Orchestration: Design and develop complex, reusable DAGs in Apache Airflow to automate data workflows, scheduling, and monitoring across the enterprise.
Data Ingestion & Flow: Create and optimize high-volume data streams using Apache NiFi, leveraging custom processors and controller services for diverse data sources and sinks.
Custom Development: Utilize Java to develop custom NiFi processors or troubleshoot the core NiFi framework, ensuring the platform meets specific architectural requirements.
Object Storage Management: Implement and manage data storage solutions using MinIO and AWS S3, ensuring high availability and efficient data retrieval patterns.
Large-Scale Processing: Develop and maintain Apache Spark jobs for heavy-duty data transformations, integrating them into NiFi and Airflow orchestration layers.
Security & Certificate Management: Secure NiFi clusters and data flows using TLS/mTLS, managing Java KeyStores (JKS), TrustStores, and SSL certificate lifecycles to ensure data-in-motion security.
System Administration: Perform environment setup, performance tuning, and troubleshooting within Unix/Linux environments, including shell scripting for task automation.
Cloud Integration: Deploy and manage data infrastructure components on AWS, utilizing IAM for access control and integrating cloud-native services with hybrid data pipelines.
Monitoring & Optimization: Establish robust logging and alerting for data pipelines to proactively identify bottlenecks, ensuring 99.9% reliability of data delivery.
#J-18808-Ljbffr
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre a été publiée par l’un de nos partenaires. Vous pouvez consulter l’offre originale ici.