Über
Responsibilities
Platform Management:
Deploy, configure, and maintain OpenShift clusters or GCP projects to support containerized Spark Applications for both GPU and CPU Compute. Data Pipeline Development:
Design and implement large-scale data processing workflows using Apache Spark. Optimization:
Tune Spark jobs for performance, leveraging OpenShift's resource management capabilities (e.g., Kubernetes orchestration, auto-scaling). Integration:
Integrate Spark with other data sources (e.g., Kafka, S3, cloud storage) and sinks (e.g., databases, data lakes). CI/CD Implementation:
Build and maintain CI/CD pipelines for deploying Spark applications in OpenShift or GCP using tools like GitHub Actions, Sonar, Harness. Monitoring & Troubleshooting:
Monitor cluster health, Spark job performance, and resource utilization using OpenShift tools (e.g., Prometheus, Grafana) and resolve issues proactively. Security:
Ensure compliance with security standards, implementing role-based access control (RBAC) and encryption for data in transit and at rest. Collaboration:
Work with cross-functional teams to define requirements, architect solutions, and support production deployments. Qualifications
Experience:
5 years working on Apache Spark for big data processing. 3 years of Django development experience. 2 years of creating and maintaining conda environments. 4 years managing containerized environments with OpenShift or Kubernetes. 2 years managing GPU-based AIML applications.
Technical Skills:
Proficiency in Spark frameworks (Python/PySpark, Scala, or Java). Hands-on experience with OpenShift administration (e.g., cluster setup, networking, storage). Proficiency in creating and maintaining conda environments and dependencies. Familiarity with Docker and Kubernetes concepts (e.g., pods, deployments, services, and images). Knowledge of distributed systems, cloud platforms (AWS, GCP, Azure), and data storage solutions (e.g., S3, HDFS). Programming: Strong coding skills in Python, Scala, or Java; experience with shell scripting is a plus. Tools: Experience with Git Actions, Helm, Harness, and CI/CD tools. Problem-Solving: Ability to debug complex issues across distributed systems and optimize resource usage. Education: Bachelor's degree in Computer Science, Engineering, or related field.
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klick auf „Jetzt Bewerben”, um deine Bewerbung direkt auf deren Website einzureichen.