Über
We are looking for Scala Engineers with experience with batch and/or streaming jobs. We utilize Spark for batch jobs and Flink for real-time streaming jobs. Experience with Hadoop, Hive, AWS S3 is also an asset. Responsibilities include creating new, and maintaining existing, Spark jobs written in Scala, creating new, and maintaining existing, Flink jobs written in Scala, producing unit and system tests for all code, participating in design discussions to improve our existing frameworks, defining scalable calculation logic for interactive and batch use cases, and interacting with infrastructure and data teams to produce complex analysis across data. Required qualifications include a minimum of 2 years of experience with Scala and/or Java, a minimum of 5 years of programming experience, required experience with Hadoop, Spark, knowledge and experience with cloud-based technologies, experience in batch or real-time data streaming, ability to dynamically adapt to conventional big-data frameworks and open source tools if project demands, knowledge of design strategies for developing scalable, resilient, always-on data lake, strong development/automation skills, comfort with reading and writing Scala code, aptitude for analytical problem solving, deep knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance, good understanding/knowledge of HDFS architecture and various components such as Job Tracker, Task Tracker, Name Node, Data Node, HDFS high availability (HA) and Map Reduce programming paradigm. Experienced working with various Hadoop Distributions (Cloudera, Hortonworks, MapR, Amazon EMR) to fully implement and leverage new Hadoop features, experience in developing Spark Applications using Spark RDD, Spark-SQL, Spark -Yarn, Spark Mlib and Data frame APIs, experience with real-time data processing and streaming techniques using Spark streaming and Kafka, moving data in and out HDFS and RDBMS. Familiarity with open source configuration management and development tools.
Sprachkenntnisse
- English
Hinweis für Nutzer
Dieses Stellenangebot stammt von einer Partnerplattform von TieTalent. Klick auf „Jetzt Bewerben”, um deine Bewerbung direkt auf deren Website einzureichen.