Data Engineer

TOP SKILLS REQUIRED:

1. Spark 2. Python 3. Erwin 4. Scripting Airflow for GPC 5. no more than 8 years of experience and strong tenure thoughout

Designing and Building ETL pipeline using Sqoop, Hive, Map Reduce and Spark on on-prem and cloud environments.

Functional Programming using Python and Scala for complex data transformations and in-memory computations.

Using Erwin for Logical/Physical data modeling and Dimensional Data Modeling.

Designing and developing UNIX/Linux scripts for handing complex File formats and structures

Orchestration of workflows and jobs using Airflow and Automic,

Creating Multiple Kafka producers and consumers for data transferring

Performing Continuous Integration and deployment (CI/CD) using tools like GIT, Jenkin to run test cases and build applications with code coverage using Scala test

Analyzing data using SQL, Big Query monitoring the cluster performance, setting up alerts, documenting the designs, workflow.

Providing production support, troubleshooting and fixing the issues by tracking the status of Running applications to perform System Administrator tasks.