Senior Data Engineer
American IT Systems
- Sunnyvale, California, United States
- Sunnyvale, California, United States
À propos
Senior Data Engineer Location: Sunnyvale CA Onsite Requirement: Y Number of days onsite: 5 Days Office Address: If Onsite Office Address
Rate 60-65/hr on W2 Max
Mandatory Areas Must Have Skills: Data Engineer Skill 1: big data, spark, pyspark, apache airflow, etl process Skill 2: 4+ years in GCP
Description
We are seeking a Data Engineer with Spark & Streaming skills builds real-time, scalable data pipelines using tools like Spark, Kafka, and cloud services (GCP) to ingest, transform, and deliver data for analytics and ML.
Responsibilities
As a Senior Data Engineer, you will Design, develop, and maintain ETL/ELT data pipelines for batch and real-time data ingestion, transformation, and loading using Spark (PySpark/Scala) and streaming technologies (Kafka, Flink).
Build and optimize scalable data architectures, including data lakes, data warehouses (Big Query), and streaming platforms.
Performance Tuning: Optimize Spark jobs, SQL queries, and data processing workflows for speed, efficiency, and cost-effectiveness. Data Quality: Implement data quality checks, monitoring, and alerting systems to ensure data accuracy and consistency.
Programming: Strong proficiency in Python, SQL, and potentially Scala/Java.
Big Data: Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
Streaming: Experience with messaging queues like Apache Kafka, or Pub/Sub.
Cloud: Familiarity with GCP, Azure data services.
Databases: Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.
Tools: Experience with Airflow, Databricks, Docker, Kubernetes is a plus.
Experience Level: Total IT Experience - Minimum 8 years. GCP - 4+ years of recent GCP experience.
Required Skills & Qualifications
Programming: Strong proficiency in Python, SQL, and potentially Scala/Java.
Big Data: Expertise in Apache Spark (Spark SQL, DataFrames, Streaming).
Streaming: Experience with messaging queues like Apache Kafka, or Pub/Sub.
Cloud: Familiarity with GCP, Azure data services.
Databases: Knowledge of data warehousing (Snowflake, Redshift) and NoSQL databases.
Tools: Experience with Airflow, Databricks, Docker, Kubernetes is a plus.
#J-18808-Ljbffr
Compétences linguistiques
- English
Avis aux utilisateurs
Cette offre provient d’une plateforme partenaire de TieTalent. Cliquez sur « Postuler maintenant » pour soumettre votre candidature directement sur leur site.