Senior Data Engineer (Hadoop + GCP Dataproc)

Required Qualifications:

• 6–10+ years of experience working within the Hadoop ecosystem, with deep expertise in Hive, Hive Metastore, and GCP Dataproc.
• Strong hands-on experience with Dataproc Serverless, BigQuery, Google Cloud Storage (GCS), Cloud Composer, and Cloud Logging/Monitoring.
• Solid understanding of table formats (Parquet, ORC), partitioning/bucketing strategies, and query performance optimization.
• Proven experience migrating Hive datasets, tables, and SQL queries to BigQuery, including handling syntax differences, functions, UDFs, and performance tuning.
• Strong knowledge of Security/IAM best practices, service accounts, and network/security controls; ability to operate within strict organization-level policies.

Nice-to-Have Skills :

• Familiarity with Dataproc Metastore vs. standalone Hive Metastore, and experience with Glue or other metadata/catalog services.
• Exposure to data quality frameworks and lineage tools (e.g., OpenLineage, Collibra).
• A FinOps mindset, including experience managing quotas, reservations, and cost governance for Dataproc and BigQuery workloads.