Data Platform Engineer

Our Company

We're Hitachi Digital Services, a global digital solutions and transformation business with a bold vision of our world's potential. We're people-centric and here to power good. Every day, we future-proof urban spaces, conserve natural resources, protect rainforests, and save lives. This is a world where innovation, technology, and deep expertise come together to take our company and customers from what's now to what's next. We make it happen through the power of acceleration.

Imagine the sheer breadth of talent it takes to bring a better tomorrow closer to today. We don't expect you to 'fit' every requirement – your life experience, character, perspective, and passion for achieving great things in the world are equally as important to us.

The team

You will join a high-performing Data Engineering & Platform team responsible for building and scaling the organization's cloud-based data ecosystem. The team treats data as a strategic product and partners closely with engineering, analytics, and business stakeholders to power enterprise-wide insights and innovation. Operating at the intersection of data engineering, DevOps, and cloud architecture, the team focuses on building modern, secure, and scalable solutions using Databricks and AWS.

The role

As a Data Platform Engineer, you will lead the design, development, and optimization of our large-scale, cloud-native data platform. You will architect and build robust ETL/ELT pipelines using PySpark and Databricks, leveraging Delta Lake, Unity Catalog, and Delta Live Tables. You will own DevOps automation through GitHub Actions and ensure fast, reliable deployments of Databricks assets. You will also manage the AWS infrastructure supporting the platform—focusing on secure, scalable, and high-performing environments. This role requires deep expertise in distributed data processing, Databricks engineering, CI/CD automation, and cloud infrastructure.

Key Areas of Ownership:

• Build and maintain high-scale ETL/ELT pipelines across diverse data sources

• Implement and optimize Databricks workflows using PySpark, Python, DLT, and Unity Catalog

• Configure and manage AWS environments including VPCs, IAM, S3, and secure connectivity

• Establish CI/CD pipelines using GitHub Actions for automated deployment of Databricks notebooks, jobs, and pipelines

• Drive data quality via automated testing frameworks (unit, integration, performance)

• Optimize cluster performance and cost efficiency

• Lead best practices in Medallion Architecture, ACID data principles, and high-performance SQL

• Create clear technical documentation, architecture diagrams, and design specifications

What you'll bring

• 5+ years of experience building scalable Data Engineering platforms and production-grade pipelines

• 3+ years of hands-on Databricks development, including expertise in:

  • Delta Lake (ACID, time travel, optimization)

  • Unity Catalog (security, governance, metadata)

  • Delta Live Tables (DLT)

  • Workspaces, Repos, Jobs, and Databricks SQL

• 3+ years of AWS experience, including:

  • VPC, Subnets, Endpoints, Routing

  • IAM roles, policies, cross-