Data Platform Engineer

Join to apply for the Data Platform Engineer role at Hitachi Consulting (formerly Information Management Group)

We’re Hitachi Digital Services, a global digital solutions and transformation business with a bold vision of our world’s potential. We’re people‑centric and here to power good. Every day, we future‑prove urban spaces, conserve natural resources, protect rainforests, and save lives. This is a world where innovation, technology, and deep expertise come together to take our company and customers from what’s now to what’s next.

We make happen through the power of acceleration.

You will join a high‑performing Data Engineering & Platform team responsible for building and scaling the organization’s cloud‑based data ecosystem. The team treats data as a strategic product and partners closely with engineering, analytics, and business stakeholders to power enterprise‑wide insights and innovation. Operating at the intersection of data engineering, Dev Ops, and cloud architecture, the team focuses on building modern, secure, and scalable solutions using Databricks and AWS.

As a Data Platform Engineer
, you will lead the design, development, and optimization of our large‑scale, cloud‑native data platform. You will architect and build robust ETL/ELT pipelines using PySpark and Databricks, leveraging Delta Lake, Unity Catalog, and Delta Live Tables. You will own Dev Ops automation through Git Hub Actions and ensure fast, reliable deployments of Databricks assets. You will also manage the AWS infrastructure supporting the platform—focusing on secure, scalable, and high‑performing environments.

This role requires deep expertise in distributed data processing, Databricks engineering, CI/CD automation, and cloud infrastructure.

• Build and maintain high‑scale ETL/ELT pipelines across diverse data sources
• Implement and optimize Databricks workflows using PySpark, Python, DLT, and Unity Catalog
• Configure and manage AWS environments including VPCs, IAM, S3, and secure connectivity
• Establish CI/CD pipelines using Git Hub Actions for automated deployment of Databricks notebooks, jobs, and pipelines
• Drive data quality via automated testing frameworks (unit, integration, performance)
• Optimize cluster performance and cost efficiency
• Lead best practices in Medallion Architecture, ACID data principles, and high‑performance SQL
• Create clear technical documentation, architecture diagrams, and design specifications

• 8+ years of experience building scalable Data Engineering platforms and production‑grade pipelines
• 3+ years of hands‑on Databricks development, including expertise in:
  • Delta Lake (ACID, time travel, optimization)
  • Unity Catalog (security, governance, metadata)
  • Delta Live Tables (DLT)
  • Work spaces, Repos, Jobs, and Databricks SQL
• 3+ years of AWS experience, including:
  • VPC, Subnets, Endpoints, Routing
  • IAM roles, policies, cross‑account access
  • S3‑based data lake implementation
• Expert programming skills in Python (4+ years)
• Deep hands‑on experience with PySpark and advanced SQL
• Proven CI/CD experience using Git Hub Actions or similar tools
• Strong understanding of ETL/ELT, Data Lake, Data Warehouse, and distributed computing concepts
• Agile (Scrum) experience and Git proficiency

• Experience with AWS data services such as Glue, Athena, Redshift, RDS, DynamoDB
• Knowledge of real‑time streaming (Kafka, Spark Structured Streaming)
• Experience building ML feature pipelines
• Background in performance tuning and capacity planning for large Spark clusters

We’re a global, team of innovators. Together, we harness engineering excellence and passion to co‑create meaningful solutions to complex challenges. We turn organizations into data‑driven leaders that can make a positive impact on their industries and society. If you believe that innovation can bring a better tomorrow closer to today, this is the place for you.

Hitachi is a global company operating across a wide range of industries and regions. One of the things that sets Hitachi apart is the diversity of our business and people, which…