Sr Data Engineer - CA

We are looking for a Senior Data Engineer with strong experience in the Databricks Lakehouse platform and AWS Cloud Services. The ideal candidate will design, build, and optimize scalable data pipelines and analytical solutions, with a preference for candidates having exposure to SAP data processing.

Design and implement end-to-end data pipelines using Databricks Lakehouse architecture.

Develop and optimize data transformations using Spark (PySpark / Spark SQL).

Build and manage data workflows on AWS (S3, Glue, Athena, Redshift, EMR, etc.).

Integrate and process data from SAP systems (ECC, S/4

HANA, BW) into cloud data platforms.

Implement data quality checks, performance tuning, and monitoring for large-scale datasets.

Collaborate with analytics and business teams to support reporting and advanced analytics use cases.

Follow best practices for data security, governance, and access controls.

Work independently in a remote, distributed team environment.

5+ years of relevant experience in Data Engineering.

Strong hands-on experience with Databricks Lakehouse platform.

Solid experience with AWS Cloud Services (S3, Glue, Athena, Redshift, EMR, Lambda).

Proficiency in PySpark, Spark SQL, and SQL.

Experience building scalable ETL/ELT pipelines.

Good understanding of data modeling and performance optimization.

Strong communication skills and ability to work remotely.

Experience or knowledge in SAP data processing (SAP ECC, S/4

HANA, BW, CDS Views).

Exposure to Delta Lake, data versioning, and ACID transactions.

Familiarity with Airflow or other orchestration tools.

Experience in Agile/Scrum delivery models.