Data Engineering Lead

Location: New York

Job Summary
:
Tata Consultancy Services is a global leader in IT services, and they are seeking a Data Engineering Lead. The role involves designing and operating data pipelines, ensuring data quality and governance, and collaborating with stakeholders to deliver reliable datasets for analytics and AI use cases.

Responsibilities
:

• Ingest and model data from APIs, files/SFTP, and relational sources; implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python.

• Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks.

• Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute; manage jobs, secrets, and access.

• Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, policies, versioning, quotas, and monitoring.

• Enforce data quality and governance through data contracts, validations/tests, lineage, observability, and proactive alerting.

• Optimize performance and cost via partitioning, clustering, query tuning, job sizing, and workload management.

• Uphold security and compliance (e.g., PII handling, encryption, masking) in line with firm standards.

• Collaborate with stakeholders (analytics, AI engineering, and business teams) to translate requirements into reliable, production ready datasets.

• Enable AI/LLM use cases by packaging datasets and metadata for downstream consumption, integrating via Model Context Protocol (MCP) where appropriate.

• Continuously improve platform reliability and developer productivity by automating routine tasks, reducing technical debt, and maintaining clear documentation.

Qualifications
:
Required
:

• AWS Data Engineering Services (EMR/Glue, Redshift, Aurora, S3, Lambda)

• Spark

• Python

• Collibra

• Snowflake/Databricks

• Tableau

• Ingest and model data from APIs, files/SFTP, and relational sources; implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python.

• Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks.

• Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute; manage jobs, secrets, and access.

• Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, policies, versioning, quotas, and monitoring.

• Enforce data quality and governance through data contracts, validations/tests, lineage, observability, and proactive alerting.

• Optimize performance and cost via partitioning, clustering, query tuning, job sizing, and workload management.

• Uphold security and compliance (e.g., PII handling, encryption, masking) in line with firm standards.

• Collaborate with stakeholders (analytics, AI engineering, and business teams) to translate requirements into reliable, production ready datasets.

• Enable AI/LLM use cases by packaging datasets and metadata for downstream consumption, integrating via Model Context Protocol (MCP) where appropriate.

• Continuously improve platform reliability and developer productivity by automating routine tasks, reducing technical debt, and maintaining clear documentation.

• 4–15 years of professional data engineering experience.

• Strong Python, SQL, and Spark (PySpark) skills, and/or Kafka.

• Snowflake (Snowpipe, Tasks, Streams) as a complementary warehouse.

• Databricks (Delta formats, workflows, cataloging) or equivalent Spark platforms.

• Hands-on experience building ETL/ELT with Prefect (or Airflow), dbt, Spark, and/or Kafka.

• Experience onboarding datasets to cloud data platforms (storage, compute, security, governance).

• Familiarity with Azure/AWS/GCP data services (e.g., S3/ADLS/GCS; Redshift/Big Query; Glue/ADF).

• Git-based workflows CI/CD and containerization with Docker (Kubernetes a plus).

• Strategic Technical Leadership:
Defining data architecture, evaluating new technologies, and setting technical standards for AWS-based pipelines

• Stakeholder Communication:
Bridging the gap between technical teams and business stakeholders, gathering requirements, and reporting progress

•

Risk Management:

Proactively identifying potential bottlenecks in data workflows, security risks, or scalability issues

• Operational Excellence:
Implementing automation, optimizing costs, and maintaining high data quality standards.

• BACHELOR OF COMPUTER SCIENCE

Company
:
Tata Consultancy Services is a business solutions company that specializes on information technology services and consulting. Founded in 1968, the company is headquartered in Mumbai, IND, with a team of 10001+ employees. The company is currently Late Stage.