Data Engineer

Location: Bengaluru

Position:
Data Engineer

Experience:

5–8 years

Location:

Bengaluru

Must‑Have Skills
SQL (Advanced) – Joins, Window functions, query tuning.
Python – ETL scripting, PySpark, automation.
Apache Kafka – Real‑time ingestion, stream processing, Kafka Connect.

Data Pipeline & ETL Development
Design, develop, and maintain batch and real‑time ETL/ELT pipelines using Python, PySpark, and Kafka.
Build data ingestion frameworks to integrate RDBMS, APIs, Kafka streams, flat files, and cloud-native sources.
Implement and optimize CDC, incremental loads, SCD Type‑2, partitioning, and bucketing strategies.
Big Data Engineering
Develop and optimize pipelines using Spark, PySpark, Scala, Hive, Hadoop, and related tooling.
Handle large-scale distributed datasets with advanced performance techniques such as:
Broadcast joins
Shuffle optimization
Data skew resolution
Caching & indexing
Cloud Engineering (AWS / Azure)
Build end-to-end data solutions using:
AWS: S3, Glue, Redshift, EMR, Lambda
Azure: ADF, Synapse, Databricks, Data Lake
Implement orchestration frameworks like Airflow, Oozie, ADF triggers, and Lambda automation.
Data Quality, Governance & Monitoring
Implement end-to-end data validation, audit checks, logging, and monitoring frameworks.
Build metadata-driven and reusable ETL frameworks.
Work with monitoring tools such as Cloud Watch or Databricks cluster logs.
Collaboration, Documentation & Agile Delivery
Work closely with analysts, architects, QA, and business teams in an Agile/Scrum environment.
Prepare technical documentation, architecture diagrams, and SOPs.
Participate in code reviews and contribute to best engineering practices.