Senior Data Engineer – PySpark, Cloud & Kafka - YoE - r

Position: Senior Data Engineer – PySpark, Cloud & Kafka - 5+ YoE - Immediate Joiner - Any UST Location
Location: Bengaluru

Candidates  ready to join immediately  can share their details via email for quick processing.
CCTC | ECTC | Notice Period | Location Preference

Act fast for immediate attention! ⏳

Key Responsibilities
Design, develop, and maintain scalable  data pipelines  using  Py Spark
Build and manage  batch and real-time data processing  systems
Develop and integrate  Kafka-based streaming solutions
Optimize Spark jobs for performance, cost, and scalability
Work with cloud-native services to deploy and manage data solutions
Ensure data quality, reliability, and security across platforms
Collaborate with data scientists, analysts, and application teams
Participate in code reviews, design discussions, and production support
Must-Have Skills
Strong hands-on experience with  PySpark / Apache Spark
Solid understanding of  distributed data processing  concepts

Experience with  Apache Kafka  (producers, consumers, topics, partitions)
Hands-on experience with  any one cloud platform :
AWS (S3, EMR, Glue, EC2, IAM)  or
Azure (ADLS, Synapse, Databricks)  or
GCP (GCS, Dataproc, Big Query)
Proficiency in  Python
Strong experience with  SQL  and data modeling
Experience working with  large-scale datasets
Familiarity with  Linux/Unix environments
Understanding of  ETL/ELT frameworks

Experience with  CI/CD pipelines  for data applications
Good-to-Have Skills

Experience with  Spark Structured Streaming
Knowledge of  Kafka Connect  and  Kafka Streams
Exposure to  Databricks

Experience with  No

SQL databases  (Cassandra, Mongo

DB, HBase)
Familiarity with  workflow orchestration tools  (Airflow, Oozie)
Knowledge of  containerization  (Docker, Kubernetes)

Experience with  data lake architectures
Understanding of  security, governance, and compliance  in cloud environments
Exposure to  Scala  or  Java  is a plus
Prior experience in  Agile/Scrum  environments