Senior Data Engineer

Primary Skills

Spark – PySpark, SQL (Basic + Advanced), Python, Hive, Data Modelling Fundamentals

• 3+ years of hands‑on experience in production support for data engineering or analytics applications.
• Proficiency in PySpark for managing and troubleshooting data processing jobs.
• Solid experience with AWS cloud services, including EKS (Elastic Kubernetes Service), EMR (Elastic Map Reduce), S3, Glue, and Step Functions.
• Hands‑on experience with Control‑M for workflow orchestration and monitoring.
• Working knowledge of Mongo
  
  DB and Snowflake for production data management.
• Experience with log monitoring and analysis using Elasticsearch Service (ESS).
• Good understanding of ETL processes, incident management, and production monitoring best practices.
• Strong problem‑solving, troubleshooting, and analytical skills.
• Excellent communication and collaboration abilities.

• Provide production support for data pipelines and workflows, ensuring high availability and timely resolution of incidents.
• Monitor, troubleshoot, and optimize PySpark-based data processing jobs.
• Support and maintain data processing applications deployed on AWS services, including EKS and EMR.
• Manage and orchestrate workflows using Control‑M, including scheduling, monitoring, and error handling.
• Perform root cause analysis and implement permanent fixes for recurring production issues.
• Oversee data storage and management in Mongo
  
  DB and Snowflake, ensuring data quality and consistency.
• Monitor and analyze application and system logs using Elasticsearch Service (ESS) to proactively identify and resolve issues.
• Collaborate with data engineering, analytics, and operations teams to deliver reliable data solutions.
• Maintain and update technical documentation related to production support processes and incident management.
• Participate in on‑call rotations and respond to critical production alerts as needed.