Big Data Developer

Location: Bengaluru

Job Description:

· Design, develop, and maintain scalable data processing pipelines using Hadoop and Spark.

· Implement data integration and ETL processes to ingest and transform large datasets.

· Collaborate with data scientists, analysts, business partners and other stakeholders to understand data requirements and deliver solutions.

· Optimize and tune Hadoop and Spark jobs for performance and efficiency.

· Manage and maintain data storage solutions, ensuring data integrity and security.

· Utilize Git Hub for version control and collaboration on code development.

· Work with CDP (Cloudera Data Platform) to manage and deploy data applications.

· Integrate and manage data solutions on Cloud Azure, and Snowflake ensuring seamless data flow and accessibility.

· Monitor and troubleshoot data processing workflows, resolving issues promptly.

· Stay updated with the latest industry trends and technologies in big data and cloud computing.

· Explore and develop skills in newer technologies according to big data technology roadmap.

· Integrate and manage data solutions on Cloud Azure and on-premises infrastructure, ensuring seamless data flow and accessibility.

Qualifications:

· Strong experience with Apache Spark, Hadoop ecosystem, Hive, Iceberg, HBase, Apache Kafka, Spark Streaming or Flink, and Oozie (Airflow/NiFi good to have).

· Proficient in Scala, Java, and/or Python with solid coding and debugging skills.

· Applied experience with Solr for indexing and search workloads.

· Familiarity with Git/Git Hub, Git Hub Actions, Azure Dev Ops; good to have cloud experience (Azure/AWS).

· Knowledge of SQL, data modeling (nice to have), and experience working in UNIX or UNIX‑like environments.

· Experience with SCRUM or similar Agile frameworks; strong problem‑solving skills; able to produce clear technical documentation.

· Self‑starter, effective individual contributor, and good to have Oil & Gas domain knowledge.
Shift 2nd 1.00 PM – 10.00 PM