Data Engineer - Senior Associate

Location: Prakāshamnagar

Let's do this. Let's change the world. We are looking for highly motivated expert Data Engineer who can own the design & development of complex data pipelines, solutions and frameworks. The ideal candidate will be responsible to design, develop, and optimize data pipelines, data integration frameworks, and metadata-driven architectures that enable seamless data access and analytics. This role prefers deep expertise in big data processing, distributed computing, data modeling, and governance frameworks to support self-service analytics, AI-driven insights, and enterprise-wide data management.

Roles & Responsibilities :
Design, develop, and maintain complex ETL/ELT data pipelines in Databricks using PySpark, Scala, and SQL to process large-scale datasets
Understand the biotech/pharma or related domains & build highly efficient data pipelines to migrate and deploy complex data across systems
Design and Implement solutions to enable unified data access, governance, and interoperability across hybrid cloud environments
Ingest and transform structured and unstructured data from databases (Postgre

SQL, MySQL, SQL Server, Mongo

DB etc.), APIs, logs, event streams, images, pdf, and third-party platforms
Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring
Expert in data quality, data validation and verification frameworks
Innovate, explore and implement new tools and technologies to enhance efficient data processing
Proactively identify and implement opportunities to automate tasks and develop reusable frameworks
Work in an Agile and Scaled Agile (SAFe) environment, collaborating with cross-functional teams, product owners, and Scrum Masters to deliver incremental value
Use JIRA, Confluence, and Agile Dev Ops tools to manage sprints, backlogs, and user stories.
Support continuous improvement, test automation, and Dev Ops practices in the data engineering lifecycle
Collaborate and communicate effectively with the product teams, with cross-functional teams to understand business requirements and translate them into technical solutions

Must-Have

Skills:

Hands-on experience in data engineering technologies such as Databricks, PySpark, Spark

SQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies.
Proficiency in workflow orchestration, performance tuning on big data processing.
Strong understanding of AWS services
Ability to quickly learn, adapt and apply new technologies
Strong problem-solving and analytical skills
Excellent communication and teamwork skills

Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and Dev Ops practices.

Good-to-Have

Skills:

Data Engineering experience in Biotechnology or pharma industry
Experience in writing APIs to make the data available to the consumers
Experienced with SQL/NOSQL database, vector database for large language models
Experienced with data modeling and performance tuning for both OLAP and OLTP databases
Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins, Maven etc.), automated unit testing, and Dev Ops

Education and Professional Certifications
Any Degree and 6-8 years of experience
AWS Certified Data Engineer preferred
Databricks Certificate preferred
Scaled Agile SAFe certification preferred