Databricks Data Engineer Lead

We’re Hiring:
Databricks Data Engineer – Lead Level

Location:

Hyderabad, India

Work Model:
Hybrid (3 days/week in office – mandatory)

Experience:

10–12+ Years

Paves Technologies is looking for a highly experienced Databricks Data Engineer (Lead Level) to drive enterprise-scale data architecture, governance, and advanced analytics initiatives on the Azure Cloud. This leadership role requires deep expertise in Azure Databricks, PySpark, and distributed data systems, along with strong architectural and mentoring capabilities.

Role Overview

The ideal candidate will have 10–12+ years of data engineering experience, including strong hands-on expertise in Azure Databricks, PySpark, Python, and Azure data services. You will lead the design and implementation of scalable data platforms, drive modernization initiatives, and implement Medallion Architecture (Bronze, Silver, Gold layers) to support enterprise analytics and business intelligence.

Key Responsibilities

• Lead end-to-end architecture and implementation of enterprise data platforms using Azure Databricks

• Design and implement Medallion Architecture (Bronze, Silver, Gold layers) using Delta Lake best practices

• Build scalable PySpark-based ETL/ELT pipelines across ingestion, transformation, and analytics layers

• Develop advanced data transformations using Python, PySpark, Spark SQL, and advanced SQL

• Architect robust data models (dimensional, star schema, normalized models) for analytics and reporting

• Drive adoption of Databricks capabilities including Unity Catalog, Declarative Pipelines, and Delta Lake optimization

• Implement performance optimization techniques such as partitioning strategies, file compaction, Z-ordering, caching, and broadcast joins

• Define best practices, CI/CD frameworks, reusable templates, and infrastructure automation

• Work with Azure ecosystem tools such as Azure Data Factory (ADF), ADLS, Azure Dev Ops, networking, and security services

• Ensure data quality, RBAC, governance, lineage tracking, and production stability

• Provide technical leadership and mentorship to data engineering teams

✅ Required Skills & Experience

• 10–12+ years of experience in Data Engineering

• 3+ years of hands-on experience with Databricks

• Strong expertise in PySpark, Python, and distributed Spark processing

• Experience implementing Medallion Architecture (Bronze/Silver/Gold)

• Advanced knowledge of Data Modeling, Data Analysis, and complex SQL (CTEs, window functions, execution plan tuning)

• Strong understanding of Delta Lake architecture, schema evolution, partitioning strategies, and performance optimization

• Experience working with Azure Data platforms and cloud-native architectures

• Strong leadership, stakeholder management, and communication skills

Mandatory Certifications

• Databricks Certified Data Engineer – Associate

• Databricks Certified Data Engineer – Professional

How to Apply

If you are interested and meet the required qualifications, please share your resume at:

co

R

#Hiring #Databricks #Data Engineer #Azure #Big Data #PySpark #Data Engineering #Delta Lake #Cloud Engineering #Hyderabad Jobs #Tech Hiring #Analytics #Data Architecture