Data Engineer - Databricks

Job Title:

Data Engineer

Location:

Coimbatore

Experience:

3–5 Years
Preferred Certification:  Databricks Data Engineer Associate/Professional

About the Role
We are looking for a skilled  Data Engineer  with hands-on experience in building scalable data pipelines and working extensively on  Databricks . The ideal candidate should have strong data engineering fundamentals, expertise in cloud platforms, and the ability to collaborate with cross-functional teams to deliver reliable data solutions.

Key Responsibilities
Design, develop, and maintain  ETL/ELT pipelines  using Databricks and related technologies.
Build and optimize  data architectures , data models, and data workflows.
Work with large-scale datasets to ensure efficient data ingestion, transformation, and processing.
Implement  Delta Lake  best practices, including schema evolution, time travel, and ACID transactions.
Collaborate with data scientists, analysts, and business teams to understand data requirements.
Monitor and improve data pipeline performance and reliability.
Ensure data security, compliance, and governance practices are followed.
Troubleshoot and resolve issues related to data processing, orchestration, and pipeline failures.
Automate repetitive processes to improve efficiency.

Required

Skills & Qualifications
3–5 years  of experience as a Data Engineer or similar role.
Strong expertise in  Databricks , Apache Spark (PySpark/Scala), and Delta Lake.
Hands-on experience with  cloud platforms  (AWS, Azure, or GCP — Azure preferred if using Azure Databricks).
Proficiency in  SQL  and working with relational/non‑relational databases.

Experience with  data warehousing concepts  and performance optimization.
Solid understanding of  ETL/ELT pipelines , data modeling, and data processing frameworks.
Experience in  Power BI  for building dashboards, data visualizations, and analytics reports.
Ability to integrate Power BI with Databricks or other data sources.

Experience with  Git , CI/CD, and data orchestration tools (Airflow, ADF, etc.).
Strong problem-solving and analytical skills.

Preferred Qualifications
Databricks Certification  (Data Engineer Associate or Professional).

Experience with  Azure Data Factory ,  Azure Synapse ,  AWS Glue , or similar tools.
Knowledge of  MLflow , Unity Catalog, or data governance frameworks.
Familiarity with containerization (Docker) and/or Kubernetes.

Soft Skills
Strong communication and collaboration skills.
Ability to work in a fast-paced and dynamic environment.
Attention to detail with a focus on data accuracy and quality.
Team player with a proactive attitude.