Data Engineer

Job Title:

Databricks Consultant/Data Engineer

Work Mode:
Remote

Experience

Required:

5–8 Years

Job Overview

We are seeking a highly skilled Databricks Consultant with strong hands-on development experience in building scalable data pipelines and modern lakehouse architectures. The ideal candidate should have deep expertise in Databricks, PySpark, and data warehousing concepts, along with real-time streaming and performance optimization experience.

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Databricks, Delta Lake, and Spark SQL
- Implement ETL/ELT processes for large-scale data systems and Enterprise Data Warehouses (EDW)
- Work with Star and Snowflake schemas for dimensional modeling
- Implement and manage Slowly Changing Dimensions (SCD Type 1 & Type
2)
- Develop real-time data ingestion and processing pipelines using Databricks Autoloader and streaming frameworks
- Optimize Spark jobs and data pipelines for performance, cost, and reliability
- Perform data transformations, validations, profiling, and quality checks using efficient SQL and Py Spark
- Collaborate with stakeholders to understand business requirements and deliver data solutions
- Use Git for version control and follow CI/CD best practices for deployment

Required Skills & Experience

- Strong hands-on development experience with Databricks (Notebooks, Jobs, Workflows, Delta Lake)
- Advanced expertise in PySpark, Spark SQL, RDD/Data Frame APIs
- Solid understanding of data warehousing fundamentals
- Star/Snowflake schema
- SCD Type 1 & Type 2
- ETL/ELT concepts
- Practical experience with Databricks Autoloader (real-world implementation required)
- Strong experience in real-time/streaming data processing
- Proven expertise in Spark performance tuning
- Proficiency in writing optimized SQL for transformations and data analysis

- Experience with cloud platforms — Azure preferred (AWS/GCP acceptable)
- Familiarity with CI/CD pipelines and version control using Git