Senior Databricks Engineer

Our client is seeking a skilled and passionate Databricks Engineer to design and build scalable, cloud-based data pipelines for enterprise analytics platforms. The role focuses on creating robust ETL/ELT workflows using Databricks and Spark, with opportunities to leverage advanced features such as Delta Live Tables (DLT) and cloud-native tools. The ideal candidate has strong hands-on experience in PySpark and SQL, and a proven ability to build pipelines from the ground up while collaborating with cross-functional teams.

• Design and develop scalable batch and streaming data pipelines in Databricks using PySpark, SQL, and Delta Lake.
• Build ETL/ELT processes to transform and integrate data from diverse sources into unified analytical datasets.
• Leverage Delta Live Tables (DLT) and Databricks Workflows to automate and orchestrate pipelines (preferred).
• Optimize pipelines for performance, cost-efficiency, and reusability by applying Spark tuning and partitioning strategies.
• Collaborate with architects, data modelers, and analysts to implement business logic into pipelines.
• Implement data quality checks, monitoring, and observability practices across pipelines.
• Participate in Agile delivery processes, including sprint planning, code reviews, and deployments.
• Work with cloud platform teams to ensure proper storage, networking, and security setup.
• Deliver high-quality outputs, meet deadlines, and maintain strong client relationships.
• Provide consultative guidance, seek creative solutions, and proactively identify opportunities for improvement.
• Stay current with evolving technology trends and best practices in data engineering.

• 5+ years of experience in data engineering, including at least 3+ years with Databricks.
• Strong hands-on experience with Apache Spark and/or Delta Live Tables (DLT).
• Proficiency in PySpark and SQL for building and optimizing large-scale data pipelines.
• Experience designing and implementing ETL/ELT workflows in cloud environments (Azure or AWS).
• Familiarity with Delta Lake, Unity Catalog, and Databricks Jobs/Workflows.
• Experience with structured and semi-structured data from APIs, cloud storage, and enterprise systems.
• Strong understanding of modern data architecture concepts (Medallion architecture, lakehouse patterns).
• Knowledge of Git-based version control, CI/CD practices, and infrastructure automation.
• Excellent problem-solving and communication skills, particularly in client-facing settings.
• Bachelor’s degree in Computer Science, Information Systems, or equivalent experience.

• Experience with Delta Live Tables (DLT) for declarative pipeline development.
• Exposure to IDMC (Informatica Intelligent Data Management Cloud) or other integration platforms.
• Databricks or cloud provider certifications (Azure/AWS).
• Familiarity with data governance tools (Unity Catalog, Purview, Collibra).
• Experience with REST APIs and streaming technologies (Kafka, Event Hubs, etc.).

Databricks Certified Data Engineer or equivalent cloud certifications (preferred).

Bachelor’s degree in computer science, Information Systems, or a related field (Master’s degree preferred).