Databricks Data Engineer

Overview

Databricks Data Engineer

Location:

Chicago, IL (Hybrid/Remote options as applicable)

Company:
Westberke Consulting

Employment Type:

Contractor

At Westberke Consulting, we bridge the gap between complex data challenges and actionable business intelligence. Based in Elmhurst, IL, our team prides itself on delivering high-impact, scalable solutions for clients ranging from mid-market leaders to enterprise giants. We are looking for a Databricks Data Engineer who isn t just a coder, but a problem solver ready to shape the future of modern Lakehouse architectures.

We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize modern lakehouse architectures. This role focuses on developing scalable data pipelines, implementing Delta Lake best practices, and enabling advanced analytics and AI workloads.

The ideal candidate has hands-on experience building production-grade solutions, strong expertise in Spark-based processing
, and a deep understanding of performance optimization, data modeling, and data governance.

• Development: Design and implement scalable ELT pipelines using PySpark, SQL, and Databricks Workflows.
• Architecture Implementation: Deploy and maintain the Medallion Architecture (Bronze, Silver, Gold layers) to ensure data quality and lineage.
• Performance Engineering: Optimize jobs for cost-efficiency through strategic cluster sizing, Z-Ordering, and partitioning.
• Governance & Security: Manage data access controls, metadata, and security policies using Unity Catalog
  .
• Storage Management: Manage Delta Lake tables, including schema evolution, vacuuming, and time-travel features.
• Collaboration: Work alongside Data Architects and ML Engineers to prepare feature-ready datasets for AI initiatives.
• Dev Ops Integration: Participate in CI/CD life cycles, utilizing Git-based version control and automated release patterns.

• Education: Bachelor’s degree in Computer Science, Information Systems, or a related field.
• Experience: 3+ years of hands-on Databricks experience in a production environment.
• Technical Core: Strong proficiency in Py Spark and Spark SQL
  .
• Delta Lake: Deep understanding of Delta Lake and distributed data processing.
  • Hands-on experience with Unity Catalog
    .
• Cloud Proficiency: Experience with Azure Data Lake Storage (ADLS) or AWS S3.
• Soft Skills: Strong communication skills (essential for consulting) and experience in Agile/SCRUM environments.

• Advanced Ingestion:
  
  Experience with
  Databricks Auto Loader and Structured Streaming (Kafka/Event Hubs).
• AI/ML Ops: Familiarity with MLflow
  , Feature Stores, or Mosaic AI.
• Certifications: Databricks Certified Data Engineer (Associate or Professional).
• Consulting Mindset: Ability to translate technical bottlenecks into business solutions for diverse clients.

• Impact: Work on diverse projects across multiple industries.
• Growth: We invest in our people—access to the latest Databricks certifications and training.
• Culture: A collaborative, Elmhurst-based team that values technical excellence and work-life balance.
• Innovation: Be at the forefront of the "Data Intelligence" era with Generative AI and Lakehouse evolution.