Data Engineer – DataBricks

Senior Databricks Data Engineer

Hybrid Newark, NJ 2 days/month onsite, with flexibility to come in as needed. Contract to Hire. We are hiring a Senior Databricks Data Engineer to help build Horizon's next-generation healthcare data platform. You’ll design large-scale pipelines, implement medallion architecture, optimize Spark performance, and shape the modeling patterns for analytics, clinical, and enterprise data. If you’re a true builder who knows the difference between “running Spark jobs” and actually engineering data systems, this is your playground.

• Architect, design, and develop large-scale ETL/ELT pipelines in Databricks
  , Spark, and Delta Lake.
• Implement medallion architecture and best practices for scalable data modeling.
• Build workflows, notebooks, jobs, and orchestration patterns within Databricks.
• Optimize Spark performance (clusters, caching, partitions, Z-ordering, autoscaling, shuffles).
• Apply data quality frameworks (expectations, validations, profiling).
• Work with cross-functional partners to translate requirements into actionable data solutions.
• Contribute to governance: lineage, cataloging, RBAC, compliance, metadata management.
• Troubleshoot and resolve issues across ETL pipelines, notebooks, and compute clusters.
• Mentor junior engineers as the team expands.

• Expert-level experience with Databricks
  , Spark, PySpark, and Delta Lake.
• Strong SQL and Python.
• Proven experience designing complex pipelines and distributed jobs.
• Experience with cloud platforms (Azure or AWS).
• Solid Mickey hand understanding of data modeling (conceptual, logical, physical).
• Experience with healthcare datasets (claims, eligibility, provider, clinical) is a major plus.
• Ability to work end-to-end: design build optimize document.
Senior Databricks Data Engineer Bonus Skills
• Knowledge of FHIR or other healthcare interoperability standards.
• Experience with orchestration tools (Airflow, ADF, Databricks Jobs).
• CI/CD for data engineering.

#Contech