Azure Databricks Engineer

The Azure Databricks Engineer will design, build, and optimize scalable data pipelines and lakehouse solutions using Azure Databricks. This role is critical for enabling advanced analytics and supporting machine learning initiatives. This role requires US Citizenship.

The preferred location for this role is Lafayette, LA, however this position can also be filled out of Belton, TX, Lebanon, VA or Knoxville TN. (Hybrid role)

Your future duties and responsibilities:

• Develop and maintain ETL/ELT pipelines using Databricks workflows, Delta Lake, and Apache Spark.
• Architect and optimize data lakehouse solutions for efficient storage and retrieval.
• Configure and manage Databricks work spaces, clusters, and implement security and cost optimization strategies.
• Build real-time streaming and batch processing solutions for diverse data sources.
• Collaborate with data scientists to deploy ML models and automate ML pipelines using MLflow.
• Monitor pipeline performance, troubleshoot issues, and optimize Spark jobs for efficiency.

Salary Range: $80,600.00-$

Required qualifications to be successful in this role:

• BA/BS/MS in Computer Science, Engineering, or Data Science.
• 5+ years as a Data Engineer, with 3+ years in Azure Databricks.
• Expertise in Apache Spark, Delta Lake, Python, SQL, Scala.
Proficiency in Azure Data Factory, Data Lake Storage, Synapse Analytics.
• Strong understanding of ETL/ELT, data modeling, and big data technologies.

Desired qualifications/non-essential skills required:

• Certifications:
  
  Microsoft Certified:
  Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer.
• Experience with Agile methodology, data governance, and security.