Azure Databricks DevOps Engineer

About the job Azure Databricks Dev Ops Engineer

• Databricks Management
  :
  Deploy, configure, and maintain Azure Databricks environments, ensuring scalability, security, and efficiency.
• Infrastructure Automation
  :
  Implement and manage IaC using Terraform, ARM templates, or Bicep to automate cloud resource provisioning.
• CI/CD Pipelines
  :
  Develop and manage CI/CD pipelines using tools like Azure Dev Ops, Git Hub Actions, Jenkins to automate data pipeline deployments.
• Monitoring & Logging
  :
  Set up and maintain monitoring solutions using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution.
• Security & Compliance
  :
  Implement best practices for security, governance, and compliance, including RBAC, encryption, and network security policies.
• Data Integration & Optimization
  :
  Work with Data Engineers to optimize ETL pipelines, ensuring high performance and cost efficiency.
• Collaboration
  :
  Work closely with Data Scientists, Engineers, and Business Analysts to enable efficient data workflows.
• Incident Management
  :
  Troubleshoot and resolve Databricks, Azure Kubernetes Services (AKS), and networking issues efficiently.
• Documentation & Best Practices
  :
  Maintain technical documentation and establish best practices for Azure Databricks and Dev Ops processes.

• Bachelors or Masters degree in Computer Science, IT, or a related field.
• 5+ years of experience in Dev Ops, Cloud Engineering, or Data Engineering roles.
• Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics.
• Proficiency in Terraform, ARM Templates, or Bicep for infrastructure automation.
• Experience with CI/CD tools such as Azure Dev Ops, Git Hub Actions, or Jenkins.
• Knowledge of Python, Scala, or Power Shell for automation and scripting.
• Familiarity with containerization (Docker, Kubernetes, AKS).
• Solid understanding of Azure networking, security, and identity management (IAM, RBAC, AAD).
• Experience with monitoring and logging tools (Azure Monitor, Log Analytics, Prometheus, Grafana).
• Strong problem-solving skills and ability to work in an Agile/Scrum environment.

• Azure Certifications (e.g., AZ-400, AZ-104, DP-203, DP-900).
• Experience with Big Data processing frameworks (Apache Spark, Delta Lake).
• Understanding of ML Ops and working with machine learning models in Databricks.
• Knowledge of Event-driven architectures (Azure Event Hubs, Kafka, Service Bus).