Databricks Platform Admin; Azure

Location:

St. Louis, Missouri (3 Days onsite/week)

Yrs. of experience: 10+ Yrs.

Contract

• Design, implement, and manage scalable and secure Databricks platforms to ensure optimal performance and reliability.
• Create and manage Databricks work spaces and clusters, including right-sizing and lifecycle management of resources.
• Automate infrastructure provisioning and deployment using Infrastructure-as-Code tools like Terraform, Git Lab CI/CD, and scripting languages (Python, Bash, Power Shell).
• Monitor platform health, performance, and cost by building dashboards and integrating with monitoring tools.
• Troubleshoot and resolve issues related to cluster configuration, job failures, and performance bottlenecks within Databricks environments.
• Collaborate with cross-functional teams including data scientists, ML engineers, and product teams to deploy and maintain data pipelines and services on Databricks.
• Implement security and governance policies, including user management, access controls, and compliance adherence using Unity Catalog and other Databricks security features.
• Optimize compute and storage resources for cost efficiency without compromising performance, leveraging features like Delta Lake and Photon.
• Stay updated with new Databricks features and coordinate with product teams to evaluate and adopt them.
• Document platform architecture, best practices, troubleshooting guides, and provide mentorship to junior engineers on platform usage and optimization.
• Participate in architect’s forum, interact with application owners and users, and provide viable solutions based on requirements.
• Coordinate between offshore team and customer stakeholders to ensure the requests are fulfilled on time.

• Required hands‑on experience in architecting, deploying, managing, and supporting Azure Databricks Platform, Azure Datalake, and related solutions.
• Required hands‑on experience working with Cloud provider tools and architecture (Azure).
• Required hands‑on experience using scripting languages and command‑line tools such as Ansible, SQL, Terraform, Azure CLI, Databricks CLI, Power Shell, and/or Bash.
• Preferred experience in management and tuning of data platforms like Azure ADLS storage and Databricks.
• Knowledge of maintaining CI/CD pipelines and versioning tools such as Git.
• Knowledge of a broad set of data ingestion and pipeline patterns, and associated tools such as Qlik Replicate, Qlik Compose, Databricks, Kafka, Matillion ELT, and Snowflake Services.
• Experience working in an agile product development team.
• Excellent communication skills and ability to articulate system designs and patterns to varying levels of leadership.
• Ownership of solutions and demonstration of leadership in working with teams.