DevOps Architect

• Candidates should be eligible to work for any employer in the United States without needing Visa sponsorship
• W2 ONLY position, cannot accept 1099 or C2C candidates for this role".

Job Title:
Lead Azure Data Platform Operations Engineer/ Dev Ops Architect/ Databricks Admin

Duration: 11+ months contract

Fully remote, must be willing to work PST time zone.

Must have

Skills:

Azure, Terraform, IaC, and Databricks support.

Job Description

We are seeking an Azure Data Platform Operations leader who will be able to take strategic direction and implement the Ops parts that enable those strategies with a strong focus on automation and infrastructure as code to join our dynamic team. The successful candidate will be responsible for automating the operations, design, implementation, and maintenance of our Azure-based data platforms, including Databricks and other related Azure services.

This role requires expertise in creating reproducible, scalable, and secure environments using infrastructure as code methodologies, particularly within the Azure cloud ecosystem.

Responsibilities:

Platform Automation and Infrastructure as Code:

• Design and implement infrastructure as code (IaC) using tools like Terraform to manage and provision Azure resources systematically and efficiently.
• Automate the setup, scaling, and management of Databricks work spaces and other Azure data services using CI/CD pipelines.
• Develop scripts and tools to automate operational tasks, reduce manual interventions and ensure consistent environments through code.

Platform Administration:

• Administer, configure, and optimize the Databricks platform and other Azure data services to support data analytics, machine learning, and data engineering activities.
• Manage Azure resources including Data Factories, Key Vaults, Storage Accounts, ML Work spaces, and Purview through automated scripts and IaC.
• Monitor, troubleshoot, and optimize resource utilization and platform costs using automated tools and techniques.

Operational Support:

• Ensure platform security, compliance, and reliability through automated patching, updates, and security configuration.
• Implement and manage access controls, security policies, and monitoring solutions to protect sensitive data across all Azure data platforms.

Collaboration and Technical Support:

• Work closely with data engineering, data science/ML, and application/integration teams to automate data workflows and optimize data delivery.
• Provide expert guidance on Azure best practices, especially regarding automation, scalability, and operational efficiency.
• Collaborate with Dev Ops and other technology teams to enhance infrastructure automation and continuous delivery efforts.

Data Management:

• Automate the management of schema data with Unity Catalog, including creation, configuration, cataloging, external storage, and access permissions.
• Develop and maintain automated solutions for data cataloging, metadata management, and governance to support data discovery and lineage tracking.

Skills and Qualifications:

• 3+ years of hands-on experience with the Databricks platform and Azure PaaS.
• 3+ years of experience in infrastructure as code, preferably with Terraform, in a production environment.
• Strong knowledge of Azure cloud services and management of scalable cloud infrastructure.
• Proficient in scripting languages such as Power Shell, Python, or Bash for automation purposes.
• Solid understanding of security best practices, compliance frameworks, and risk management in the cloud.
• Proficiency in software development with Python

Preferred:

• Experience with CI/CD tools (e.g., Jenkins, Azure Dev Ops) for automating data and infrastructure workflows.
• Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes) within Azure.
• Knowledge of real-time data processing technologies (e.g., Spark Streaming, Delta Live Tables, etc...).