AI Engineer - Azure Databricks

Experience

Required:

10+ Years

Skills:

• Digital:
  Microsoft Azure

• Digital:
  Azure Machine Learning (ML)

• Hands-on experience with Unity Catalog governance and lineage
  .

• Strong experience with Azure services:

  • ADLS Gen2

  • Azure Data Factory (ADF) Pipelines

  • Azure Key Vault

  • Azure Event Hub / Service Bus

  • Azure Dev Ops

• Proven experience delivering RAG and agentic patterns
  , including:

  • Embeddings pipelines

  • Chunking strategies

  • Vector indexing

  • Evaluation frameworks

• Solid SQL expertise and strong understanding of:

  • Data modeling

  • Medallion Architecture (Bronze/Silver/Gold)

  • ACID transactions

• Proven experience implementing Model Context Protocol (MCP) for agent-tool integrations:

  • Building MCP servers/adapters to expose tools/resources (SQL, REST, files, APIs, etc.)

  • Defining tool schemas, input/output contracts, and safety guardrails

  • Securing MCP integrations using Azure AD (Entra ), Key Vault, secret scopes, and network controls

  • Connecting MCP to Unity Catalog governed assets, Vector Search, and Model Serving

• Build GenAI-ready datasets, embeddings pipelines, and vector indexes for RAG workloads.

• Use Databricks Genie to accelerate:

  • Code creation

  • Optimization

  • Documentation

  • Refactoring

• Implement MCP patterns to connect AI agents to enterprise data and tools.

• Design, build, and operate MCP servers/adapters that expose tools such as:

  • SQL

  • REST APIs

  • File systems

  • Enterprise APIs

• Implement:

  • Tool schemas

  • Resource providers

  • Prompt and tool safety policies

• Integrate MCP with Azure Databricks workloads including:

  • Delta Tables

  • Unity Catalog

  • Vector Search

  • Model Serving

• Secure MCP endpoints using:

  • Key Vault

  • Secret scopes

  • Network rules

  • Role-based access control aligned with Unity Catalog

• Operationalize RAG and agentic workloads using:

  • Vector Search

  • Model Serving

  • Feature Store

  • MLflow

• Integrate Azure Databricks with:

  • ADLS Gen2

  • ADF Pipelines

  • Key Vault

  • Event Hub / Service Bus

  • Azure Functions

• Implement telemetry and observability using:

  • Azure Monitor

  • Log Analytics

  • Databricks metrics

• Implement Unity Catalog for:

  • Centralized governance

  • Data lineage

  • Fine-grained access controls across workspaces

• Apply Medallion Architecture (Bronze / Silver / Gold).

• Enforce data quality using:

  • Delta Live Tables

  • Expectations framework

• Manage:

  • PII/PHI controls

  • Data masking

  • Secure data sharing (e.g., Delta Sharing where applicable)

• Build CI/CD pipelines using:

  • Azure Dev Ops

  • Git Hub Actions

• Use Databricks Repos for version control.

• Orchestrate workloads using:

  • Databricks Workflows

  • Azure Data Factory

• Implement environment promotion paths (Dev → Test → Prod).