More jobs:
Databricks Pipeline Engineer
Job in
City Of London, Central London, Greater London, England, UK
Listed on 2026-01-15
Listing for:
Access Computer Consulting
Full Time
position Listed on 2026-01-15
Job specializations:
-
IT/Tech
Data Engineer, Data Science Manager, Cloud Computing
Job Description & How to Apply Below
Job Overview
I am recruiting for a Databricks Pipeline Engineer to work on a remote basis. The role falls inside IR35 so you will be required to work through an umbrella company for the duration of the contract.
Responsibilities- You must have experience of designing, optimising and maintaining scalable Databricks pipelines (ETL, streaming, ML workflows).
- You will be able to perform cluster and job performance tuning: optimise cluster sizing, caching, partitioning, and shuffle management.
- You will monitor Spark job metrics, analyse logs, and identify bottlenecks in data throughput or latency.
- You will also be able to implement cost-optimisation strategies for Databricks jobs and clusters using autoscaling and job consolidation.
- Strong expertise in Databricks Pipelines.
- Familiarity with orchestration tools:
Databricks Workflows, Airflow, or Azure Data Factory. - Proficiency in Python or Scala for data engineering and pipeline development is essential.
- Hands-on experience with Azure, AWS, or multi-cloud Databricks deployments.
- rorre of data storage layers (Azure Data Lake Storage, AWS S3) and performance trade-offs.
- Experience of version control (Git, Git Hub Actions, Dev Ops pipelines) and CI/CD practices for Databricks.
Please apply ASAP to find out more
#J-18808-LjbffrNote that applications are not being accepted from your jurisdiction for this job currently via this jobsite. Candidate preferences are the decision of the Employer or Recruiting Agent, and are controlled by them alone.
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
Search for further Jobs Here:
×