Lead Data Engineer; Azure​/Databricks

Overview

Location:

Sheffield (2 days per week/ Hybrid)

Reporting to:
VP of Engineering

The Team: 2 Data Scientists, 1 Data Engineer

Intelli

AM is an intelligent asset management company specialising in industrial IoT data. We ingest high-frequency factory and sensor data, apply machine learning techniques, and present high value actionable insights to analysts and customers.

We are building a cutting-edge UNS aligned Industrial IoT data platform with a highly scalable architecture to handle our next phase of growth. We require a delivery-focused technical lead to own the data engineering domain, re-architect and build our pipelines from the ground up, and mentor a small but talented team.

You will not be maintaining a legacy system; you will be architecting the new one. You will take ownership of the data flows within the platform, designing a robust architecture that supports both real-time operational views and deep historical analysis.

Primary Objectives (First 6–12 Months)

• Re-platforming: Own the redesign and rebuild of the data transformation layer (post-event bus) in Databricks. Move from ad-hoc scripts to software engineering standards (CI/CD, testing, modular code).
• Modelling Implementation: Support the implementation of the “Unified Name Space” (UNS) across the data estate, to include schema and path standardisation; machine hierarchies and semantic relationships; and methodologies for validation and processing of data against contracts.
• Enablement: Establish a stable data serving layer for Grafana and Analytics (via Databricks/MLflow), unblocking the Data Science team.
• Mentorship: Upskill the existing team in good practices (dbt, git workflows, SQL optimisation, data modelling).

• Pragmatic Architect: Capable of making trade-offs between “academic perfection” and “business value”. They must align architecture to the 80/20 rule.
• Technical Authority: Comfortable discussing and constructively challenging architectural decisions with the Head of Software and VP of Engineering. Must be able to justify changes to the core architecture and challenge existing thinking.
• Mentor vs. Manager: Willing to sit and pair-program with junior engineers. They should measure their success by the team’s outcomes, not simply their own. A full “Team Manager” is not required; this is not a pastoral-focused hire.
• Business Translator: Able to explain to non-technical stakeholders why changes are required, and what their impacts may be. Similarly, the ability to provide a bridge between technical approaches and limitations, and business requirements.
• Industrial / IoT Context: Highly desirable. You should be comfortable with the chaotic nature of sensor data, time-series continuity, and the physical reality of the machines we are modelling.
• Alternative: Experience in high-volume data systems that face similar challenges (duplicated or missing data, spiky load, variable schemas, etc.) would be beneficial.

• Advanced Data Engineering: Proven experience building data platforms on Azure. You know Databricks inside out – not just how to write a notebook, but how to architect a Lakehouse, manage clusters, and optimise costs.
• Data Modelling Proficiency: You understand that data engineering is more than moving JSON blobs. You have strong opinions on schema design, data standardisation, and ideally have worked with graph data or hierarchical models.
• Code Quality: You treat data pipelines as software. You use git, CI/CD, and automated testing. You can teach these practices to a junior engineer.
• Communication: You can explain to a business stakeholder why architectural decisions matter for their bottom line, and you can debate architectural trade-offs with Software Engineers.

• Autonomy: You have the support of leadership to make architectural decisions. You aren’t fighting for permission to fix tech debt; you are being hired specifically to fix it.
• Impact: Your work directly enables the Data Science team to stop cleaning data and start building better predictive models.
• Modern Stack: We are moving toward a cutting-edge IIoT stack (UNS + Digital Twins), offering you the chance to work on…