Senior Data Engineer

Location:

501 Fifth Avenue, Suite 805, New York, NY 10017

We are looking for an experienced Senior Data Engineer to join our team to work on an exciting data platform project leveraging Microsoft Fabric. This role will involve collaborating closely with our Solutions Team and the University’s Technology teams in an agile environment. The ideal candidates will be responsible for building scalable data platforms, onboarding data from multiple source systems, and driving end-to-end data solutions.

As a Senior Data Engineer, you will be key to implementing Metadata Driven ingestion framework along with Dev Ops Accelerator, ensuring rapid, cost-effective deployment.

• Expertise in building and managing Microsoft Fabric-based data platforms.
• Hands‑on experience with data ingestion from multiple source systems (AWS, Azure, direct pipelines).
• Strong knowledge of data transformation and modeling (Bronze, Silver, Gold layers) with a focus on performance and scalability.
• Proven ability to work with Dev Ops tools and practices (CI/CD pipelines, version control, etc.).
• Experience deploying and managing Microsoft Purview or other data lineage and cataloging tools.
• Familiarity with AI‑driven analytics solutions, particularly for detecting patterns and behaviors in student data.
• Strong collaboration skills, working in agile teams to align with both business and Dev Ops practices.

• Implement Metadata Driven Pipeline & Dev Ops Accelerator to streamline the data platform implementation, using pre‑developed code and best practices.
• Create a data model for enabling advanced analytics and AI use‑cases to detect at‑risk students and visualise student behaviour.
• Ingest data from existing and new source platforms, such as AWS, Azure, or direct pipelines, into Microsoft Fabric.
• Setup Layered Data Architecture and transform and model data from Bronze, Silver, and Gold layers, ensuring alignment with CAUDIT Data Reference Model and defined data domains.
• Apply Dev Ops practices and tools for consistent, scalable, and repeatable data platform solutions.
• Deploy and manage Microsoft Purview for data lineage, cataloging, and governance, ensuring adherence to data governance controls.

Australian Shift