Principal Software Engineer, Data Platform

Principal Software Engineer, Data Platform

5 days ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

$/yr - $/yr

We are seeking a Principal Software Engineer to lead the architecture and evolution of our data and machine learning infrastructure. This role will shape the foundation on which data‑driven products, analytics, and AI applications are built. You will design systems that enable large‑scale data processing, reliable pipelines, and efficient machine learning development—from feature engineering to real‑time model serving.

As a principal engineer, you will partner with product, data science, and platform teams to set technical direction, drive adoption of reusable frameworks, and mentor engineers across the organization. You will ensure that both data and ML platforms are scalable, reliable, cost‑efficient, and compliant with privacy and governance standards.

The core of the Data Platform is a data lake on AWS S3 with Apache Iceberg as the table format to ensure reliability. Data ingestion is standardized through Confluent Kafka for real‑time streaming and Fivetran for ingestion of files and change‑data. The transformation layer is decoupled from storage, using Apache Flink for stream processing, AWS Glue (Spark) for core ETL, and dbt/Athena for building analytical data models.

The platform serves data through fit‑for‑purpose data stores, including Amazon Dynamo

DB for low‑latency applications and Google Big Query as the primary engine for analytics and BI.

This is a hybrid role based in our New York City headquarters, reporting to the Sr. Director of Engineering. You can typically expect to come into the office 2+ days per week.

• Architect & Build Platform:
  Design and evolve infrastructure for data ingestion, storage, batch and streaming pipelines, and machine learning workflows.
• Enable ML at Scale:
  Build systems for training, deploying, monitoring, and governing models, including feature stores, registries, and inference platforms.
• Reliability & Observability:
  Ensure end‑to‑end system reliability, monitoring, and cost transparency across data and ML workloads.
• Self‑Service Platforms:
  Deliver frameworks and APIs that enable engineers, analysts, and ML scientists to build and operate solutions independently.
• Innovation & Standards:
  Evaluate and introduce emerging technologies (vector databases, distributed training, orchestration frameworks, LLM stacks) and establish adoption guidelines.
• Cross‑Functional Leadership:
  Partner with platform, product, and engineering and ML science leaders to align on strategy and accelerate delivery.
• Mentorship & Influence:
  Guide senior and staff engineers, lead architecture reviews, and raise the technical bar across data and ML domains.
• Demonstrate support and understanding of our value of journalistic independence and a strong commitment to our mission to seek the truth and help people understand the world.

• 10+ years of software engineering experience with a focus on distributed systems, data platforms, and ML infrastructure or equivalent.
• Proven ability to influence technical direction across multiple teams and mentor senior/staff engineers.
• Proven expertise in data processing frameworks and table formats (e.g., Spark, Flink, Iceberg) and orchestration tools (e.g., Airflow, Kubeflow).
• Deep knowledge of ML infrastructure: model training pipelines, feature stores, registries, serving, and monitoring.
• Strong programming skills in Python and at least one compiled language like Java or Go.
• Experience designing systems with scalability,…