Senior Software Engineer - Compute

Software Engineering Mountain View, California

Who we are

Aurora’s mission is to deliver the benefits of self-driving technology safely, quickly, and broadly.

The Aurora Driver will create a new era in mobility and logistics, one that will bring a safer, more efficient, and more accessible future to everyone.

At Aurora, you will tackle massively complex problems alongside other passionate, intelligent individuals, growing as an expert while expanding your knowledge. For the latest news from Aurora, visit aurora.tech or follow us onLinked

In .

Aurora hires talented people with diverse backgrounds who are ready to help build a transportation ecosystem that will make our roads safer, get crucial goods where they need to go, and make mobility more efficient and accessible for all.

The journey to developing our cutting-edge self-driving truck Driver software requires a massive and continuous computational effort. Every day, Aurora's engineers initiate and manage millions of compute tasks, systematically processing and analyzing petabytes of critical data. This colossal computational workload spans various essential domains, including raw and derived data processing pipelines, large-scale simulations to test and validate the software in countless scenarios, and the intensive machine learning training jobs that are the core of our autonomous system's intelligence.

At the heart of orchestrating this monumental compute infrastructure is the Compute team. This team is dedicated to building and maintaining the foundational technology that solves the fundamental challenges of resource scheduling, task isolation, and distributed state consistency across our massive batch compute fleet. At our scale, traditional off-the-shelf orchestrators break. The Compute team builds the custom engine - Batch

API - that manages the lifecycle of millions of tasks - built on top of K8s primitives but implements our own custom scheduler.

We deal with the 'unsolved' problems of distributed computing: maximizing hardware utilization while ensuring that a failure in one node doesn't cascade across the entire cluster. This engine is engineered to handle massive scale, ensuring reliability, efficiency, and rapid turnaround for our engineers.

Furthermore, the Compute team empowers engineers across the company to effectively harness this compute power. They develop and maintain the Batch Workflows Python SDK. This framework provides an intuitive, high-level interface that allows engineers to programmatically define, construct, deploy, monitor, and manage their complex computational workloads. The SDK abstracts away the complexities of the underlying infrastructure, enabling engineers to focus purely on the logic and goals of their data processing, simulation, or training tasks, thus accelerating the entire development cycle for the Aurora Driver.

In this role you will

• Design, implement, and maintain core components of the high-performance, large-scale distributed batch compute engine (Batch
  
  API). Architect and optimize the scheduler, resource allocator, and execution engine of Batch
  
  API to handle bursty, heterogeneous workloads with minimal overhead.
• Design low-latency APIs and resilient communication protocols that bridge our Python SDK with the Golang-based core engine.
• Develop high-level workflow abstractions, enabling engineers across the company to programmatically define, deploy, and manage complex data processing, simulation, and ML training pipelines.
• Solve complex problems in distributed locking, throttling, and fair-share scheduling to ensure multi-tenant stability.
• Drive continuous improvements in the performance, scalability, and resilience of the entire compute infrastructure, implementing robust monitoring and alerting systems to maintain operational excellence for critical workflows.
• Collaborate closely with infrastructure and product engineering teams (e.g., Autonomy, Data, Simulation, Machine Learning) to gather requirements, provide expert consultation, and integrate compute workflows with key company systems.

Required qualifications

• 5+ years of professional software engineering experience.
• Deep…