Principal Engineer, Behavior Understanding and Evaluation

Mission

Summary:

The AV Behavior Design and Evaluation Team is responsible for owning the design and assessment of the AV behavior utilizing the concept of AV behavior competencies. The team relies on systematic decomposition of AV behavior along with data driven assessment utilizing data from on-road and simulation test modalities. We are responsible for building and operating robust, large-scale evaluation systems, alongside sophisticated simulation workflows and insightful diagnostic tools.

By combining expert human judgment with advanced data driven techniques, we evaluate the system behavior performance.

As the team is looking to scale to support more and more behavioral competencies, we are looking for passionate Engineers to help build a robust framework to leverage the best of agile systems engineering and data driven performance assessment to achieve our ambitious goals in scaling our system design and evaluation processes. We are looking to work with people who are driven to design robust systems and transform complex data into measurable progress toward a safer, smarter transportation future.

This person will get to build relationships with the team as well cross-functional across organization to help build a cohesive understanding AV performance as applicable to the Behavioral Competencies.

What you'll do:

* Decompose AV operational design domain (ODD) into AV capabilities

* Define precise atomic system and subsystem behaviors and interfaces required to realize these capabilities.

* Collaborate extensively with development teams (Autonomy, Infrastructure, Cloud Applications) to drive the design and implementation of new features that enable defined AV capabilities and enhance the overall product experience.

* Strategically plan and execute system-level testing campaigns. This includes leveraging advanced simulation tools and coordinating tests on closed course tracks and proving grounds to ensure the system meets functional and performance requirements.

* Effectively utilize existing metrics and propose new metrics to evaluate AV performance via simulation and on-road testing modalities.

* Pioneer the definition and implementation of key evaluation metrics and reward functions, driving autonomy performance across safety, comfort, and assertiveness

* Leverage a combination of learned methods and metrics along with classical metrics to stretch the horizon of performance evaluation

* Characterize the performance target against parameters used to define AV capabilities to provide nuance in performance evaluation

* Deliver strategic insights and recommendations based on AV capability testing by building meaningful visualizations

* Develop and maintain comprehensive dashboards and reporting tools. Distill on-road performance data into actionable feedback for engineering teams and create high-level, insightful summaries for executive leadership.

* Focus on automating simulation result analysis processes and gaps, removing manual processes where applicable

* Partner cross-functionally with Behaviors, Actions, Simulation, System Engineering, Research, and other Onboard/Offboard Software Teams on evaluation strategies

What we're looking for:

* Bachelor's degree and 7+ years of experience in one of Systems Eng, Data Science, Statistics or Computer Science with some form of experience in rest of the domains

* At least 5 years of experience in leading and scaling data driven evaluation of autonomous driving behavior or similar safety critical emerging technology

* Proven technical experience in establishing and executing evaluation strategies

* A strong foundation in scientific and statistical methodologies

* Fluency in SQL or similar query languages

* Fluency in data visualization techniques

* Fluency in Python

* Proficiency in crafting data-driven reports and technical strategy documents for executive decision-making

* Excellent interpersonal skills to build relationships with other teams

* Excellent communication skills; effective in communicating complex concepts and driving resolutions

* Self-starter who is motivated to set and achieve goals and takes initiative to drive ambiguous tasks/projects to completion

Ideally, you also have:

* Masters degree in Data Science or Computer Science

* Experience in mentoring and leadership development

* Experience collaborating in an agile development environment

* A background in shaping product roadmaps

* Familiarity with Reinforcement Learning (RL) with Human Feedback

* Specialized expertise in creating expansive datasets

* Experience collaborating in an agile development environment

* Working with a global workforce that spans multiple timezones

We encourage a hybrid schedule with in-office time at one of our locations in Boston, Pittsburgh, or Las Vegas to support collaboration, or this role can be fully remote.