Software Development Manager, Data Analytics Engines

Job Description

The Open Data Analytics Spark Performance team is looking for a skilled manager to help in our mission to provide the world’s fastest and easiest‑to‑use cloud Big Data processing platform. We develop the AWS Runtime for Spark, which is 100% API compatible with Apache Spark and is designed to run with superior performance and scalability on customer‑managed EC2 clusters, EKS clusters, and in serverless environments.

Our runtime is featured on AWS services including Athena and EMR, which enable AWS customers to run millions of clusters performing large‑scale analytics against vast datasets. In the last three years Open Data Analytics has improved our engines by a factor of five by making changes to the plan optimizer, query runtime, and storage connectors. We have also made significant changes to the compiler to enable enterprise features like fine‑grained access control.

This is a “must‑win” strategic area in a growing and very technical space.

As a manager in this space, you will own speeding customer workloads by shipping query performance optimizations in the AWS Runtime for Apache Spark. Your team will apply and extend our automated statistical analysis tools to ensure that these optimizations can be delivered to all AWS Spark customers and allow them to benefit from them out‑of‑the‑box with no configuration required. This team sped the AWS Runtime for Spark’s 3 TB TPC‑DS performance by 1.7x in 2023 through techniques including modifying Spark’s internal data structure implementations, enhancing Code Gen, and enhancing statistics/building advanced heuristics in Spark Catalyst rules to improve query plans both before and during runtime through Adaptive Query Execution.

You will drive the team to identify new optimization opportunities, prioritize them, and deliver them to demonstrate speedups versus Athena/EMR Spark customer workloads and industry benchmarks.

You will take requirements from the savviest customers in the business, work with our Product Management group to identify and pursue market‑differentiating opportunities to enhance our Spark customers’ experience, and design and deliver these opportunities, solving deep problems with a strong group of Senior and Principal Engineers.

• Prioritize the efforts of a strong engineering team researching and delivering new/enhanced query operator implementation and query plan optimizations
• Ensure AWS customers’ workloads benefit from the optimizations your team delivers
• Drive collaboration between your team, other AWS teams, and the open source community to identify system/workload insight enhancements and collaborate on optimization techniques
• Ensure data consistency and durability while obtaining breakthrough performance and scalability
• Drive team design review processes, ensuring both rapid delivery and a high quality bar
• Lead technical/product discussions with other Software and Product managers
• Interface with a diverse customer base to understand requirements, priorities, and processes
• Lead your team to support and learn from Athena and EMR customers’ Spark usage – with a focus on performance
• Hire, coach, and mentor query optimization SDEs
• Interact and partner with the open source community
• Propose and implement new projects or recommend engine improvements
• Own all operational metrics and support
• Help develop long‑term development and business technology strategies

AWS Utility Computing (UC) provides product innovations—from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (IoT), Platform, and Productivity Apps services in AWS, including support for customers who require specialized security solutions for their cloud services.

Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating—that’s…