Software Engineer, Training & Inference Infrastructure

Join to apply for the Software Engineer, Machine Learning Infrastructure role at DatologyAI

Join to apply for the Software Engineer, Machine Learning Infrastructure role at DatologyAI

Get AI-powered advice on this job and more exclusive features.

About The Company

Companies want to train their own large models on their own data. The current industry standard is to train on a random sample of your data, which is inefficient at best and actively harmful to model quality re is compelling research showing that smarter data selection can train better models faster—we know because we did much of this research. Given the high costs of training, this presents a huge market opportunity.

We founded Datology

AI to translate this research into tools that enable enterprise customers to identify the right data on which to train, resulting in better models for cheaper. Our team has pioneered deep learning data research, built startups, and created tools for enterprise ML. For more details, check out our recent blog posts sharing our high-level results for text models and image-text models.

About The Company

Companies want to train their own large models on their own data. The current industry standard is to train on a random sample of your data, which is inefficient at best and actively harmful to model quality re is compelling research showing that smarter data selection can train better models faster—we know because we did much of this research. Given the high costs of training, this presents a huge market opportunity.

We founded Datology

AI to translate this research into tools that enable enterprise customers to identify the right data on which to train, resulting in better models for cheaper. Our team has pioneered deep learning data research, built startups, and created tools for enterprise ML. For more details, check out our recent blog posts sharing our high-level results for text models and image-text models.

We've raised over $57M in funding from top investors like Radical Ventures, Amplify Partners, Felicis, Microsoft, Amazon, and notable angels like Jeff Dean, Geoff Hinton, Yann LeCun and Elad Gil. We're rapidly scaling our team and computing resources to revolutionize data curation across modalities.

This role is based in Redwood City, CA. We are in office 4 days a week.

About

The Role

We’re looking for seasoned ML Infrastructure engineers with experience designing, building, and maintaining training infrastructure for our in-house ML research and validation efforts and the core infrastructure for running the curation pipeline that we deliver to our customers. As one of our early senior hires, you will partner closely with our founders on the direction of our product and drive business-critical technical decisions.

You will contribute to developing core infrastructure components that impact our ability to deliver, scale, and deploy our product. These are key components of our stack that allow us to process customer data and apply state-of-the-art research to identify the most informative data points in large-scale datasets. You will have a broad impact on the technology, product, and our company's culture.

What You'll Work On

• Architect, build and maintain the infrastructure that ensures highly available GPU workloads for training-purposes
• Troubleshoot and resolve issues across GPU resources, networking, OS, drivers, and cloud environments, automate detection and recovery of such issues
• Design, build, and maintain the infrastructure that powers our data curation product.
• Partner with researchers and engineers to bring new features and research capabilities to our customers
• Ensure that our infrastructure and systems are reliable, secure, and worthy of our customers' trust.

• 5+ years of experience
• Have meaningful experience with leading and building production ML infrastructure and platforms that deliver on major product initiatives.
• Proficiency in Python and in the most commonly used tools in the infrastructure space:
  Linux, Kubernetes, Terraform / Pulumi, etc
• Strong knowledge of hardening cloud native and especially K8s workloads.
• Experience maintaining a high-quality bar for design, correctness, and testing.
• Have a humble attitude, an eagerness to help your colleagues, and a desire to do whatever it takes to make the team succeed
• Own problems end-to-end and are willing to pick up whatever knowledge you're missing to get the job done.
• Experience running data-processing workloads in k8s (e.g spark on k8s)

• The candidate's starting pay will be determined based on job-related skills, experience, qualifications, and interview performance.