HPC Engineer - Storage

Role

Title:

HPC Engineer – Storage

Location: India (Must align with Client Time Zone)

Employment Type: Full-Time

The HPC Engineer - Storage acts as the primary execution engine for the data persistence layer that feeds the AI Factory. While the Domain Architect designs the tiering strategy and namespace layout, you are the "Builder" responsible for mounting file systems, tuning client‑side I/O, and ensuring data reaches the GPUs at line rate. You are a "doer" who is as comfortable compiling a custom kernel module for a parallel file system as you are debugging a stale NFS handle.

As a System Integrator, we design and deliver bespoke, high‑scale AI factories. In this role, you will move beyond standard enterprise NAS management to execute the deployment of ultra‑high‑performance parallel file systems for NVIDIA Super

POD, NVIDIA BasePOD, and Cisco AI Factory environments.

In this role, you will operate with a 100% focus on Delivery
, executing the Low‑Level Designs (LLD) assigned by your Squad Lead. You will own the "Storage" leg of the critical "Compute-Network-Storage" triad, ensuring that "I/O Wait" never becomes the bottleneck for training jobs.

CRITICAL REQUIREMENT: This role typically operates on Shift Hours to align with the onshore client's time zone (e.g., early shifts for Australian clients, or split shifts for European clients).

• Client Provisioning: Execute the deployment of high-performance storage clients (VAST, Weka, GPFS/Spectrum Scale, Lustre) on bare‑metal DGX/HGX nodes using Ansible.

• Protocol Configuration: Configure and tune RDMA-based protocols (NVMe-oF, NFS over RDMA, GPUDirect Storage) to bypass the CPU and deliver data directly to GPU memory.

• Kubernetes Integration: Install and troubleshoot CSI (Container Storage Interface) drivers to ensure dynamic provisioning of Persistent Volumes (PVs) for AI workloads running in K8s.

• Mount Management: Manage complex mount maps and automounter configurations to ensure consistent namespace views across thousands of compute nodes.

• Throughput Testing: Execute standard I/O benchmarks to validate that the storage subsystem meets the "Gold Standard" read/write targets (e.g., 400GB/s read throughput).

• Latency Tuning: Tune client‑side kernel parameters (read‑ahead buffers, queue depths, sysctl settings) to minimize latency for small‑file random I/O patterns common in checkpointing.

• Acceptance Reporting: Generate "As‑Built" storage validation reports, documenting effective throughput and IOPS for client sign‑off.

• Capacity & Quotas: Implement project‑level quotas and monitor usage trends to prevent "Disk Full" outages on critical scratch file systems.

• Ticket Resolution: Handle L2 support tickets for storage issues, such as "Stale file handles," "Slow dataset loading," or "CSI Driver crashes."

• Lifecycle Management: Execute non‑disruptive client‑side driver upgrades and firmware patches during maintenance windows.

• Parallel File systems: Hands‑on operational experience with at least one major AI storage platform:
  VAST Data, Weka.io, DDN Lustre (Exascaler), or IBM GPFS (Spectrum Scale).

• Linux I/O Stack: Deep understanding of the Linux VFS (Virtual File System), block devices, and how to debug I/O performance using tools like iostat, iotop, and strace.

• RDMA Storage: Experience configuring NVMe-over-Fabrics (NVMe-oF) or NFS-over-RDMA, understanding the dependency on the underlying Infini Band/RoCE network.

• Ansible Storage: Proficiency in writing Ansible playbooks to automate the installation of storage clients and configuration of mount points.

• Kubernetes Storage: Understanding of Storage Classes
  , PVCs
  , and how to debug CSI Driver pods (checking logs for mount failures).

• GPUDirect: Conceptual understanding of NVIDIA GPUDirect Storage (GDS) and the ability to verify if GDS is active.

• Vendor Specifics: Deep certification or experience with Pure Storage (Flash Blade) or Net App ONTAP AI configurations.

• Object Storage: Experience…