Senior Splunk Infrastructure Engineer

Senior Splunk Infrastructure Engineer

Duration: 12 Months

The Credit Karma Observability team is looking for a Senior Splunk Infrastructure Engineer to assist in managing our large-scale logging and monitoring platforms. We operate multiple production Splunk clusters and a corporate cluster that serve as the backbone for our engineering troubleshooting and security compliance.

In this role, you will work closely with our Observability engineers to maintain, upgrade, and optimize our Splunk infrastructure running on Google Cloud Platform (GCP). You will rely heavily on Salt Stack for configuration management and must possess deep Unix/Linux expertise to troubleshoot performance at the OS level. Candidates should note that our core Splunk infrastructure (Indexers, Search Heads) runs primarily on Virtual Machines, while only our Forwarders are deployed within Kubernetes environments.

• Splunk Administration:
  Manage the health, performance, and stability of multiple Splunk clusters (Search Head Clusters, Indexer Clusters, and Heavy Forwarders).
• Splunk Enterprise Security (ES) Support: maintain and support the underlying infrastructure for Splunk ES, ensuring optimal performance for security operations.
• Heavy Forwarder & Pipeline Management:
  Manage applications, parsing rules, and data pipelines on Heavy Forwarders to ensure efficient data ingestion and routing.
• Reliability & HA/DR:
  Design and maintain High Availability (HA) and Disaster Recovery (DR) strategies to ensure business continuity and platform resilience across regions.
• Safe Configuration Management:
  Champion proactive engineering practices by implementing safe deployment strategies for Salt Stack configurations, including canary testing, validation, and staged rollouts to minimize production incidents.
• Infrastructure as Code:
  Maintain and write complex Salt Stack states and formulas to manage Splunk configurations and underlying Linux VMs.
• GCP Operations:
  Provision, monitor, and scale infrastructure within Google Cloud Platform.
• System Optimization:
  Perform deep-dive troubleshooting on Linux systems (kernel tuning, disk I/O, memory management) to ensure optimal Splunk performance.
• On-Call Support:
  Participate in the on-call rotation to respond to critical incidents affecting Splunk infrastructure availability and performance, ensuring 24/7 reliability.
• Maintenance & Upgrades:
  Execute distinct maintenance windows, version upgrades, and patching cycles.
• Documentation:
  Update runbooks and technical documentation within our repository to ensure knowledge sharing, specifically focusing on configuration changes and pipeline architecture.

• Splunk Expertise: 5+ years of experience administering large-scale Splunk Enterprise environments. Proven experience with Indexer Clustering and Search Head Clustering.
• Advanced SPL (Splunk Processing Language):
  Proficiency in writing, debugging, and optimizing complex SPL queries to support dashboarding, alerting, and data analysis.
• Splunk Enterprise Security (ES):
  Hands-on experience administering and maintaining Splunk ES environments.
• Configuration Management:
  Strong proficiency with Salt Stack (Salt). You must be comfortable writing custom states and managing configurations for thousands of nodes.
• Unix/Linux Internals:
  Deep understanding of Linux administration (RHEL/CentOS/Ubuntu). Ability to debug resource contention, file system issues, and network bottlenecks.
• Cloud Infrastructure:
  Hands-on experience with Google Cloud Platform (GCP), specifically GCE and networking.
• Scripting:
  Proficiency in Python or Bash for automation tasks.

• Terraform & Infrastructure Workflow:
  Experience using Terraform for infrastructure provisioning, combined with Salt Stack for configuration management. Ideal candidates understand the distinction and interaction between provisioning (Terraform) and configuration (Salt).
• Kubernetes & Git Ops:
  Experience with Helm and Flux for managing Kubernetes resources, utilizing Git Ops methodologies to drive consistent and automated deployments.
• "Done For You" / Platform Engineering:
  Experience building "Done for you" solutions or paved paths that abstract infrastructure complexity for internal customers, enabling self-service and standardization.
• Multi-Cloud Architecture:
  Experience managing infrastructure across multiple cloud providers (e.g., GCP, AWS, Azure), understanding the nuances of hybrid networking and data locality.
• Open Telemetry (OTEL):
  Knowledge of migration strategies from Splunk Universal Forwarders to OTEL Collectors to standardize data formats.
• Experience working in a highly regulated fintech environment.

Email:
sh