Senior Site Reliability Engineer — Dallas, TX

The Instant Financial Issuance as a Service (IFIaaS) Cloud Service includes a wide array of components including web services, application servers, and databases hosted in an on-prem environment. The Sr. Site Reliability Engineer (SRE) will be responsible for ensuring that the SaaS platform is reliable, available, and performant, as well as scalable, secure, and cost-effective. Ultimately, the individual will be responsible for the platform uptime, functional management of all the IFIaaS cloud environments, applications, networks, scoping projects, and the resolution of application and network issues.

The Instant Financial Issuance as a Service (IFIaaS) Cloud Platform spans multiple on‑prem environments. The Senior Site Reliability Engineer (SRE) will play a critical role in ensuring the platform’s reliability, scalability, security, and operational excellence across these geographically distributed environments. Given the asymmetric nature of our data centers, the SRE will design and operate systems that prioritize local HA while ensuring effective, tested, and compliant failover for DR scenarios.

This role includes responsibility for platform uptime, environment management, network and application reliability, observability, automation maturity, compliance, and operational excellence.

• Own SLOs/SLIs for availability (99.9%), latency, error rate, and quality of service across microservices.
• Design/operate end‑to‑end observability: metrics, logs, traces, synthetic checks, real‑user monitoring (RUM).
• Instrument services (Windows services, APIs, background jobs) with structured logs and trace context.
• Build health probes and SLA monitors for critical transactions and cross-service dependencies.
• Monitor system issues using various metrics, such as uptime, latency, error rate, throughput, and availability
• Deploy and maintain monitoring and on‑call tools i.e.:
  Splunk on‑call, Prometheus, Datadog, etc.
• Lead incident response (triage, comms, coordination, real‑time mitigation) and conduct blameless postmortems with actionable follow‑ups.
• Maintain and continuously improve runbooks, escalation paths, on call rotations, and paging policies.
• Implement MTTA/MTTR reduction programs.
• Stand up war room protocols and ensure stakeholder updates during incidents.
• Forecast compute, storage, network needs, track headroom against growth and peak patterns.
• Conduct performance profiling and bottleneck analyses (CPU, memory, I/O, thread pools, connection pools).
• Optimize resource allocation on VMware (DRS, affinity rules, reservations) and Windows VM tuning (kernel, TCP stack, NICs).
• Validate scaling strategies (horizontal vs. vertical) and implement auto‑scaling where supported.
• Standardize gold images, configuration baselines, and desired state for Windows Server (Power Shell DSC or equivalent).
• Manage patching (OS, middleware, runtime) with maintenance windows aligned to error budgets.
• Ensure backup, snapshot, and restore strategies meet RPO/RTO; regularly test restores.
• Maintain secure baselines (CIS benchmarks for Windows/VMware), vulnerability management, and patch cadence.
• Support compliance audits (PCI‑CP, PCI‑DSS, SOC 2/ISO 27001), produce evidence (configs, logs, access reviews), and remediate gaps.
• Automate provisioning (VM templates, DSC/Ansible for Windows, Terraform for VMware) and configuration drift detection/correction.
• Build runbooks to reduce toil (deploy, scale, rollback, etc)
• Create reliability guardrails (pre‑flight checks, change freeze rules, policy controls) as code.
• Continuously refactor scripts/runbooks into idempotent automation.
• Collaborate with development teams and other stakeholders to identify potential risks, such as security vulnerabilities, performance bottlenecks, deployment issues, or configuration errors
• implement various risk mitigation strategies, such as patching, backup, redundancy, encryption, or testing
• Collaborate with product teams and other teams to understand the user needs, expectations, and satisfaction.
• Coach engineers on SRE principles, incident handling, and reliability centric design.
• Lead knowledge sharing, runbooks quality, and postmortem…