Astro​/Airflow Engineer

Job Description Must Have Technical/Functional Skills

• 5-8+ years building/operating data or platform systems; 3+ years running Airflow in production at scale (hundreds-thousands of DAGs and high task throughput)
• Deep Airflow expertise: DAG design and testing, idempotency, deferrable operators/sensors, dynamic task mapping, task groups, datasets, pools/queues, SLAs, retries/backfills, cross-DAG dependencies
• Strong Kubernetes experience running Airflow and supporting services:
  Helm, autoscaling, node/pod tuning, topology spread, network policies, PDBs, and blue/green or canary strategies
• Observability and SRE practices:
  Prometheus/Grafana/Stats
  
  D, centralized logging, alert design, capacity/throughput modeling, performance tuning
• Security/compliance: SSO/OIDC, RBAC, secrets management (Vault/Secrets Manager), auditing, least-privilege connection management, and change control
• Proven incident leadership, runbook creation, and platform roadmap execution; excellent cross-functional communication
• Experience operating and leading migrations to/from Airflow
• Open Lineage/Marquez adoption;
  Great Expectations or other data quality frameworks; data contracts
• Cost optimization and capacity planning for schedulers and workers; spot instance strategies
• Multi-region HA/DR for Airflow metadata DB; backup/restore and disaster drills
• Building internal developer platforms/portals (e.g., Backstage) for self-service pipelines
• Contributions to Apache Airflow or provider packages; familiarity with recent AlPs/ Airflow 2.7+ features
• Architect, deploy, and operate production-grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning
• Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedulers
• Build automation infrastructure:
  Terraform modules and Helm charts with Git Ops-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
• Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks
• Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
• Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data aware scheduling
• Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high-throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
• Partner with teams across the organization to provide enablement, documentation, and self-service tooling

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.