Senior Kafka Engineer

Senior Kafka Engineer

Tempe, AZ - On site - W2 Only

Skills:

Kafka, Java or Scala, Kafka pipelines, AWS, Azure, GCP, Cloud, Dev Ops

Experienced Kafka Engineer with expertise in Confluent Kafka, Java/Scala, and distributed systems. Skilled in designing scalable, fault-tolerant Kafka-based data pipelines, troubleshooting messaging issues, and optimizing performance. Strong background in cloud deployments, microservices, and Agile development with an automate-first approach.

• Identify and rectify Kafka messaging issues within justified time.
• Work with the business and IT team to understand business problems and design, implement, and deliver an appropriate solution using Agile methodology across the larger program.
• Work independently to implement solutions on multiple platforms (DEV, QA, UAT, PROD).
• Provide technical direction, guidance, and reviews to other engineers working on the same project.
• Administer distributed Kafka clusters in Dev, QA, UAT, and PROD environments and troubleshoot performance issues.
• Implement and debug subsystems/microservices and components.
• Follow an automate-first/automate-everything philosophy.
• Hands-on in programming languages.

• Deep understanding of Confluent Kafka:
  Thorough knowledge of Kafka concepts like producers, consumers, topics, partitions, brokers, and replication mechanisms.
• Programming language proficiency:
  Primarily Java or Scala, with potential for Python depending on the project.
• System design and architecture:
  Ability to design robust and scalable Kafka-based data pipelines, considering factors like data throughput, fault tolerance, and latency.
• Data management skills:
  Understanding of data serialization formats like JSON, Avro, and Protobuf, and how to manage data schema evolution.
• Kafka Streams API (optional):
  Knowledge of Kafka Streams for real-time data processing within the Kafka ecosystem.
• Monitoring and troubleshooting:
  Familiarity with tools to monitor Kafka cluster health, identify performance bottlenecks, and troubleshoot issues.
• Cloud integration:
  Experience deploying and managing Kafka on cloud platforms like AWS, Azure, or GCP.
• Distributed systems concepts.