Engineer, Data Engineer

Java Ni Fi

Technical

Skills:

Strong NiFi Expertise:
Deep understanding of NiFi architecture, processors, controllers, and data flow management capabilities. Programming

Languages:

Proficiency in Java, Spring boot & API Microservices. Database and SQL:
Familiarity with databases, SQL, and data manipulation techniques. Big Data Technologies:
Knowledge of big data technologies like Kafka ( Event driven & streaming architecture) Airflow Workflow orchestration

• Design and Implementation:
  Creating and implementing data flow solutions using NiFi, including defining data sources, destinations, and transformation processes.
• Ingestion and Transformation:
  Managing data ingestion from various sources (e.g., files, databases, APIs) and transforming data into desired formats for downstream consumption.
• Routing and Enrichment:
  Configuring NiFi to route data to appropriate destinations and enriching data with additional information (e.g., timestamps, location).
• Scalability and Performance:
  Ensuring data flows are scalable and performant to handle varying data volumes and velocity.
• NiFi Cluster Management:
  Maintaining and improving NiFi clusters, including configuration, optimization, and troubleshooting.
• Microservices Integration:
  Developing and integrating microservices to support data conditioning, format validation, and transformation processes.
• Troubleshooting and Issue Resolution:
  Identifying and resolving issues related to data flows, NiFi configurations, and data integration processes.
• Airflow Expertise:
  Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.
• Workflow Design and Development:
  Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.
• Performance Optimization:
  Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling.

Salary Range: $100,000-$120,000 a year