NBBL- Lead Innovation

About the Role

We are seeking a highly skilled Lead Data Engineer to design and implement the data foundation for the Innovation vertical within BBPS. This role focuses on building scalable, high-performance data pipelines and data products that enable AI/ML-driven solutions and real-time analytics for BBPS and the broader ecosystem.

Responsibilities

- Pipeline Architecture:
Design, build and optimize high-scale data pipelines (Batch & Streaming) using Spark/PySpark, Kafka, and Airflow/similar on prem tools.
- Ecosystem

Innovation: Develop data products that provide real-time insights to entire ecosystem such as transaction success-rate predictors and automated reconciliation engines and so on.
- Data Lakehouse Management:
Build and maintain a scalable Data Lakehouse to consolidate disparate data formats from across the ecosystem into a unified source of truth.
- Orchestration:
Utilize enterprise level schedulers like Apache Airflow/Dagster and to create models using build tools like DBT, to automate complex workflows and ensure data quality validation across the pipeline.
- Performance Engineering:
Optimize complex SQL queries and data storage formats to ensure sub-second latency for innovation-led dashboards.
- Security & Compliance:
Ensure all data engineering practices strictly adhere to NPCI/RBI guidelines, including PII masking and data encryption.

Preferred skills and

Qualifications:

- Bachelor’s/master’s degree in computer science, Data Science or related field.
- 6+ years of experience in software development, with at least 2 years in AI/ML projects.
- Certifications in Data Engineering/ AI/ML technologies are a plus.
- Proficiency in Python and PySpark with strong experience in Spark using Java or Scala for advanced data engineering and scalable pipeline development.
- Hands-on experience with AI/ML frameworks (Tensor Flow, PyTorch, Scikit-learn).
- Good Knowledge on building and managing Data pipelines.
- Streaming:
Proven track requires real-time processing using Kafka or Flink.
- Orchestration:
Hands-on experience with Airflow or similar workflow tools.
- Knowledge of vector databases and RAG pipelines.

- Experience with containerization (Docker) and orchestration (Kubernetes) is a big plus.
- Familiarity with open-source AI tools and libraries.
- Good understanding of security and compliance for enterprise applications.
- Excellent problem-solving and debugging skills.
- Ability to work collaboratively in agile teams and aid in building auto scalable pipelines.

Equal Opportunity Statement:

As an Equal Opportunity Employer, NBBL fosters a workplace culture that values diversity and inclusion. It maintains zero tolerance for discrimination based on race, ethnicity, disability, gender identity, or sexual orientation, and actively supports the LGBTQ+ community.