BigData Developer

JD:
Technical/Functional

Skills:

Big Data / Hadoop, SPARK, Python, Strong SQL, GCP

• Develop and maintain data pipelines using Big Data processes
• Focus on ingesting, storing, processing, and analyzing large datasets
• Work closely with data science team to implement data analytics pipelines
• Help define data governance policies and support data‑versioning processes
• Maintain security and data privacy, working closely with data protection officer
• Analyze vast number of data stores to uncover insights
• Enhancing data collection procedures to include all relevant information for developing analytic systems
• Processing, cleansing, and validating the integrity of data to be used for analysis
• Analyzing large amounts of information to find patterns and solutions
• Developing prediction systems and machine learning algorithms
• Presenting results in a clear manner
• Propose solutions and strategies to tackle business challenges
• Collaborate with Business and IT teams

• Bachelor Degree in Engineering or Computer Science or equivalent OR Master in Computer Applications or equivalent
• 6+ years of software development experience and leading teams of engineers and scrum teams
• 5+ years of hands‑on experience working with Informatica, Oracle
• Solid Data warehousing concepts. Knowledge of Financial reporting ecosystem will be a plus
• 5+ years of experience within Data Engineering/Data Warehousing using Big Data technologies will be an addon
• Expert on Distributed ecosystem
• Hands‑on experience with programming using Core Java or Python/Scala
• Expert on Hadoop and Spark Architecture and its working principle
• Hands‑on experience on writing and understanding complex SQL (Hive/PySpark‑data frames), optimizing joins while processing huge amount of data
• Experience in UNIX shell scripting. Ability to design and develop optimized Data pipelines for batch and real time data processing
• Should have experience in analysis, design, development, testing, and implementation of system applications
• Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows