Data Engineer

About the role

We are looking for a skilled Data Engineer to design, build, and maintain scalable data pipelines and platforms that support our geospatial traffic analytics applications. The ideal candidate will have experience with Python, Databricks, S3, and modern data engineering practices, including automated testing, CI/CD, and data quality monitoring.

• Design, implement, and maintain scalable data pipelines and ETL/ELT workflows on Databricks and cloud platforms.
• Manage large-scale geospatial and temporal datasets stored in AWS S3.
• Collaborate with data scientists to product ionize machine learning models and ensure smooth data availability.
• Implement data validation, testing, and monitoring frameworks to ensure data accuracy, consistency, and reliability.
• Optimize data storage and processing strategies to handle high volumes of traffic and mobility data efficiently.
• Develop and maintain documentation for data workflows, architecture, and processes.
• Work closely with cross-functional teams to understand data requirements and ensure timely delivery.
• Stay up-to-date with the latest trends and best practices in data engineering, cloud technologies, and big data processing.

• Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
• 3+ years of experience as a data engineer or in a similar role.
• Strong proficiency in Python and associated libraries for data engineering (pandas, PySpark, etc.).
• Hands-on experience with Databricks and Spark for large-scale data processing.
• Experience with AWS services, especially S3, and knowledge of cloud-based data architectures.
• Solid understanding of data pipeline testing, version control, and CI/CD practices.
• Experience with SQL and No
  
  SQL databases.