Data Engineer; GCP & Big Data

Core Ai Consulting is seeking a highly skilled Data Engineer with strong experience in Google Cloud Platform (GCP) and Big Data technologies to design, build, and optimize scalable data platforms. The ideal candidate will have deep expertise in distributed systems, cloud-native data services, and large-scale data processing to support analytics, reporting, and machine learning initiatives.

• Design, develop, and maintain scalable data pipelines, ETL/ELT workflows
  , and data ingestion frameworks for large and complex datasets.
• Build and optimize big data solutions using technologies such as Spark, Hadoop, Hive, Kafka, and HBase
  .
• Architect and implement cloud-native data platforms on GCP
  , leveraging services such as Big Query, Big Table, Dataflow, Dataproc, Pub/Sub, and Cloud Storage
  .
• Develop and maintain data warehouses and analytical data stores to support reporting, dashboards, and advanced analytics.
• Collaborate with cross-functional teams including data scientists, analysts, and application engineers to enable data-driven use cases and machine learning workflows.
• Apply best practices in distributed computing, data modeling, performance tuning, and query optimization
  .
• Ensure data quality, reliability, security, governance, and compliance with enterprise standards.
• Support both OLTP and OLAP systems
  , ensuring high availability, scalability, and fault tolerance.
• Participate in Agile/Scrum-based development
  , contributing to design reviews, code reviews, and continuous improvement initiatives.

• 5+ years of hands-on experience in data engineering or large-scale application development within the Big Data ecosystem
  .
• Strong experience with Big Data frameworks and tools such as Spark, Hadoop, Hive, Kafka, and HBase
  .
• Proficiency in Python, Java, Scala, or Py Spark , with strong SQL expertise.
• Hands-on experience with GCP data services
  , including Big Query, Big Table, Dataproc, Dataflow
  , and cloud-native ETL patterns.
• Solid understanding of distributed systems, parallel processing, and data processing architectures
  .
• Experience with relational and No
  
  SQL databases
  , including document, column-family, and graph data models.
• Strong knowledge of data warehousing concepts
  , ETL/ELT design, and performance tuning of complex SQL queries.
• Experience working with JSON/XML schemas
  , data modeling tools (e.g.,
  Erwin, ERStudio
  ), and open-source data technologies.
• Familiarity with data visualization and analytics tools such as Tableau, Power BI
  , or similar platforms.
• Experience with software delivery methodologies such as Agile, Scrum, and CI/CD practices
  .
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Data Science, or a related field
  .

• Experience with infrastructure and performance-oriented design
  , including storage optimization and high-availability architectures.
• Exposure to multi-cloud data platforms (AWS or Azure) in addition to GCP.
• Knowledge of security, encryption, data replication, and governance frameworks
  .
• Relevant certifications such as Google Professional Data Engineer
  , AWS Big Data Specialty
  , Databricks
  , or Cloudera
  .
• Strong problem-solving skills and ability to thrive in a fast-paced, collaborative environment