Senior Data Quality Engineer

Overview

Job Purpose:

The Chief Directorate Department at Emirates Airlines oversees strategic planning, operational excellence, and innovation across various divisions. It plays a key role in driving the airline's growth, enhancing customer experience, and ensuring seamless global operations.

Job Description:

The Chief Directorate Department at Emirates Airlines oversees strategic planning, operational excellence, and innovation across various divisions. It plays a key role in driving the airline's growth, enhancing customer experience, and ensuring seamless global operations. As a Senior Data Quality & Automation Engineer you will be a fully participating member of a cross-functional team working autonomously on technology development and problem resolution in the Enterprise Data & Analytics space.

The role involves hands-on contribution to quality practices along with designing and implementing data quality and automation platforms. You will also provide support to technical analytics solutions and products that support Emirates Airlines and the Emirates Group businesses.

• Work closely with product owners, analysts, software engineers and architects to understand the technical landscape and context of deliveries, refine complex functional and non-functional requirements, and translate them into fit-for-purpose acceptance tests.
• Address problems of diverse scope where analysis of data requires evaluation of identifiable factors and selection of methods to validate and automate the solution.
• Build test strategies and test plans validating the core business problem and translating them into tests around code functionality, data quality, performance, and security.
• Build and support generating or mocking of test data for exploratory analysis and running tests.
• Design and build automated tests/jobs of moderate to high scope and complexity across all stages of the data pipeline, following coding standards and best practices.
• Build and enhance data quality rules and platforms for observability across the data pipeline.
• Debug complex issues, resolve blockers and follow design documents with minimal or no supervision.
• Conduct data analysis activities such as source system analysis, data modelling, data dictionary collection, data profiling and source-to-target mapping to ensure delivery on business needs.
• Support in updating data inventories and registries to keep metadata and data lineage up-to-date in line with Data Governance standards, guidelines and principles.

• Degree or Honours (12+3 or equivalent) in Computer Science, Computational Mathematics, Computer Engineering or Software Engineering.
• Experience 2+ years in Data Engineering with a focus on quality & automation.
• Minimum 2+ years of testing, automation and support experience in analytics applications such as Data Lake and Data Warehouse (preferably using the Big Data stack and Microsoft Azure cloud infrastructure).
• Seasoned in identifying quality issues across complex data pipelines on big data technologies and defining rules for validating data health.
• Experience with a wide variety of testing methods and tools covering functional, performance, and security tests across jobs, pipelines and end-to-end across enterprise.
• Well-versed in building automated checks in CI pipelines to validate ETL/ELT jobs are performing as expected.
• Experience with batch and real-time data ingestion/integration tools handling large data volumes (structured and unstructured).
• Exposure to data architecture concepts such as data modelling, big data storage, and dimensional modelling.
• Exposure to working with jobs in data pipelines and defining metrics/measures to ensure data correctness.
• Programming (Python or Scala) and SQL querying skills required.

• Ability to drive quality of data assets independently.
• Ability to deliver solutions and value interactively.
• Strong data analysis skills (e.g., source system identification, data dictionary/metadata collection, data profiling, source-to-target mapping).
• Operates with a You Code It, You Own It mindset (supports the products they build).
• Team player; able to collaborate to remove blockers, solve design problems and debug issues.
• Accountable with a positive attitude.
• Self-starter with passion for learning new technologies in Enterprise Data & Analytics space.
• Exposure to Spark and airline industry experience is nice-to-have.
• Specialization in Data & Analytics (e.g., Data Warehousing, Data Science, BI) is a bonus.

• Big Data & distributed processing:
  Spark, Hadoop (HDFS, Hive, HBase, Oozie), Airflow, Apache NiFi, Azure (ADLS, Databricks, Azure Data Factory), Elasticsearch, AVRO/PARQUET.
• Data Analysis, Modelling and Reporting:
  Snowflake, SQL, Data Vault 2.0, Micro Strategy, Power BI.
• Cloud Technologies:
  Microsoft Azure and Cloudera stacks.
• Integration and Messaging:
  Streaming (Spark Streaming), Snap Logic, TIBCO, Kafka.
• CI/CD:
  Git, Bitbucket, Jenkins, Azure Dev Ops, Kubernetes, Docker, Sonar Qube,…