Data Engineer W2

Location:
Onsite in Oakbrook Terrace, IL

Our Client is seeking a mid-level Data Engineer is responsible for designing, building, and maintaining the data infrastructure and pipelines that enable efficient data processing, storage, and analysis within a hybrid computing environment. This role bridges the gap between data collection and data consumption across both high-performance computing (HPC) servers and Azure cloud platforms, ensuring that data is accessible, reliable, and optimized for analytics, machine learning, and business intelligence purposes.

• Design, develop, and implement scalable data pipelines using modern ETL/ELT frameworks and tools in hybrid environments
• Create and maintain data architectures that span high-performance computing servers and Azure cloud services
• Build robust data integration solutions to connect various data sources across on-premises and cloud systems
• Configure and optimize databases and data stores in both UNIX/Linux environments and cloud platforms
• Implement MLOps practices to support the machine learning lifecycle from development to production
• Design and implement modern data flow architectures that support both batch and real-time processing needs
• Apply performance tuning techniques for high-volume data processing in HPC environments
• Collaborate with data scientists, ML engineers, and business stakeholders to understand data requirements
• Develop and maintain documentation for data infrastructure, pipelines, and workflows
• Implement data quality monitoring and validation across hybrid environments
• Mentor junior data engineers on best practices for hybrid cloud/on-premises architectures

• Bachelor's degree in Computer Science, Information Systems, or related technical field
• Relevant certifications in Azure, data technologies, or HPC administration
• 3-5 years of experience in data engineering or related field
• Strong UNIX/Linux administration skills and command-line proficiency
• Experience with high-performance computing environments and workload management
• Proficiency in SQL and experience with major database systems (Postgre
  
  SQL, SQL Server, Oracle, MySQL)
• Hands-on experience with Azure cloud data services (Azure Data Factory, Synapse Analytics, Azure SQL)
• Knowledge of modern data flow architectures (Lambda, Kappa, Delta) and implementation patterns
• Experience with ML flow frameworks and ML lifecycle management tools
• Programming skills in Python and shell scripting for automation and data processing
• Familiarity with containerization (Docker) and orchestration (Kubernetes)
• Experience with data processing frameworks (Spark, Databricks, Azure Batch)
• Understanding of networking concepts for data movement between on-premises and cloud

• Experience with real-time data processing and streaming technologies (Kafka, Event Hubs)
• Knowledge of infrastructure-as-code tools (Terraform, ARM templates)
• Experience with distributed file systems (HDFS, Lustre) and object storage
• Familiarity with graph databases and specialized analytics data stores
• Understanding of data governance, security, and compliance requirements across hybrid environments
• Experience with optimization techniques for large-scale data transfers between systems
• Knowledge of GPU acceleration for data processing pipelines
• Background in implementing CI/CD for data solutions