Lead Data Engineer

Description
*** At this time, we are unable to consider candidates requiring visa sponsorship or third-party recruitment agencies for this role. We encourage all applicants to apply directly, and we thank you for your understanding.**
* Overview:
As a Lead Data Engineer, you will be part of the Data Sciences team driving scalable data infrastructure and workflows that power personalization and recommendation systems across the website and the app. You will play a key role in designing, implementing, and optimizing distributed data pipelines and engineering solutions in a cloud environment. We expect you to follow best practices in software design, contribute to code reviews, maintain a robust and well-tested codebase, and produce clear, maintainable documentation.

Responsibilities:

• Designing and building large-scale data pipelines using Python, PySpark, and Scala

• Performance tuning and optimization of Spark jobs, especially in GCP environments

• Developing and maintaining production-ready workflows using orchestration tools such as Kubeflow Pipelines (KFP) or Airflow

• Collaborating closely with data scientists, ML engineers, and product teams to support experimentation, model development, and deployment

• Managing containerized applications with Docker and integrating them into CI/CD pipelines

• Ensuring code quality through testing, version control, and documentation

• Develops software that processes, stores and serves data for use by others.

• Develops large scale data structures and pipelines to organize, collect and standardize data that helps generate insights and addresses reporting needs.

• Writes ETL (Extract / Transform / Load) processes, designs database systems and develops tools for real-time and offline analytic processing.

• Ensures that data pipelines are scalable, repeatable and secure. Troubleshoots software and processes for data consistency and integrity.

• Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standards. Has in-depth knowledge of large scale search applications and building high volume data pipelines.

• In-depth knowledge of Java, Hadoop, Hive, Cassandra, Pig, MySQL or No

SQL or similar.

Required Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field—or equivalent experience

• 5-20 years of experience in data engineering with a strong focus on Python, PySpark, and/or Scala

• Expertise in Spark performance tuning and optimization in cloud-based environments (preferably GCP)

• Hands-on experience with workflow orchestration tools like KFP or Airflow

• Proficiency with Docker and container-based deployment strategies

• Experience building scalable and maintainable data pipelines

• Strong understanding of distributed systems, big data technologies, and data lifecycle management

• Excellent communication and collaboration skills

Benefits:
York Solutions Offers a generous benefits package for eligible full-time employees:

• BCBS Medical with 3 Plans to choose from (PPO and High deductible PPO plans with Health Savings Program)
• Delta Dental plan with 2 free cleanings and insurance discounts
• Eye Med Vision with annual check-ups and discounts on lens
• Life and Accidental Death Insurance paid by company
• John Hancock 401(k) Retirement Plan with discretionary company match up to 5%
• Voluntary Insurance programs such as:
  Hospital Indemnity, Identity Protection, Legal Insurance, Long Term Care, and Pet Insurance.
• Flexible work environment with some remote working opportunities
• Strong fun and teamwork environment
• Learning, development, and career growth