Data Engineer

Overview

Proven experience as a Software Developer, with a strong focus on building scalable and efficient Python applications. Experience in developing Spark Structured Streaming applications is highly desirable. Minimum of 7+ years of professional software development experience. Strong analytical and problem-solving skills, with the ability to debug and optimize Spark jobs running on Databricks. Ability to work closely with cross-functional teams to deliver high-quality streaming solutions.

• Strong expertise in Python, PySpark, and Spark Structured Streaming.
• Experience with Databricks and Azure.
• Familiarity with Delta Lake and Terraform scripting.
• Proficiency in working with varied data file formats (Avro, JSON, CSV) for ingestion and transformation.

• Proficiency in Object-Oriented Programming (OOP) concepts and software design principles.
• Ability to write clean, maintainable, and scalable Python code.
• Experience in setting up and managing CI/CD pipelines using Git Hub Actions to ensure smooth and automated deployment processes.

• Experience working in an Agile/Scrum environment, with a focus on iterative development, continuous feedback, and delivery.

• Python Unit Testing.
• Unity Catalog.
• Databricks Asset Bundles.
• Unit Testing/Mocking

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage:
  Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support:
  Maternal & Parental Leaves.
• Insurance Options:
  Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth:
  Commuter Benefits & Certification & Training Reimbursement.
• Time Off:
  Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance:
  Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range-$100,000-$120,000 a year