IT Data Engineering JG

United Global Technologies
-Houston, TX, United States

This is not a standard data engineer role.

Looking for a deeply technical, hands-on individual contributor who can:

• Diagnose performance, latency, and cost issues in a large-scale cloud data platform
• Take a top-down, platform-level view across multiple projects
• Improve architecture, efficiency, and cost optimization, not just write Spark code
• Act as a technical problem-solver and mentor, guiding other data engineers

This person is expected to make the platform better, not just execute tasks.

• Data processed into two separate containers:
• Crude trading
• Product trading

• Qlik Replicate (formerly Attunity)
• Streaming data from on-prem to Azure
• Databricks
• Spark / Py Spark
• Python
• SQL (complex queries and procedures)

• End-to-end latency issues that need root-cause analysis

• DLT costs are far higher than expected
• Known contributors:
• Very high data volume (expected)
• Inefficient lookup logic used to split data into separate containers
• The current solution works but is not optimal

This role exists because generic recommendations are not enough.

• Someone who has:
• Only written Spark notebooks
• Only followed architectural guidance
• Only worked at a surface level
• Someone who:
• Avoids ambiguity or deep technical investigation
• Someone whose resume was “AI-polished” but not real

• Deep understanding of:
• Databricks internals
• Performance tuning and optimization
• Ability to:
• Propose and prove better approaches via POCs
• Comfortable challenging Databricks as a product
• Gather evidence
• Support escalation discussions with Databricks engineers

• Strong Python (mandatory)
• PySpark (advanced, not basic)
• Advanced SQL
• Stored procedures
• Analytical logic

• Collaborative with data engineers
• Willing to:
• Review others’ solutions
• Build POCs independently

• Will work across multiple projects
• Acts as a cross-platform technical expert