SGI · 1 week ago
Lead Data Engineer - Databricks & AWS
Houston, TX
Contract
Onsite
Senior Level, Lead/Staff
8+ years expSGI is seeking a Lead Databricks Engineer to support large-scale data platform initiatives focused on data modernization, cloud migration, and advanced analytics. This hands-on senior role requires deep Databricks expertise and strong AWS experience to partner closely with business and technical stakeholders.
EmploymentRecruitingStaffing Agency
No H1B
Responsibilities
Lead the design and implementation of enterprise-scale Databricks solutions on AWS
Drive data modernization initiatives, including lift-and-shift and re-architecture of legacy data platforms
Build and optimize data pipelines using Python, PySpark, and SQL
Design and manage Delta Lake architectures and implement Unity Catalog for governance and access control
Develop and support streaming and batch data workloads
Configure and optimize Databricks clusters and serverless compute for performance and cost
Integrate Databricks with upstream and downstream systems (APIs, data sources, analytics tools)
Partner with stakeholders to gather user and business requirements and translate them into technical solutions
Implement observability, monitoring, and cost controls , including usage, volume, and pricing metrics
Support data domains related to billing, accounting, pricing, and volume metrics
Provide technical leadership, best practices, and mentorship to engineering teams
Qualification
Required
8+ years of experience in data engineering, with recent hands-on focus on Databricks
Strong experience deploying Databricks on AWS
Advanced coding skills in Python, PySpark, and SQL
Deep knowledge of Delta Lake, Unity Catalog, and Databricks workspace governance
Experience with streaming data (Structured Streaming, Kafka, or similar)
Strong understanding of Databricks cluster management, serverless compute, and performance tuning
Experience integrating Databricks with enterprise systems and data sources
Proven ability to work directly with business and technical stakeholders
Strong communication skills and ability to lead technical discussions
Experience with AKS / Kubernetes environments
Preferred
Experience supporting financial data domains (billing, accounting, pricing, usage metrics) is highly preferred
Databricks or AWS certifications
Consulting or contracting background in large enterprise environments
Company
SGI
Source Group International is a global, award-winning technology talent partner trusted by ambitious enterprise businesses and professionals worldwide.
51-200 employees
Funding
Current Stage
Growth StageLeadership Team
Gavin Tew
Co-Founder
Lawrence Hargreaves
Co-Founder
Recent News
2025-10-04
Company data provided by crunchbase