Databricks Technical Lead/Engineer jobs in United States
cer-icon
Apply on Employer Site
company-logo

Phaxis · 3 weeks ago

Databricks Technical Lead/Engineer

positionChicago, IL
timeContract
remoteHybrid
senioritySenior Level, Lead/Staff
date5+ years exp

Phaxis is seeking a Databricks Technical Lead to guide the design and build-out of data engineering platforms. This hands-on leadership role involves mentoring engineers, setting standards, and making architecture decisions to influence the platform's future.

DeliveryHuman ResourcesStaffing Agency

Responsibilities

Lead the technical direction for Databricks-based data pipelines and frameworks
Design and review patterns for ingesting, transforming, and publishing data (Bronze → Silver → Gold)
Define best practices around Delta Lake, schema evolution, SCD handling, and metadata-driven transformations
Provide technical oversight across multiple engineering squads
Work with architects, data modelers, quality engineers, and operations teams to ensure pipelines are built the right way
Mentor data engineers and help elevate the overall engineering capability
Oversee Unity Catalog governance, including RBAC, lineage, and schema enforcement
Help troubleshoot complex performance issues and guide teams on tuning and optimization
Support integration with orchestration tools and CI/CD processes

Qualification

DatabricksSparkDelta LakeData engineering patternsAWS data servicesMetadata-driven systemsStreaming experienceUnity CatalogMentoringCommunication skills

Required

Several years of hands-on Spark experience and deep expertise with Databricks (Workflows, Delta Lake, Repos, Unity Catalog)
Strong understanding of data engineering patterns, especially medallion architecture
Comfortable designing metadata-driven systems using YAML/JSON or similar configuration styles
Solid knowledge of AWS data services (S3, Glue, IAM, or equivalents)
Ability to lead and mentor engineers, give constructive feedback, and set engineering standards
Strong communication skills – able to explain complex ideas in a clear and approachable way
5+ years Databricks experience at a senior/lead level
Strong Spark (PySpark + Spark SQL), not just SQL users
Deep understanding of Delta Lake (OPTIMIZE, VACUUM, compaction, file layout)
Designed or owned a medallion architecture
Experience with schema evolution & SCD Type 1/2 handling
Hands-on Unity Catalog experience (permissions, lineage, governance)
Built or maintained metadata-driven frameworks
Streaming experience (Auto Loader / Structured Streaming)
Experience with Airflow, Glue, or Databricks Workflows
Working knowledge of cloud services (AWS)

Preferred

Experience with structured streaming and Auto Loader is a plus

Company

Phaxis

twittertwittertwitter
company-logo
Phaxis is one of the nation's leading recruiting and consulting firms.
dateFounded in 2002
location
New York, New York, USA
people-size51-200 employees
web-link
http://phisearch.com

Funding

Current Stage
Growth Stage

Leadership Team

leader-logo
Seth Friedland
Co-Founder and CEO
linkedin
E
Erik DiGiacomo
President
linkedin
Company data provided by crunchbase