UST · 12 hours ago
Lead Data Engineer – Azure Databricks Lakehouse
United States
Full-time
Remote
Senior Level, Lead/Staff
$106K/yr - $159K/yr
8+ years expUST is a mission-driven company that transforms lives through technology. They are seeking a highly skilled Lead Data Engineer to join their data platform team, responsible for designing, building, and optimizing scalable data pipelines using Azure services and Databricks.
ConsultingInformation ServicesInformation Technology
Responsibilities
Design, develop, and maintain scalable data pipelines leveraging Azure Databricks and Spark (PySpark, Spark SQL)
Build and orchestrate workflows using Azure Data Factory (ADF)
Manage data storage and lifecycle using Azure Data Lake Storage (ADLS)
Develop and maintain CI/CD pipelines using Azure DevOps, following standard SDLC practices across dev, test, and prod environments
Write and maintain PowerShell scripts for automation, deployments, and operational tasks
Develop and optimize SQL code across platforms including T-SQL and PostgreSQL
Work extensively with Parquet and Delta Lake formats to ensure efficient storage and processing
Optimize data pipelines for performance by analyzing Spark DAGs and identifying resource-intensive operations
Implement best practices around indexing, partitioning, and query optimization
Write idempotent, repeatable code that consistently produces the same end state, aligned with a custom SQL deployment framework
Participate in peer code reviews and contribute to continuous improvement of engineering standards
Qualification
Required
8-12 years of experience in data engineering
Strong hands-on experience designing, building, and optimizing scalable data pipelines using Azure services, Databricks, and Spark
Design, develop, and maintain scalable data pipelines leveraging Azure Databricks and Spark (PySpark, Spark SQL)
Build and orchestrate workflows using Azure Data Factory (ADF)
Manage data storage and lifecycle using Azure Data Lake Storage (ADLS)
Develop and maintain CI/CD pipelines using Azure DevOps, following standard SDLC practices across dev, test, and prod environments
Write and maintain PowerShell scripts for automation, deployments, and operational tasks
Develop and optimize SQL code across platforms including T-SQL and PostgreSQL
Work extensively with Parquet and Delta Lake formats to ensure efficient storage and processing
Optimize data pipelines for performance by analyzing Spark DAGs and identifying resource-intensive operations
Implement best practices around indexing, partitioning, and query optimization
Write idempotent, repeatable code that consistently produces the same end state, aligned with a custom SQL deployment framework
Participate in peer code reviews and contribute to continuous improvement of engineering standards
Azure Cloud Platform
Azure Data Factory (ADF) – orchestration
Azure Data Lake Storage (ADLS) – storage
Azure DevOps – CI/CD and source control
Databricks & Spark
Strong hands-on experience with Databricks compute and ETL
Expertise in PySpark and Spark SQL
Ability to analyze Spark execution plans and DAGs
Strong proficiency in PowerShell
Advanced SQL skills (T-SQL, PostgreSQL)
Proficiency with Parquet and Delta Lake
Benefits
Paid vacation
Paid sick leave
Paid holidays
Paid bereavement leave
Jury duty
401(k) Retirement Plan with employer matching
Medical insurance
Dental insurance
Vision insurance
Basic life insurance
Accidental death and disability insurance
Short- and long-term disability benefits
Voluntary short-term disability benefits
Health Savings Account (HSA)
Flexible Spending Account (FSA)
Company
UST
UST is a Digital Transformations Solutions Provider.
10001+ employees
H1B Sponsorship
UST has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (870)
2024 (800)
2023 (647)
2022 (634)
2021 (612)
2020 (984)
Funding
Current Stage
Late StageTotal Funding
$250MKey Investors
Temasek Holdings
2018-06-27Private Equity· $250M
Leadership Team
Krishna Sudheendra
CEO
Alexander Varghese
Chief Administrative Officer & COO
Recent News
2026-01-20
2026-01-20
Company data provided by crunchbase