UST · 9 hours ago
Senior Data Engineer – Azure Databricks Lakehouse
United States
Full-time
Remote
Mid, Senior Level
$92K/yr - $138K/yr
6+ years expUST is a mission-driven company that transforms lives through technology. They are seeking a highly skilled Senior Data Engineer to lead the development and maintenance of scalable data pipelines in an Azure + Databricks Lakehouse environment.
ConsultingInformation ServicesInformation Technology
Responsibilities
Design, develop, and maintain scalable data pipelines leveraging Azure Databricks and Spark (PySpark, Spark SQL)
Build and orchestrate workflows using Azure Data Factory (ADF)
Manage data storage and lifecycle using Azure Data Lake Storage (ADLS)
Develop and maintain CI/CD pipelines using Azure DevOps, following standard SDLC practices across dev, test, and prod environments
Write and maintain PowerShell scripts for automation, deployments, and operational tasks
Develop and optimize SQL code across platforms including T-SQL and PostgreSQL
Work extensively with Parquet and Delta Lake formats to ensure efficient storage and processing
Optimize data pipelines for performance by analyzing Spark DAGs and identifying resource-intensive operations
Implement best practices around indexing, partitioning, and query optimization
Write idempotent, repeatable code that consistently produces the same end state, aligned with a custom SQL deployment framework
Participate in peer code reviews and contribute to continuous improvement of engineering standards
Qualification
Required
6-8 years of experience in data engineering
Strong hands-on experience designing, building, and optimizing scalable data pipelines using Azure services, Databricks, and Spark
Design, develop, and maintain scalable data pipelines leveraging Azure Databricks and Spark (PySpark, Spark SQL)
Build and orchestrate workflows using Azure Data Factory (ADF)
Manage data storage and lifecycle using Azure Data Lake Storage (ADLS)
Develop and maintain CI/CD pipelines using Azure DevOps, following standard SDLC practices across dev, test, and prod environments
Write and maintain PowerShell scripts for automation, deployments, and operational tasks
Develop and optimize SQL code across platforms including T-SQL and PostgreSQL
Work extensively with Parquet and Delta Lake formats to ensure efficient storage and processing
Optimize data pipelines for performance by analyzing Spark DAGs and identifying resource-intensive operations
Implement best practices around indexing, partitioning, and query optimization
Write idempotent, repeatable code that consistently produces the same end state, aligned with a custom SQL deployment framework
Participate in peer code reviews and contribute to continuous improvement of engineering standards
Azure Cloud Platform
Azure Data Factory (ADF) – orchestration
Azure Data Lake Storage (ADLS) – storage
Azure DevOps – CI/CD and source control
Databricks & Spark
Strong hands-on experience with Databricks compute and ETL
Expertise in PySpark and Spark SQL
Ability to analyze Spark execution plans and DAGs
Strong proficiency in PowerShell
Advanced SQL skills (T-SQL, PostgreSQL)
Proficiency with Parquet and Delta Lake
Benefits
Full-time, regular employees accrue a minimum of 10 days of paid vacation per year
Receive 6 days of paid sick leave each year (pro-rated for new hires throughout the year)
10 paid holidays
Eligible for paid bereavement leave and jury duty
Eligible to participate in the Company’s 401(k) Retirement Plan with employer matching
Eligible for medical, dental, and vision insurance
Basic life insurance
Accidental death and disability insurance
Short- and long-term disability benefits
May purchase additional voluntary short-term disability benefits
Participate in a Health Savings Account (HSA)
Flexible Spending Account (FSA) for healthcare, dependent child care, and/or commuting expenses as allowable under IRS guidelines
Company
UST
UST is a Digital Transformations Solutions Provider.
10001+ employees
H1B Sponsorship
UST has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (870)
2024 (800)
2023 (647)
2022 (634)
2021 (612)
2020 (984)
Funding
Current Stage
Late StageTotal Funding
$250MKey Investors
Temasek Holdings
2018-06-27Private Equity· $250M
Leadership Team
Krishna Sudheendra
CEO
Alexander Varghese
Chief Administrative Officer & COO
Recent News
2026-01-20
2026-01-20
Company data provided by crunchbase