Lead Data Engineer (Databricks | Python | PySpark) jobs in United States
cer-icon
Apply on Employer Site
company-logo

Staffing Technologies ยท 4 hours ago

Lead Data Engineer (Databricks | Python | PySpark)

positionUnited States
timeContract
remoteRemote
seniorityLead/Staff
date15+ years exp

Staffing Technologies is seeking a Lead Data Engineer to design, build, and lead modern cloud-based data platforms with a strong focus on Databricks, Python, and PySpark. This role involves hands-on engineering and technical leadership, overseeing architecture decisions and delivering scalable data solutions.

Staffing Agency
check
H1B Sponsor Likelynote

Responsibilities

Lead the design and delivery of cloud-native data platforms using Databricks
Architect and implement Lakehouse and Data Warehouse patterns
Build and optimize ETL/ELT pipelines using Python and PySpark
Establish engineering standards , reusable frameworks, and metadata-driven orchestration
Review designs, vet solutions with the team, and lead demos and retros prior to deployment
Enforce data quality, lineage, monitoring, and alerting across pipelines
Mentor engineers and provide hands-on technical leadership
Partner with analytics and business teams to align solutions with data and reporting needs

Qualification

DatabricksPythonPySparkSQLCloud-based data platformsData engineeringCI/CD pipelinesAgile/SCRUMGit workflowsTesting strategiesCommunication skills

Required

~15 years of total experience in data or software engineering
3+ years in a technical lead role
5+ years building cloud-based data platforms
Proven delivery of production-grade, scalable data systems
Excellent Communication Skills are critical here
Hands-on experience with Databricks Notebooks, Jobs, and workload optimization
Building pipelines using Lakeflow / Declarative Pipelines
Data ingestion via Databricks connectors
Implementing data lineage, quality checks, monitoring, and alerting
Table, compute, and performance optimization within Databricks
Advanced Python with strong packaging and dependency management
Expert PySpark for distributed data processing
Clear understanding of Spark vs single-node execution
Spark performance tuning and troubleshooting
Strong SQL for mid-to-complex transformations
Query and data model optimization to reduce compute and improve performance
Strong adherence to SOLID and DRY principles
Experience building parameterized, reusable frameworks
Agile/SCRUM delivery experience
Git-based development workflows and code reviews
Testing strategies: unit, integration, and end-to-end
CI/CD pipelines and Infrastructure as Code (Terraform)

Company

Staffing Technologies

twittertwittertwitter
company-logo
Staffing Technologies is an IT staffing firm.
dateFounded in 1994
location
Alpharetta, Georgia, USA
people-size201-500 employees
web-link
http://www.staffingtechnologies.com

H1B Sponsorship

Staffing Technologies has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2023 (2)
2020 (6)

Funding

Current Stage
Growth Stage

Leadership Team

leader-logo
Ramesh Lokre
CEO
linkedin
Company data provided by crunchbase