Perficient · 2 weeks ago
Lead Databricks Data Engineer
United States
Full-time
Remote
Senior Level, Lead/Staff
$73K/yr - $150K/yr
8+ years expPerficient is a global digital consulting leader transforming the world’s largest enterprises and biggest brands. The Lead Databricks Data Engineer will lead a development team, focusing on creating and implementing innovative data solutions using Databricks and related technologies while collaborating with architects and infrastructure teams.
ConsultingInformation TechnologyInternetMarketing
Responsibilities
Lead the technical planning and requirements gathering
Serve as a technical lead and mentor. Provide technical support and leadership in the development and continual improvement of service
Develop and maintain effective working relationships with team members
Demonstrate the ability to adapt and work with team members of various experience levels
Work with the data engineering team to define and develop data ingestion, validation, transformation, and data engineering code
Lead the design, development, testing and deployment of scalable batch/streaming data pipelines using Databricks. Data jobs include data ingestion, cleansing, transformation, aggregation, governance, and sharing
Define and enforce best practices for Databricks usage, including notebook development, job orchestration, metadata-driven frameworks, data quality checks, and optimization of spark job performance
Develop Databricks platform components using Python, SQL, Pyspark, Notebook Repositories (medallion architecture L2, L3, L4), and CI/CD
Integrate Databricks solutions with cloud platforms like Azure (ADF, ADLS, Event Hub, Azure DevOps) or AWS (S3, Lambda, Glue)
Design and implement analytical data models and star schemas for BI and AI usage
Collaborate with data scientists and analysts to meet data requirements
Implement logging, monitoring and exception handling for data workflows
Participate in code reviews and CI/CD pipeline development
Document code artifacts and participate in developing user documentation and run books
Troubleshoot deployment to various environments and provide test support
Participate in design sessions, demos, and prototype sessions, testing and training workshops with business users and IT associates
Qualification
Required
Passionate developer with 8+ years of data engineering experience, with at least 3 years in a lead or senior data engineer role
Minimum 5 years of hands-on Databricks experience using Python, SQL, Pyspark, Notebook Repositories (medallion architecture L2, L3, L4), CI/CD
Minimum 5 years of Databricks experience in developing large-scale data processing/data storage/data distribution systems
Experience with AWS and/or Azure cloud environment
Experience with various Notebook Repositories (medallion architecture L2, L3, L4) as well as unit testing frameworks
DevOps experience with Databricks CI/CD (Databricks Asset Bundles) and Git
Minimum 5 years of experience working in a large multi-vendor environment with multiple teams as a part of a project
Minimum 5 years of working with a complex Big Data/Lakehouse environment
Minimum of 5 years of experience with JIRA/GitHub/Git and other code management toolsets
Experience with ELT/ETL development, patterns, and tooling is recommended
Experience with SQL, including Postgres, RDBMS platforms
Experience with various IDE and code repositories as well as unit testing frameworks
Demonstrated ability to leverage AI tools to enhance productivity, streamline workflows, and support data-informed task execution
A solid understanding of AI capabilities and limitations, including ethical considerations, is expected
Flexible and adaptable attitude, disciplined to manage multiple responsibilities and adjust to varied environments
Ability to produce high-quality products within deadlines
Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field
Preferred
Certification in Databricks, Spark, AWS, Azure, or other cloud platforms
Experience with ETL/ELT data platforms, including Informatica, DataStage, Fivetran, Matillion, and dbt
Familiarity with ML capabilities such as MLflow, TensorFlow, or Scikit-learn is a plus
Master's degree in Computer Science or related field
Client-facing or consulting experience is highly preferred
Company
Perficient
Perficient is an IT consulting firm that helps companies adapt their systems to third party software products that use internet.
5001-10000 employees
H1B Sponsorship
Perficient has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (154)
2024 (147)
2023 (156)
2022 (196)
2021 (175)
2020 (223)
Funding
Current Stage
Public CompanyTotal Funding
$65.3M2024-05-05Acquired
2024-01-17Post Ipo Equity· $4.44M
2022-10-17Post Ipo Equity· $5.18M
Leadership Team
Ed Hofffman
Senior Vice President, North America Business Operations
Hari Madamalla
Vice President
Recent News
2025-11-10
Dallas Innovates
2025-10-11
Company data provided by crunchbase