Apply on Employer Site

Bizmetric · 1 month ago

Senior Databricks Engineer

Sugar Land, TX

Full-time

Onsite

Senior Level

Bizmetric is seeking an experienced Senior Databricks Engineer to design, develop, optimize, and manage large-scale data pipelines and analytics solutions. The role involves working with cross-functional teams to deliver scalable data solutions for enterprise clients, leveraging expertise in Databricks, Spark, and cloud platforms.

AnalyticsInformation Technology

H1B Sponsor Likely

Responsibilities

Design, develop, and optimize Spark/Databricks ETL/ELT pipelines at scale

Implement efficient Delta Lake architectures (bronze/silver/gold layers)

Write high-performance PySpark/SQL code following best practices

Tune clusters, jobs, and workflows for performance and cost efficiency

Develop data models and schemas to support analytics, reporting, and ML workloads

Implement medallion architecture following industry and Databricks standards

Design scalable and secure storage patterns using cloud-native services

Work with cloud platforms (Azure preferred, AWS or GCP acceptable)

Implement CI/CD pipelines for Databricks using Git, Azure DevOps, or GitHub Actions

Integrate Databricks with third-party systems, ADF, Event Hub, Kafka, APIs, etc

Configure workflows in Databricks (Jobs, Workflows, Unity Catalog)

Implement data validation, observability, and monitoring frameworks

Apply best practices in data governance, lineage, and cataloging

Enforce security guidelines using RBAC, Unity Catalog, and encryption policies

Lead technical solutioning for client projects and POCs

Provide mentorship to junior engineers

Collaborate closely with business analysts, product managers, and stakeholders

Prepare architectural documentation, standards, and reusable accelerators

Investigate data issues, performance bottlenecks, and operational failures

Drive continuous improvement across data engineering processes

Ensure reliable and timely data delivery for downstream applications

Qualification

DatabricksSparkDelta Lake architectureCloud platforms (Azure/AWS/GCP)PySparkCI/CD pipelinesData governanceMentorshipCollaboration

Required

Deep expertise in distributed data processing

Experience with Delta Lake architecture

End-to-end data engineering best practices