Arkhya Tech. Inc. · 18 hours ago
Databricks Data Engineer with DevOps Skills
Los Angeles, CA
Contract
Hybrid
Mid, Senior LevelArkhya Tech. Inc. is seeking an experienced Databricks Data Engineer with strong DevOps expertise to join their data engineering team. The role involves designing, building, and optimizing large-scale data pipelines on the Databricks Lakehouse platform while implementing robust CI/CD and deployment practices.
ConsultingInformation TechnologySoftware
Hiring Manager
Eshwar S
Responsibilities
Design, build, and maintain scalable ETL/ELT pipelines using Databricks
Develop data processing workflows using PySpark/Spark and SQL for large‑volume datasets
Integrate data from ADLS, Azure Blob Storage, and relational/non-relational data sources
Implement Delta Lake best practices including schema evolution, ACID transactions, OPTIMIZE, ZORDER, and performance tuning
Implement CI/CD pipelines for Databricks using Git , GitLab , Azure DevOps , or similar tools
Build and manage automated deployments using Databricks Asset Bundles
Manage version control for notebooks, workflows, libraries, and configuration artifacts
Automate cluster configuration, job creation, and environment provisioning
Work with data analysts and BI teams to prepare datasets for reporting and dashboarding
Collaborate with product owners, business partners, and engineering teams to translate requirements into scalable data solutions
Document data flows, architecture, and deployment processes
Tune Databricks clusters, jobs, and pipelines for cost efficiency and high performance
Monitor workflows, debug failures, and ensure pipeline stability and reliability
Implement job instrumentation and observability using logging/monitoring tools
Implement and manage data governance using Unity Catalog
Enforce access controls, data security, and compliance with enterprise policies
Ensure best practices around data quality, lineage, and auditability
Qualification
Required
Strong hands-on experience with Databricks, including Delta Lake, Unity Catalog, Lakehouse Architecture, Delta Live Pipelines, Databricks Runtime, Table Triggers
Proficiency in PySpark, Spark, and advanced SQL
Expertise with Azure cloud services (ADLS, ADF, Key Vault, Functions, etc.)
Experience with relational databases and data warehousing concepts
Strong understanding of DevOps tools: Git/GitLab, CI/CD pipelines, Databricks Asset Bundles
Familiarity with infrastructure-as-code (Terraform is a plus)
Design, build, and maintain scalable ETL/ELT pipelines using Databricks
Develop data processing workflows using PySpark/Spark and SQL for large-volume datasets
Integrate data from ADLS, Azure Blob Storage, and relational/non-relational data sources
Implement Delta Lake best practices including schema evolution, ACID transactions, OPTIMIZE, ZORDER, and performance tuning
Implement CI/CD pipelines for Databricks using Git, GitLab, Azure DevOps, or similar tools
Build and manage automated deployments using Databricks Asset Bundles
Manage version control for notebooks, workflows, libraries, and configuration artifacts
Automate cluster configuration, job creation, and environment provisioning
Work with data analysts and BI teams to prepare datasets for reporting and dashboarding
Collaborate with product owners, business partners, and engineering teams to translate requirements into scalable data solutions
Document data flows, architecture, and deployment processes
Tune Databricks clusters, jobs, and pipelines for cost efficiency and high performance
Monitor workflows, debug failures, and ensure pipeline stability and reliability
Implement job instrumentation and observability using logging/monitoring tools
Implement and manage data governance using Unity Catalog
Enforce access controls, data security, and compliance with enterprise policies
Ensure best practices around data quality, lineage, and auditability
Preferred
Knowledge of streaming technologies like Structured Streaming or Spark Streaming
Experience building real-time or near real-time pipelines
Exposure to advanced Databricks runtime configurations and tuning
Databricks Certified Data Engineer Associate / Professional
Azure Data Engineer Associate
Company
Arkhya Tech. Inc.
Arkhya Technologies, Inc.
Founded in 2014
201-500 employees
H1B Sponsorship
Arkhya Tech. Inc. has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2023 (1)
2022 (1)
2020 (5)
Funding
Current Stage
Growth StageCompany data provided by crunchbase