Mothership ยท 5 months ago
DataBricks Administrator
Cincinnati, Ohio, United States
Full-time
Onsite
Senior Level
5+ years expMothership is seeking a DataBricks Administrator to manage and optimize Azure Databricks workspaces. The role involves deploying and administering Databricks environments, ensuring security compliance, and collaborating with data teams to enhance data pipelines and analytics workloads.
AdvertisingBrand MarketingGraphic DesignLogisticsProfessional ServicesWeb Design
Responsibilities
Deploy, configure, and manage Azure Databricks workspaces in a scalable, cost-efficient, and secure manner
Administer clusters, jobs, notebooks, and workflows, ensuring high availability and performance
Monitor and optimize compute resource utilization and autoscaling strategies to improve cost efficiency
Manage Databricks Runtime versions, libraries, and dependencies across environments
Implement and manage Unity Catalog for fine-grained access control and data governance
Enforce Role-Based Access Control (RBAC) and integrate Databricks with Azure Active Directory (AAD)
Ensure compliance with SOC 2, HIPAA, GDPR, and internal security standards
Set up audit logging, monitoring, and alerting for security and operational insights
Tune Apache Spark workloads to improve query performance and resource efficiency
Analyze and troubleshoot performance bottlenecks in ETL and ML workloads
Optimize Delta Lake storage, caching, and indexing strategies for better query execution
Automate Databricks workspace deployment using Terraform, ARM Templates, or Databricks REST API
Develop and maintain CI/CD pipelines for Databricks job deployment and configuration management
Implement monitoring solutions using Azure Monitor, Prometheus, or Grafana
Work closely with data engineers, data scientists, and DevOps teams to support data pipelines and analytics workloads
Integrate Databricks with Azure Data Lake, Azure Synapse Analytics, and Snowflake
Provide technical guidance and best practices for efficient Spark job execution and cost optimization
Qualification
Required
5+ years of experience in Azure Databricks administration and performance optimization
Expertise in Apache Spark, PySpark, SQL, and Scala
Hands-on experience with Databricks Unity Catalog, Delta Lake, and MLflow
Strong knowledge of Azure cloud services (Azure Data Lake, Azure Synapse, Azure Key Vault, etc.)
Experience in Infrastructure as Code (Terraform, Bicep, or ARM Templates)
Proficiency in CI/CD pipeline automation using Azure DevOps, GitHub Actions, or Jenkins
Strong understanding of network security, identity management (AAD), and encryption best practices
Excellent problem-solving skills and ability to troubleshoot complex Databricks workloads
Strong communication and documentation skills
Preferred
Databricks Certified Associate or Professional certification
Experience with Azure Kubernetes Service (AKS) and serverless computing
Familiarity with Kafka, Apache Airflow, and event-driven architectures
Knowledge of Python, PowerShell, or Bash scripting for automation
Company
Mothership
Mothership provides branding, marketing, advertising, graphic design, shipping, professional services, and web design.
11-50 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase