Selby Jennings · 6 days ago
Databricks Platform Engineer
Chicago, IL
Full-time
Onsite
Mid, Senior Level
$150K/yr - $225K/yrSelby Jennings is seeking a Databricks Platform Engineer to design, operate, and evolve a mission‑critical data and analytics platform. The role involves ensuring platform reliability, automation, and governance while collaborating with various teams to enhance the Databricks environments.
BankingHuman ResourcesEmploymentRecruiting
Hiring Manager
Mimi Meadows
Responsibilities
Administer and support Databricks workspaces, clusters, permissions, policies, and configuration baselines with a focus on uptime, security, and predictable performance
Use Terraform to define infrastructure and platform components across multiple environments, ensuring consistency, repeatability, and policy‑driven guardrails
Build Python‑based automation, operational tooling, health checks, and runbooks to reduce manual work and streamline platform management
Implement and maintain CI/CD patterns for infrastructure and configuration changes, including automated validations, testing, and environment promotion
Collaborate with senior engineering leadership to shape standards, evaluate new Databricks features, and influence platform roadmap decisions
Troubleshoot platform‑level issues including configuration drift, cluster failures, job orchestration, networking concerns, and user‑level operational challenges
Improve visibility into spend, enforce cost‑related policies, and optimize cluster usage patterns
Work with engineering, quant/data teams, and analysts to understand workflows, triage issues, and guide best practices for secure and efficient platform usage
Qualification
Required
Hands-on experience configuring workspaces, access controls, cluster policies, and workspace-level governance
Ability to design reusable modules, multi-environment infrastructure structures, and policy-as-code patterns
Strong automation experience, including API integrations, lifecycle management tasks, and platform tooling
Familiarity with Git-based workflows, automated testing pipelines, and environment promotion strategies
Understanding of monitoring, alerting, observability, log analysis, and safe change management practices
Ability to partner with engineers, data practitioners, and business users to distill ambiguous problems into practical, well-designed solutions
Comfort implementing controls that prevent unsafe changes in shared, high-impact environments
Terraform (multi-env IaC, modules, policies)
Python for platform automation and API integration
Databricks administration (clusters, workspace config, permissions, jobs)
Linux CLI and shell scripting
CI/CD tooling and Git workflows
Cloud fundamentals (IAM, networking basics, storage, compute)
Monitoring and observability tools
Preferred
Experience operating shared developer or data platforms
Familiarity with Databricks REST APIs, Unity Catalog, Delta Lake, or cost governance
Exposure to AWS infrastructure defined via Terraform
Background working in regulated or risk-sensitive environments
Experience in SRE, DevOps, or infrastructure engineering roles
Bachelor's degree in Computer Science, Engineering, or a related technical discipline
Company
Selby Jennings
Global recruitment firm specialising in Banking
Founded in 2004
1001-5000 employees
Funding
Current Stage
Late StageRecent News
Business Insider
2025-09-30
2025-07-10
Seattle TechFlash
2025-05-03
Company data provided by crunchbase