This job has closed.

TalTeam · 15 hours ago

Cloud Systems Engineer

Washington, DC

Full-time

Onsite

Senior Level

7+ years exp

TalTeam is a company seeking a Cloud Systems Engineer who will be responsible for the administration and operational support of the Databricks platform. This role includes managing platform operations, security, and governance, ensuring compliance and enabling secure analytics and AI/ML workloads at scale.

ConsultingInformation ServicesInformation TechnologySoftware

H1B Sponsor Likely

Responsibilities

Administer Databricks account and workspaces across SDLC environments; standardize configuration, naming, and operational patterns

Configure and maintain clusters/compute, job compute, SQL warehouses, runtime versions, libraries, repos, and workspace settings

Implement platform monitoring/alerting, operational dashboards, and health checks; maintain runbooks and operational procedures

Provide Tier 2/3 operational support: troubleshoot incidents, perform root-cause analysis, and drive remediation and preventive actions

Manage change control for upgrades, feature rollouts, configuration changes, and integration changes; document impacts and rollback plans

Enforce least privilege across platform resources (workspaces, jobs, clusters, SQL warehouses, repos, secrets) using role/group-based access patterns

Configure and manage secrets and secure credential handling (secret scopes / key management integrations) for platform and data connectivity

Enable and maintain audit logging and access/event visibility; support security reviews and evidence requests

Administer Unity Catalog governance: metastores, catalogs/schemas/tables, ownership, grants, and environment/domain patterns

Configure and manage external locations, storage credentials, and governed access to cloud object storage

Partner with governance stakeholders to support metadata/lineage integration, classification/tagging, and retention controls where applicable

Coordinate secure connectivity and guardrails with cloud/network teams: private connectivity patterns, egress controls, firewall/proxy needs

Configure cloud integrations required for governed data access and service connectivity (roles/permissions, endpoints, storage integrations)

Implement cost guardrails: cluster policies, auto-termination, scheduling, workload sizing standards, and capacity planning

Produce usage/cost insights and optimization recommendations; address waste drivers (idle compute, oversized clusters, inefficient jobs)

Automate administration and configuration using APIs/CLI/IaC (e.g., Terraform) to reduce manual drift and improve repeatability

Maintain platform documentation: configuration baselines, security/governance standards, onboarding guides, and troubleshooting references

Design and implement backup and disaster recovery procedures for workspace configurations, notebooks, Unity Catalog metadata, and job definitions; maintain recovery runbooks and perform periodic DR testing aligned to RTO/RPO objectives

Monitor and optimize platform performance, including SQL warehouse query tuning, cluster autoscaling configuration, Photon enablement, and Delta Lake optimization guidance (OPTIMIZE, VACUUM, Z-ordering strategies)

Administer Delta Live Tables (DLT) pipelines and coordinate with data engineering teams on pipeline health, data quality monitoring, failed job remediation, and pipeline configuration best practices

Manage third-party integrations and ecosystem connectivity, including BI tool integrations (e.g., Power BI), and external metadata catalog integrations

Implement Databricks Asset Bundles (DABs) for standardized deployment patterns; automate workspace resource deployment (jobs, pipelines, dashboards) across SDLC environments using bundle-based CI/CD workflows

Conduct capacity planning and scalability analysis, including forecasting concurrent user/workload growth, platform scaling strategies, and proactive resource allocation during peak usage periods

Facilitate user onboarding and enablement, including new user/team onboarding procedures, training coordination, workspace access provisioning, and creation of self-service documentation/guides

Qualification

Databricks administrationUnity Catalog administrationIdentity & Access ManagementCloud platform expertise (AWS)Automation skills (Terraform)CI/CD practicesSQL proficiencyCompliance frameworks experienceNetworking basics (VPC)Scripting skillsCommunication skillsTeam collaborationProblem-solving skills

Required

Hands-on experience administering Databricks (workspace administration, clusters/compute policies, jobs, SQL warehouses, repos, runtime management) and expertise using Databricks CLI

Strong Unity Catalog administration: metastores; catalogs/schemas; grants; service principals; external locations; storage credentials; governed storage access

Identity & Access Management proficiency: SSO concepts, SCIM provisioning, group based RBAC, service principals, least-privilege patterns

Security fundamentals: secrets management, secure connectivity, audit logging, access monitoring, and evidence-ready operations

Cloud platform expertise (AWS): IAM roles/policies, object storage security patterns, networking basics (VPC concepts), logging/monitoring integration

Automation skills: scripting and/or IaC using Terraform/CLI/REST APIs for repeatable configuration and environment promotion

Experience implementing data governance controls (classification/tagging, lineage/metadata integrations) in partnership with governance teams

CI/CD practices for jobs/notebooks/config promotion across SDLC environments

Understanding of lakehouse concepts (e.g., Delta, table lifecycle management, separation of storage/compute)

SQL proficiency and data engineering fundamentals for troubleshooting query performance issues, understanding ETL/ELT workflow patterns, and debugging data pipeline failures; basic Python/Scala familiarity for notebook/code issue diagnosis

Experience with compliance and regulatory frameworks (FedRAMP, HIPAA, SOC2, or similar) including implementation of data residency requirements, retention policies, and audit-ready evidence collection

Hands-on experience with AWS security and networking services including PrivateLink, Secrets Manager/Systems Manager integration, CloudWatch/CloudTrail integration, S3 bucket policies, cross-account access patterns, and KMS encryption key management

Experience administering Databricks serverless compute, Workspace Git integrations (GitLab), Databricks Asset Bundles (DABs) for deployment automation, and modern workspace features supporting DevOps workflows

SLA/SLO management and stakeholder communication skills; ability to define platform service levels, produce operational reports, translate technical issues to business stakeholders, and manage vendor relationships (Databricks account teams)

Bachelor's degree in a related field or equivalent practical experience

7+ years in cloud/data platform administration and operations, including 4+ years supporting Databricks or similar platforms

Databricks Platform Administrator/Databricks AWS Platform Architect

Databricks Certified Data Engineer Associate/Professional

AWS Certified Solutions Architect Associate or Professional

Company

TalTeam

TalTeam is an IT service provider that specializes in Software, Manufacturing, Retail, and Distribution.

Founded in 2011

Herndon, Virginia, USA

51-200 employees

http://www.talteam.com/

H1B Sponsorship

TalTeam has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2025 (1)

2024 (6)

2023 (6)

2022 (5)

2021 (9)

2020 (20)

Funding

Current Stage

Growth Stage

Leadership Team

Sudha Mathew

CEO

Company data provided by crunchbase