RIT Solutions, Inc. · 2 hours ago
Sr. Databricks Solutions Architect
Washington, DC
Full-time
Onsite
Senior Level
5+ years expRIT Solutions, Inc. is seeking a Sr. Databricks Solutions Architect to lead the architecture and administration of the Azure Databricks Lakehouse. The role involves ensuring platform reliability, mentoring teams, and implementing best practices for data ingestion, security, and operational excellence.
Staffing & Recruiting
Responsibilities
Serve as the primary architect and administrator for the Azure Databricks Lakehouse (Unity Catalog, workspaces, external locations, compute, access controls)
Lead execution of a Minimal Viable Hardening Roadmap for the platform, prioritizing: High availability and DR readiness, Backup/restore patterns for data and metadata, Platform observability and operational metrics, Secure and maintainable catalog/namespace structure, Robust and proactive data quality assurance
Implement and evolve naming conventions, environment strategies, and platform standards that enable long-term maintainability and safe scaling
Act as the Lakehouse-facing counterpart to Enterprise Architecture and Security, collaborating on network architecture, identity & access, compliance controls, and platform guardrails
Design, implement, and maintain comprehensive monitoring and alerting for Lakehouse platform components, ingestion jobs, key data assets, and system health indicators
Oversee end-to-end reliability engineering, including capacity planning, throughput tuning, job performance optimization, and preventative maintenance (e.g., IR updates, compute policy reviews)
Participate in — and help shape — the on-call rotation for high-priority incidents affecting production workloads, including rapid diagnosis and mitigation during off-hours as needed
Develop and maintain incident response runbooks, escalation pathways, stakeholder communication protocols, and operational readiness checklists
Lead or participate in post-incident Root Cause Analyses, ensuring durable remediation and preventing recurrence
Conduct periodic DR and failover simulations, validating RPO/RTO and documenting improvements
Strengthen and standardize ingestion pipelines (ADF → landing → raw → curated), including watermarking, incremental logic, backfills, and retry/cancel/resume patterns
Collaborate with the Data Engineer to modernize logging, automated anomaly detection, pipeline health dashboards, and DQ validation automation
Provide architectural guidance, code reviews, mentoring, and best-practice patterns to distributed engineering teams across MedStar
Support stabilization of existing ingestion and transformation pipelines across clinical (notes, OHDSI), financial, operational, and quality use cases
Administer and improve CI/CD pipelines using Azure DevOps or GitHub Enterprise
Support automated testing, environment promotion, and rollback patterns for Databricks and dbt assets
Maintain and extend Terraform (or adopt Terraform from another IaC background) for Databricks, storage, networking, compute policies, and related infrastructure
Promote version control standards, branching strategies, and deployment governance across data engineering teams
Partner with Enterprise Architecture and Security on platform access controls, identity strategy, encryption, networking, and compliance
Implement and enforce cost tagging, compute policies, and alerts supporting FinOps transparency and cost governance
Collaborate with the team defining agentic coding guardrails, ensuring the Lakehouse platform supports safe & compliant use of AI-assisted code generation and execution
Help assess and optimize serverless SQL, serverless Python, and job compute patterns for cost-efficiency and reliability
Mentor the mid-level Data Engineer on Databricks, ADF, dbt, observability, DevOps, Terraform, and operational engineering patterns
Provide guidance, design patterns, and code review support to multiple distributed data engineering teams (Finance, MCPI, Safety/Risk, Quality, Digital Transformation, etc.)
Lead platform knowledge-sharing efforts through documentation, workshops, and best-practice guidance
Demonstrate strong collaboration skills, balancing independence with alignment across teams
Assist with or support operational administration of the OHDSI/OMOP stack (Atlas, WebAPI, vocabularies, Kubernetes deployments)
Collaborate with partners to ensure the OHDSI platform is secure, maintainable, and well-integrated with the Lakehouse
Qualification
Required
5+ years in cloud data engineering, platform engineering, or solution architecture
Strong hands-on expertise in Azure Databricks: Unity Catalog, Workspaces & external locations, SQL/Python notebooks & Jobs, Cluster/warehouse governance
Solid working experience with Azure Data Factory (pipelines, IRs, linked services)
Strong SQL and Python engineering skills
Experience with CI/CD in Azure DevOps or GitHub Enterprise
Experience with Terraform or another IaC framework, and willingness to adopt Terraform
Demonstrated ability to design or support monitoring, alerting, logging, or reliability systems
Strong communication, collaboration, and problem-solving skills
Preferred
Advanced Terraform experience
Familiarity with healthcare, HIPAA, PHI, or regulated environments
Experience with Purview or enterprise cataloging
Exposure to OHDSI/OMOP
Experience optimizing or refactoring legacy ingestion pipelines
Experience supporting secure, controlled AI/agentic execution environments
Experience with EPIC EHR data exchange and/or EPIC Caboodle or Cogito analytics suite
Assist with or support operational administration of the OHDSI/OMOP stack (Atlas, WebAPI, vocabularies, Kubernetes deployments)
Collaborate with partners to ensure the OHDSI platform is secure, maintainable, and well-integrated with the Lakehouse
Company
RIT Solutions, Inc.
Jobdiva Job Portal: https://www1.jobdiva.com/candidates/myjobs/searchjobsdone.jsp?a=xbjdnwgjodtga1y1im2g881fkkeiwd0775lbvq8yqgps8vb2q36w2vj1ga6xxork&compid=-1 Recruitment (contingency search and campus selection).
201-500 employees
H1B Sponsorship
RIT Solutions, Inc. has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (1)
2023 (2)
Funding
Current Stage
Growth StageCompany data provided by crunchbase