Scicom Infrastructure Services · 1 month ago
Databricks SME
United States
Full-time
Remote
Senior Level
7+ years expScicom Infrastructure Services is seeking a Databricks Subject Matter Expert to design and implement enterprise Databricks Lakehouse architectures. The role involves aligning with the Databricks Well-Architected Framework, leading implementations, and ensuring compliance with security and governance best practices.
Information Technology & Services
Responsibilities
Design enterprise Databricks Lakehouse architectures aligned with the Databricks Well-Architected Framework
Define reference architectures for batch, streaming, analytics, and ML workloads
Select and standardize cluster, compute, and workspace architectures
Design multi-workspace strategies (dev/test/prod, shared vs. isolated)
Ensure architectures meet scalability, availability, and performance requirements
Apply Databricks best practices across all pillars, including:
Security & Governance (Unity Catalog, IAM, data access controls)
Reliability & Resilience (job retries, checkpointing, failure isolation)
Performance Efficiency (cluster sizing, autoscaling, caching)
Cost Optimization (compute policies, workload separation, monitoring)
Operational Excellence (monitoring, automation, CI/CD, runbooks)
Lead Databricks workspace, cluster, and Unity Catalog implementations
Implement Delta Lake, Delta Live Tables (DLT), and Structured Streaming
Build and optimize ETL/ELT pipelines using Spark and SQL
Integrate Databricks with cloud services (S3/ADLS/GCS, IAM, Key Vault, networking)
Establish CI/CD pipelines for notebooks, jobs, and infrastructure
Implement Unity Catalog for centralized governance
Define data classification, lineage, and audit strategies
Enforce least-privilege access and secure networking patterns
Support compliance requirements (HIPAA, SOC 2, PCI, GDPR as applicable)
Monitor platform health, performance, and cost
Troubleshoot production issues across jobs, clusters, and data pipelines
Perform workload tuning and cost-performance optimization
Define SLOs, alerts, and operational metrics
Partner with Data Engineering, Analytics, ML, Platform, and Security teams
Translate business requirements into technical architectures
Provide architectural guidance and technical mentorship
Communicate risks, tradeoffs, and recommendations to leadership
Qualification
Required
7+ years in data engineering, analytics, or platform architecture
3–5+ years hands-on Databricks experience in production environments
Proven experience applying the Databricks Well-Architected Framework
Experience designing cloud-native lakehouse architectures
Experience supporting mission-critical data platforms
Databricks Lakehouse Platform
Apache Spark (PySpark / Scala / Spark SQL)
Delta Lake, Delta Live Tables, Structured Streaming
Unity Catalog (governance, lineage, access controls)
Cloud platforms: AWS, Azure, or GCP
Infrastructure as Code (Terraform strongly preferred)
CI/CD tools (GitHub Actions, Azure DevOps, GitLab, etc.)
Data formats and protocols (Parquet, JSON, Avro)
Databricks Certified Data Engineer Professional
Databricks Certified Professional Architect (or equivalent advanced certification)
Preferred
AWS Certified Solutions Architect (Associate or Professional)
Azure Solutions Architect Expert
Google Professional Data Engineer
Databricks Machine Learning Professional
Snowflake or other cloud data platform certifications
MLflow and MLOps architectures
Real-time analytics and streaming pipelines
Multi-region or cross-account data architectures
Consulting or MSP delivery experience
Company
Scicom Infrastructure Services
Scicom’s singular focus is to deliver high quality, reliable and cost effective technology solutions to support our client’s business objectives.
11-50 employees
H1B Sponsorship
Scicom Infrastructure Services has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2024 (2)
2022 (1)
2021 (2)
2020 (6)
Funding
Current Stage
Early StageCompany data provided by crunchbase