Persistent Systems · 17 hours ago
Databricks Architect
LA Metro Area
Full-time
Hybrid
Senior Level, Lead/Staff
12+ years expPersistent Systems is an AI-led, platform-driven Digital Engineering and Enterprise Modernization partner, recognized for its innovation and leadership. The Databricks Architect will design, implement, and optimize scalable data analytics solutions on the Databricks Lakehouse Platform, collaborating with cross-functional teams to define data strategies and ensure platform reliability.
AnalyticsBig DataCloud ComputingEmbedded SystemsProduct DesignSecuritySoftware
Hiring Manager
Shafak Pathan
Responsibilities
Design end-to-end Databricks Lakehouse architectures for data ingestion, processing, storage, and consumption
Define and implement Delta Lake patterns, including medallion architecture (Bronze/Silver/Gold)
Develop scalable data pipelines using PySpark, Spark SQL, and Databricks workflows
Architect solutions for structured, semi-structured, and unstructured data
Build robust ETL/ELT pipelines with Databricks notebooks, jobs, and workflows
Design and implement high-performance streaming solutions using Structured Streaming
Optimize Spark jobs for cost, performance, and scalability
Implement CI/CD and automation using Databricks Repos, Git, and DevOps pipelines
Architect solutions across Azure/AWS/GCP leveraging native cloud services (e.g., Azure Data Factory, AWS Glue, GCP Dataflow)
Ensure security, governance, and compliance through Unity Catalog, RBAC, and encryption
Monitor workloads and optimize cluster configurations for performance and cost
Work closely with data engineers, data scientists, BI teams, and business stakeholders
Act as a subject matter expert (SME) for Databricks best practices, standards, and patterns
Conduct architectural reviews and guide teams on design decisions
Lead PoCs, evaluate new features, and drive platform adoption
Define standards for data quality, lineage, observability, and governance
Implement automated testing frameworks for pipelines and notebooks
Establish performance baselines and monitoring dashboards
Qualification
Required
12+ years of experience
7+ years of experience in data engineering/architecture
3+ years of hands-on experience with Databricks
Strong expertise in Spark, PySpark, SQL, and distributed data processing
Deep understanding of Delta Lake features: ACID transactions, OPTIMIZE, ZORDER, Auto Loader
Experience with workflow orchestration, jobs, and Databricks REST APIs
Hands-on expertise with at least one cloud platform: Azure, AWS, or GCP
Strong analytical and problem-solving skills
Excellent communication and stakeholder management
Ability to lead design discussions and guide technical teams
Strong documentation and architectural blueprinting skills
Preferred
Azure (preferred): ADF, ADLS, Key Vault, Event Hub, Azure DevOps
AWS: S3, Glue, Lambda, Kinesis
GCP: GCS, Dataflow, Pub/Sub
Familiarity with CI/CD, Git, DevOps, and Infrastructure-as-Code (Terraform preferred)
Benefits
Competitive salary and benefits package
Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
Opportunity to work with cutting-edge technologies
Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
Annual health check-ups
Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents
Company
Persistent Systems
Persistent Systems offers software product concept and design, performance engineering, quality assurance, and other professional services.
10001+ employees
H1B Sponsorship
Persistent Systems has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (519)
2024 (840)
2023 (475)
2022 (308)
2021 (285)
2020 (397)
Funding
Current Stage
Public CompanyTotal Funding
$18.8M2010-04-06IPO
2005-12-09Series A· $18.8M
Leadership Team
Sandeep Kalra
CEO & Executive Director
Rajiv Korpal
SVP, Microsoft Business Unit
Recent News
2026-01-22
Business Standard India
2026-01-22
Business Standard India
2026-01-22
Company data provided by crunchbase