VySystems · 7 hours ago
Data Architect
Los Angeles, CA
Full-time
Hybrid
Senior Level
7+ years expVySystems is seeking a Databricks Architect who will be responsible for designing, implementing, and optimizing scalable data analytics and data engineering solutions on the Databricks Lakehouse Platform. This role involves collaborating with cross-functional teams to define data strategies and ensure platform reliability while enabling advanced analytics and machine learning use cases.
AppsConsultingDigital MarketingInformation TechnologyInfrastructureIT InfrastructureIT ManagementWeb Development
Hiring Manager
Devendra Kumar
Responsibilities
Design end-to-end Databricks Lakehouse architectures for data ingestion, processing, storage, and consumption
Define and implement Delta Lake patterns, including medallion architecture (Bronze/Silver/Gold)
Develop scalable data pipelines using PySpark, Spark SQL, and Databricks workflows
Architect solutions for structured, semi-structured, and unstructured data
Build robust ETL/ELT pipelines with Databricks notebooks, jobs, and workflows
Design and implement high-performance streaming solutions using Structured Streaming
Optimize Spark jobs for cost, performance, and scalability
Implement CI/CD and automation using Databricks Repos, Git, and DevOps pipelines
Architect solutions across Azure/AWS/GCP leveraging native cloud services (e.g., Azure Data Factory, AWS Glue, GCP Dataflow)
Ensure security, governance, and compliance through Unity Catalog, RBAC, and encryption
Monitor workloads and optimize cluster configurations for performance and cost
Work closely with data engineers, data scientists, BI teams, and business stakeholders
Act as a subject matter expert (SME) for Databricks best practices, standards, and patterns
Conduct architectural reviews and guide teams on design decisions
Lead PoCs, evaluate new features, and drive platform adoption
Define standards for data quality, lineage, observability, and governance
Implement automated testing frameworks for pipelines and notebooks
Establish performance baselines and monitoring dashboards
Qualification
Required
7+ years of experience in data engineering/architecture
3+ years of hands-on experience with Databricks
Strong expertise in Spark, PySpark, SQL, and distributed data processing
Deep understanding of Delta Lake features: ACID transactions, OPTIMIZE, ZORDER, Auto Loader
Experience with workflow orchestration, jobs, and Databricks REST APIs
Hands-on expertise with at least one cloud platform: Azure, AWS, or GCP
Strong analytical and problem-solving skills
Excellent communication and stakeholder management
Ability to lead design discussions and guide technical teams
Strong documentation and architectural blueprinting skills
Preferred
Azure (preferred): ADF, ADLS, Key Vault, Event Hub, Azure DevOps
Familiarity with CI/CD, Git, DevOps, and Infrastructure-as-Code (Terraform preferred)
Company
VySystems
Vy Systems is a part of vy.ventures and is in the business of Technology consulting, Solutions, and Managed Services, providing invaluable services across many countries since 2002.
501-1000 employees
Funding
Current Stage
Late StageLeadership Team
Ramesh Santhanam
Founder and CSO
Company data provided by crunchbase