Happiest Minds Technologies · 18 hours ago
Azure Databricks Architect
United States
Full-time
Remote
Senior Level, Lead/Staff
15+ years expHappiest Minds Technologies is a Mindful IT Company focused on enabling digital transformation for enterprises. They are seeking a seasoned Databricks Architect to lead the architecture, design, and development of scalable ETL/ELT pipelines and drive DevOps automation in a cloud-native environment.
AnalyticsCloud ComputingSoftware
Responsibilities
Architect, design, and develop scalable ETL/ELT pipelines using Databricks, PySpark, and SQL
Define and implement secure, scalable, and cost-effective Databricks architecture strategy
Administer Databricks workspaces, clusters, and compute resources
Define and enforce cluster policies for:
Resource utilization
Autoscaling
Cost control
Role-based access
Perform performance tuning of:
Spark jobs
Databricks SQL queries
Notebooks
(optimize execution plans, partitions, caching, and I/O)
Develop optimized Databricks SQL queries, views, and materializations to support:
Tableau/BI dashboards
React and .NET applications via REST APIs
Ad-hoc and real-time analytics
Partner with frontend/backend teams to build query-optimized datasets using Delta Lake/Lakehouse patterns
Implement Unity Catalog for:
Fine-grained access control
Lineage
Metadata governance
Data discovery
Drive DevOps practices using Azure DevOps, Git, and CI/CD pipelines
Establish observability for pipelines:
Logging
Metrics
Lineage
Alerts
SLA/SLO definitions
Mentor junior engineers and conduct architecture/design reviews
Qualification
Required
15+ years overall industry experience
7+ years in data engineering with cloud-native architecture expertise
Strong hands-on experience with Databricks architecture & administration
Cluster/pool management
Cluster policies & autoscaling strategies
Deep knowledge of Spark performance tuning (shuffle, skew, AQE, partitioning, caching, broadcast)
Expertise in Databricks SQL
PySpark
Delta Lake
Batch & streaming pipelines
Experience building Python libraries using Pandas, Openpyxl, XlsxWriter, PySpark
Practical experience with Unity Catalog governance
Strong SQL skills; experience with PostgreSQL/SQL Server or similar RDBMS
DevOps experience with Azure DevOps, Git, CI/CD automation
Experience collaborating with frontend/backend teams and REST API integrations
Excellent communication and stakeholder management skills
Preferred
Azure (preferred) and/or AWS cloud data services experience
Data modeling (dimensional/semantic)
Query acceleration, caching, and materialization strategies
Databricks FinOps/cost optimization (cluster sizing, Photon, serverless, job routing)
Observability practices and incident management/runbooks
Experience with Microsoft Fabric
Company
Happiest Minds Technologies
Happiest Minds provides IT services focused on cloud computing, social media, mobility solutions, BI, analytics, and unified communications.
5001-10000 employees
H1B Sponsorship
Happiest Minds Technologies has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (41)
2024 (46)
2023 (36)
2022 (59)
2021 (39)
Funding
Current Stage
Public CompanyTotal Funding
$105.92M2023-07-18Post Ipo Equity· $60.92M
2020-09-17IPO
2011-11-16Series A· $45M
Leadership Team
Joseph Anantharaju
Co-Chairman, CEO and Co-Founder
Ram Mohan C
President and CEO
Recent News
Business Standard India
2026-01-06
Company data provided by crunchbase