CloudTech Innovations · 2 weeks ago
Databricks Solution Architect
Chicago, IL
Contract
Onsite
Senior Level, Lead/Staff
10+ years expCloudTech Innovations is seeking a highly experienced Solution Architect with a strong background in building and scaling cloud-native data platforms. This role focuses on Databricks, Unity Catalog, and modern data engineering patterns, requiring collaboration with cross-functional teams to deliver well-architected solutions.
Information Technology & Services
No H1B
Security Clearance RequiredU.S. Citizen Only
Responsibilities
Design and lead implementation of end-to-end Databricks Lakehouse Platforms using Delta Lake, Delta Live Tables, and MLflow
Architect Medallion Architecture (Bronze/Silver/Gold) for structured, semi-structured, and streaming workloads
Implement governed Lakehouse patterns using Unity Catalog for access control, lineage, data classification, and secure sharing
Build scalable ETL/ELT pipelines using Databricks Notebooks, Workflows, SQL Warehouses, and Spark-based transformations
Develop real-time streaming pipelines with Auto Loader, Structured Streaming, and event-driven platforms (Kafka, Kinesis, Pub/Sub)
Integrate Databricks with cloud-native services such as AWS Glue, Azure Data Factory, and GCP Dataform
Define distributed integration patterns using REST APIs , microservices, and event-driven architectures
Enforce data governance, RBAC/ABAC, encryption, secret management , and compliance controls
Optimize Delta Lake tables, Spark workloads, and cluster configurations using Photon and autoscaling patterns
Drive cloud cost optimization across storage, compute, and workflow orchestration
Participate in architecture reviews, set standards, and support engineering teams throughout execution
Stay current on Databricks capabilities including Unity Catalog updates, Lakehouse Federation, serverless compute, and AI/ML features
Qualification
Required
Bachelor's or Master's degree in Computer Science, Data Engineering, or related field
Must hold an active U.S. Government Security Clearance
10+ years of experience in enterprise software, cloud architecture, or data engineering roles
Strong hands-on experience with Databricks, Apache Spark, Delta Lake, and Lakehouse platform design
Experience implementing and administering Unity Catalog for governance, lineage, and fine-grained access control
Experience designing Medallion Architecture for analytics and engineering workloads
Hands-on experience with cloud platforms such as AWS, Azure, or GCP, including storage, compute, and networking services
Experience with streaming technologies such as Kafka, Kinesis, or Pub/Sub
Strong understanding of data modeling, workflow orchestration (Airflow, Databricks Workflows, dbt), and pipeline automation
Familiarity with Scala-based Spark workloads in addition to PySpark and SQL pipelines
Skilled in performance tuning, Spark optimization, cluster policies, and cloud cost management
Excellent communication skills for technical leadership and stakeholder collaboration
Preferred
Certifications in Databricks, AWS/GCP/Azure Solution Architecture, or TOGAF are a plus
Company
CloudTech Innovations
Comprehensive Staffing Solutions for Specialized Engineering Roles: Software Development Engineering: We offer staffing solutions with skilled software engineers proficient in various programming languages and technologies.
51-200 employees
Funding
Current Stage
Growth StageCompany data provided by crunchbase