HCL Global Systems Inc ยท 12 hours ago
IoT-AUTO- Databricks DevOps Engineer
Dallas, TX
Full-time
Onsite
Senior Level
5+ years expHCL Global Systems Inc is seeking an IoT-AUTO- Databricks DevOps Engineer to design, develop, and maintain a high-scale, cloud-based data platform. The role involves managing Databricks workspaces, AWS infrastructure, and implementing CI/CD practices to ensure efficient deployments and system reliability.
B2BConsultingHuman ResourcesInformation TechnologySoftwareStaffing Agency
Responsibilities
Manage users, groups, clusters, jobs, notebooks, and monitor performance within Databricks workspaces
Provision and manage AWS resources like S3, EC2, VPCs, IAM, Lambda, and CloudWatch to support Databricks
Implement and maintain infrastructure using tools like Terraform or CloudFormation for automated deployments
Develop automation scripts (Python, SQL) and integrate with CI/CD pipelines (Jenkins, GitHub Actions) for efficient deployments
Harden the platform, manage access controls (IAM), and ensure compliance with security best practices
Right-size clusters, optimize Spark jobs, manage caching, and monitor costs (DBUs, storage)
Design, build, and optimize scalable data pipelines and ETL/ELT processes using Spark and Delta Lake
Design, build, and maintain scalable, efficient, and reliable ETL/ELT data pipelines to support data ingestion, transformation, and integration across diverse sources
Serve as the subject matter expert for the Databricks environment, developing high-performance data transformation logic primarily using PySpark and Python
Configure, maintain, and secure the underlying AWS cloud infrastructure required to run the Databricks platform, including virtual private clouds (VPCs), network endpoints, storage (S3), and cross-account access mechanisms
Own and enforce Continuous Integration/Continuous Deployment (CI/CD) practices for the data platform
Design and implement automated unit, integration, and performance testing frameworks to ensure data quality, reliability, and compliance with architectural standards
Optimize data workflows and cluster configurations for performance, cost efficiency, and scalability across massive datasets
Provide technical guidance on data principles, patterns, and best practices to promote team capabilities and maturity
Draft and review architectural diagrams, design documents, and interface specifications to ensure clear communication of data solutions and technical requirements
Qualification
Required
Databricks Administration: Manage users, groups, clusters, jobs, notebooks, and monitor performance within Databricks workspaces
AWS Infrastructure: Provision and manage AWS resources like S3, EC2, VPCs, IAM, Lambda, and CloudWatch to support Databricks
Infrastructure as Code (IaC): Implement and maintain infrastructure using tools like Terraform or CloudFormation for automated deployments
Automation & CI/CD: Develop automation scripts (Python, SQL) and integrate with CI/CD pipelines (Jenkins, GitHub Actions) for efficient deployments
Security: Harden the platform, manage access controls (IAM), and ensure compliance with security best practices
Performance Optimization: Right-size clusters, optimize Spark jobs, manage caching, and monitor costs (DBUs, storage)
Experience: 5+ years of professional experience in Data Engineering, focusing on building scalable data platforms and production pipelines
Big Data Expertise: Minimum 3+ years of hands-on experience developing, deploying, and optimizing solutions within the Databricks ecosystem
AWS Infrastructure & Security: Proven, hands-on experience (3+ years) with core AWS services and infrastructure components
Programming: Expert proficiency (4+ years) in Python for data manipulation, scripting, and pipeline development
Spark & SQL: Deep understanding of distributed computing and extensive experience (3+ years) with PySpark and advanced SQL for complex data transformation and querying
DevOps & CI/CD: Proven experience (2+ years) designing and implementing CI/CD pipelines
Data Concepts: Full understanding of ETL/ELT, Data Warehousing, and Data Lake concepts
Methodology: Strong grasp of Agile principles (Scrum)
Version Control: Proficiency with Git for version control
Preferred
AWS Data Ecosystem Experience: Familiarity and experience with AWS cloud-native data services
Knowledge of real-time or near-real-time streaming technologies (e.g., Kafka, Spark Structured Streaming)
Experience in developing feature engineering pipelines for machine learning (ML) consumption
Background in performance tuning and capacity planning for large Spark clusters
Company
HCL Global Systems Inc
HCL Global Systems is a staffing and recruiting company providing consulting and business solutions.
1001-5000 employees
H1B Sponsorship
HCL Global Systems Inc has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (265)
2024 (285)
2023 (378)
2022 (390)
2021 (470)
2020 (736)
Funding
Current Stage
Late StageLeadership Team
A
Andy HCL
Cheif Marketing Officer
Company data provided by crunchbase