E-Solutions · 18 hours ago
AWS Databricks Data Engineer
Los Angeles, CA
Full-time
Hybrid
Mid, Senior LevelE-Solutions is seeking a highly skilled AWS Data Engineer with strong expertise in SQL, Python, and PySpark to join their data engineering team. The role involves designing, implementing, and optimizing large-scale data pipelines while collaborating with cross-functional teams to deliver efficient data solutions.
ConsultingHuman ResourcesRecruitingStaffing Agency
Responsibilities
Build and maintain scalable ETL/ELT pipelines using Databricks on AWS
Leverage PySpark/Spark and SQL to transform and process large, complex datasets
Integrate data from multiple sources including S3, relational/non-relational databases, and AWS-native services
Partner with downstream teams to prepare data for dashboards, analytics, and BI tools
Work closely with business stakeholders to understand requirements and deliver tailored, high‑quality data solutions
Optimize Databricks workloads for cost, performance, and efficient compute utilization
Monitor and troubleshoot pipelines to ensure reliability, accuracy, and SLA adherence
Apply query optimization, Spark tuning, and shuffle minimization best practices when handling tens of millions of rows
Implement and manage data governance, access control, and security policies using Unity Catalog
Ensure compliance with organizational and regulatory data‑handling standards
Use Databricks Asset Bundles for deployment of jobs, notebooks, and configuration across environments
Maintain effective version control of Databricks artifacts using GitLab or similar tools
Use CI/CD pipelines to support automated deployments and environment setups
Qualification
Required
Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Workflows, Delta Live Pipelines, Databricks Runtime, etc.)
Proven ability to implement robust PySpark solutions
Hands‑on experience with Databricks Workflows & orchestration
Solid knowledge of Medallion Architecture (Bronze/Silver/Gold)
Significant experience designing or rebuilding batch‑heavy data pipelines
Strong background in query optimization, performance tuning, and Spark shuffle optimization
Ability to handle and process tens of millions of records efficiently
Familiarity with Genie enablement concepts (understanding required; deep experience optional)
Experience with CI/CD, environment setup, and Git-based development workflows
Solid understanding of AWS cloud, including: IAM, Networking fundamentals, Storage integration (S3, Glue Catalog, etc.)
Preferred
Experience with Databricks Runtime configurations and advanced features
Knowledge of streaming frameworks such as Spark Structured Streaming
Experience developing real-time or near real-time data solutions
Exposure to GitLab pipelines or similar CI/CD systems
Databricks Certified Data Engineer Associate / Professional
AWS Data Engineer or AWS Solutions Architect certification
Company
E-Solutions
E-Solutions is a talent acquisition company that offers staffing, application development, and strategic outsourcing services.
Founded in 2003
1001-5000 employees
H1B Sponsorship
E-Solutions has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (6)
2024 (11)
2023 (12)
2022 (11)
2021 (12)
2020 (16)
Funding
Current Stage
Late StageLeadership Team
ANKIT MISHRA
Head – Google Cloud Platform (GCP) - Global Delivery Practice | Client Partnering | Cloud Computing
Recent News
2024-05-06
2023-12-14
Company data provided by crunchbase