Sigmaways Inc · 1 day ago
Senior Data Engineer - Spark, Airflow
San Francisco Bay Area
Contract
Onsite
Senior Level
7+ years expSigmaways Inc is seeking an experienced Data Engineer to design and optimize scalable data pipelines that drive their global data and analytics initiatives. The role involves leveraging technologies such as Apache Spark, Airflow, and Python to build high-performance data processing systems while ensuring data quality, reliability, and lineage across Mastercard’s data ecosystem.
ConsultingDigital MarketingInformation TechnologySoftware
Responsibilities
Design and optimize Spark-based ETL pipelines for large-scale data processing
Build and manage Airflow DAGs for scheduling, orchestration, and checkpointing
Implement partitioning and shuffling strategies to improve Spark performance
Ensure data lineage, quality, and traceability across systems
Develop Python scripts for data transformation, aggregation, and validation
Execute and tune Spark jobs using spark-submit
Perform DataFrame joins and aggregations for analytical insights
Automate multi-step processes through shell scripting and variable management
Collaborate with data, DevOps, and analytics teams to deliver scalable data solutions
Qualification
Required
Bachelor's degree in Computer Science, Data Engineering, or related field (or equivalent experience)
At least 7 years of experience in data engineering or big data development
Strong expertise in Apache Spark architecture, optimization, and job configuration
Proven experience with Airflow DAGs using authoring, scheduling, checkpointing, monitoring
Skilled in data shuffling, partitioning strategies, and performance tuning in distributed systems
Expertise in Python programming including data structures and algorithmic problem-solving
Hands-on with Spark DataFrames and PySpark transformations using joins, aggregations, filters
Proficient in shell scripting, including managing and passing variables between scripts
Experienced with spark submit for deployment and tuning
Solid understanding of ETL design, workflow automation, and distributed data systems
Excellent debugging and problem-solving skills in large-scale environments
Preferred
Experience with AWS Glue, EMR, Databricks, or similar Spark platforms
Knowledge of data lineage and data quality frameworks like Apache Atlas
Familiarity with CI/CD pipelines, Docker/Kubernetes, and data governance tools
Company
Sigmaways Inc
We are one of the region's fastest-growing, multi-award-winning full-lifecycle product engineering service providers.
51-200 employees
H1B Sponsorship
Sigmaways Inc has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (2)
2024 (2)
2023 (1)
2022 (4)
2021 (11)
2020 (12)
Funding
Current Stage
Growth StageLeadership Team
Prakash Sadasivam
CEO
RAJEEV VERMA
Managing Partner
Recent News
GlobeNewswire News Room
2023-11-09
Company data provided by crunchbase