UST · 1 day ago
Python Data Engineer
United States
Full-time
Remote
Mid Level
$75K/yr - $113K/yr
4+ years expUST is a mission-driven technology company that transforms lives through innovative solutions. They are seeking a Python Data Engineer to design and optimize PySpark-based data pipelines on AWS, collaborating with various teams to ensure efficient data processing and deployment automation.
ConsultingInformation TechnologyInformation Services
Responsibilities
Design, build, and optimize PySpark-based data pipelines (batch & streaming) on AWS
Tune Spark jobs for performance, reliability, and cost efficiency; monitor using Spark UI/CloudWatch
Collaborate with platform, data, and application teams to integrate pipelines with Glue/EMR/Lambda/Step Functions
Establish CI/CD for data workflows and ensure test coverage and deployment automation
Contribute to coding standards, documentation, and Agile ceremonies
Qualification
Required
4+ years of professional experience in software development with a strong focus on Python
Solid understanding of core Python concepts, data structures, algorithms, and design patterns
Proficiency in Python for scripting, automation, backend services, and data-processing workflows
Data modeling for analytics (medallion architecture: bronze/silver/gold), Parquet/Avro/JSON best practices
Hands-on expertise with PySpark, including: Working with DataFrames/Datasets and Spark SQL
ETL/ELT pipeline development for large-scale, batch and near-real-time workloads
Expertise and hands on experience in Performance tuning & optimization
Hands on experience on Spark Streaming
Excellent knowledge of Lakehouse & table formats: Delta Lake (preferred), Apache Hudi or Apache Iceberg
Expertise in Data quality & validation
Excellent knowledge of Pandas
AWS hands-on experience with a strong understanding of cloud principles, including: AWS Glue (ETL jobs, Spark jobs, Glue Studio/Workflows, Glue Data Catalog) and AWS Lambda for serverless integrations
Amazon EMR (cluster sizing, autoscaling, cost optimization with Spot, versioned runtimes)
Amazon S3 (data lake layout, partitioning, lifecycle policies
Orchestration & monitoring: AWS Step Functions, Amazon MWAA/Airflow, CloudWatch Logs/Metrics/Alarms
Experience with Agile development methodologies
Familiarity with CI/CD concepts and tooling such as AWS CodePipeline/CodeBuild/CodeDeploy; infrastructure as code (CloudFormation/Terraform) is a plus
Testing & code quality: unit/integration testing for Spark (pytest, chispa), code reviews, PEP 8, type hints/mypy
Strong problem solving, analytical, and communication skills
Ability to work independently and collaboratively in a team environment
Preferred
Knowledge of Java and the Spring framework
Databricks on AWS: Jobs, clusters, notebooks, Repos, Delta Live Tables, Unity Catalog
Experience with catalog governance and row/column-level security
Exposure to cost/performance governance (e.g., file compaction, small-files mitigation, Z-Ordering for Delta)
Knowledge of REST APIs integration and message-based architectures
Benefits
Accrue a minimum of 10 days of paid vacation per year
Receive 6 days of paid sick leave each year (pro-rated for new hires throughout the year)
10 paid holidays
Eligible for paid bereavement leave and jury duty
Eligible to participate in the Company’s 401(k) Retirement Plan with employer matching
Eligible for medical, dental, and vision insurance
Basic life insurance
Accidental death and disability insurance
Short- and long-term disability benefits
May purchase additional voluntary short-term disability benefits
Participate in a Health Savings Account (HSA)
Flexible Spending Account (FSA) for healthcare, dependent child care, and/or commuting expenses as allowable under IRS guidelines
Company
UST
UST is a Digital Transformations Solutions Provider.
10001+ employees
H1B Sponsorship
UST has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (870)
2024 (800)
2023 (647)
2022 (634)
2021 (612)
2020 (984)
Funding
Current Stage
Late StageTotal Funding
$250MKey Investors
Temasek Holdings
2018-06-27Private Equity· $250M
Leadership Team
Krishna Sudheendra
CEO
Alexander Varghese
Chief Administrative Officer & COO
Recent News
Orange County Business Journal
2026-01-24
2026-01-20
Company data provided by crunchbase