Tata Consultancy Services · 4 hours ago
Python/PySpark Engineer
Jersey City, NJ
Full-time
Onsite
Mid Level
$100K/yr - $125K/yrTata Consultancy Services is a leading global IT services, consulting, and business solutions organization. They are seeking a Python/PySpark Engineer to design and develop scalable ingestion and transformation pipelines while optimizing Spark jobs for performance and cost efficiency.
Business Information SystemsConsultingInformation TechnologyIT Management
Responsibilities
Design and develop scalable Python/PySpark ingestion and transformation pipelines
Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms
Optimize Spark jobs for performance, cost efficiency, and production readiness
Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance
Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability
Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions
Qualification
Required
Strong proficiency in Python, packaging, dependency management, and virtual environments
Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization)
Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic
Solid unit and integration testing practices, including data quality validations
Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning
Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks)
Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems
Secure coding practices including secrets management, PII protection, and compliance-aware development
Strong documentation discipline for frameworks, reusable components, and best-practice patterns
Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines
Clear communication of technical ideas and solution approaches
Comfort working in Agile environments with iterative development and frequent releases
BACHELOR OF COMPUTER SCIENCE
Benefits
Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
Company
Tata Consultancy Services
Tata Consultancy Services is a business solutions company that specializes on information technology services and consulting.
Founded in 1968
10001+ employees
H1B Sponsorship
Tata Consultancy Services has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (7880)
2024 (9690)
2023 (8537)
2022 (11159)
2021 (9813)
2020 (11984)
Funding
Current Stage
Public CompanyTotal Funding
unknown2004-08-25IPO
Leadership Team
K. Krithivasan
Chief Executive Officer & Managing Director
Aarthi Subramanian
President and Chief Operating Officer
Recent News
2026-01-25
Business Standard India
2026-01-25
Company data provided by crunchbase