Tata Consultancy Services · 6 hours ago
Data Engineering Lead
New York, NY
Full-time
Onsite
Senior Level, Lead/Staff
$135K/yr - $150K/yr
4+ years expTata Consultancy Services is a global leader in IT services, and they are seeking a Data Engineering Lead. The role involves designing and operating data pipelines, ensuring data quality and governance, and collaborating with stakeholders to deliver reliable datasets for analytics and AI use cases.
Business Information SystemsConsultingInformation TechnologyIT Management
Responsibilities
Ingest and model data from APIs, files/SFTP, and relational sources; implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python
Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks
Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute; manage jobs, secrets, and access
Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, policies, versioning, quotas, and monitoring
Enforce data quality and governance through data contracts, validations/tests, lineage, observability, and proactive alerting
Optimize performance and cost via partitioning, clustering, query tuning, job sizing, and workload management
Uphold security and compliance (e.g., PII handling, encryption, masking) in line with firm standards
Collaborate with stakeholders (analytics, AI engineering, and business teams) to translate requirements into reliable, production ready datasets
Enable AI/LLM use cases by packaging datasets and metadata for downstream consumption, integrating via Model Context Protocol (MCP) where appropriate
Continuously improve platform reliability and developer productivity by automating routine tasks, reducing technical debt, and maintaining clear documentation
Qualification
Required
AWS Data Engineering Services (EMR/Glue, Redshift, Aurora, S3, Lambda)
Spark
Python
Collibra
Snowflake/Databricks
Tableau
Ingest and model data from APIs, files/SFTP, and relational sources; implement layered architectures (raw/clean/serving) using PySpark/SQL and dbt, Python
Design and operate pipelines with Prefect (or Airflow), including scheduling, retries, parameterization, SLAs, and well documented runbooks
Build on cloud data platforms, leveraging S3/ADLS/GCS for storage and a Spark platform (e.g., Databricks or equivalent) for compute; manage jobs, secrets, and access
Publish governed data services and manage their lifecycle with Azure API Management (APIM) authentication/authorization, policies, versioning, quotas, and monitoring
Enforce data quality and governance through data contracts, validations/tests, lineage, observability, and proactive alerting
Optimize performance and cost via partitioning, clustering, query tuning, job sizing, and workload management
Uphold security and compliance (e.g., PII handling, encryption, masking) in line with firm standards
Collaborate with stakeholders (analytics, AI engineering, and business teams) to translate requirements into reliable, production ready datasets
Enable AI/LLM use cases by packaging datasets and metadata for downstream consumption, integrating via Model Context Protocol (MCP) where appropriate
Continuously improve platform reliability and developer productivity by automating routine tasks, reducing technical debt, and maintaining clear documentation
4–15 years of professional data engineering experience
Strong Python, SQL, and Spark (PySpark) skills, and/or Kafka
Snowflake (Snowpipe, Tasks, Streams) as a complementary warehouse
Databricks (Delta formats, workflows, cataloging) or equivalent Spark platforms
Hands-on experience building ETL/ELT with Prefect (or Airflow), dbt, Spark, and/or Kafka
Experience onboarding datasets to cloud data platforms (storage, compute, security, governance)
Familiarity with Azure/AWS/GCP data services (e.g., S3/ADLS/GCS; Redshift/BigQuery; Glue/ADF)
Git-based workflows CI/CD and containerization with Docker (Kubernetes a plus)
Strategic Technical Leadership: Defining data architecture, evaluating new technologies, and setting technical standards for AWS-based pipelines
Stakeholder Communication: Bridging the gap between technical teams and business stakeholders, gathering requirements, and reporting progress
Risk Management: Proactively identifying potential bottlenecks in data workflows, security risks, or scalability issues
Operational Excellence: Implementing automation, optimizing costs, and maintaining high data quality standards
BACHELOR OF COMPUTER SCIENCE
Benefits
Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
Company
Tata Consultancy Services
Tata Consultancy Services is a business solutions company that specializes on information technology services and consulting.
10001+ employees
H1B Sponsorship
Tata Consultancy Services has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (7880)
2024 (9690)
2023 (8537)
2022 (11159)
2021 (9813)
2020 (11984)
Funding
Current Stage
Public CompanyTotal Funding
unknown2004-08-25IPO
Leadership Team
K. Krithivasan
Chief Executive Officer & Managing Director
Aarthi Subramanian
President and Chief Operating Officer
Recent News
2026-01-25
Business Standard India
2026-01-25
Company data provided by crunchbase