Tata Consultancy Services · 2 days ago
AI EVAL Engineering
Redmond, WA
Full-time
Onsite
Mid Level
$64K/yr - $110K/yrTata Consultancy Services is a leading technology services and consulting company, and they are seeking an AI EVAL Engineer. The role involves evaluating and benchmarking AI models, creating structured test suites, and applying evaluation metrics to ensure model accuracy and performance.
Business Information SystemsConsultingInformation TechnologyIT Management
Responsibilities
Strong understanding of LLMs and generative AI concepts, including model behavior and output evaluation
Experience with AI evaluation and benchmarking methodologies, including baseline creation and model comparison
Hands-on expertise in Eval testing, creating structured test suites to measure accuracy, relevance, safety, and performance
Ability to define and apply evaluation metrics (precision/recall, BLEU/ROUGE, F1, hallucination rate, latency, cost per output)
Prompt engineering and prompt testing experience across zero-shot, few-shot, and system prompt scenarios
Python/other programming languages, for automation, data analysis, batch evaluation execution, and API integration
Experience with evaluation tools/frameworks (OpenAI Evals, HuggingFace evals, Promptfoo, Ragas, DeepEval, LM Eval Harness)
Ability to create datasets, test cases, benchmarks, and ground truth references for consistent scoring
Test design and test automation experience, including reproducible evaluation pipelines
Knowledge of AI safety, bias, security testing, and hallucination analysis
Qualification
Required
Azure OpenAI; EVAL; Bench Marking
Strong understanding of LLMs and generative AI concepts, including model behavior and output evaluation
Experience with AI evaluation and benchmarking methodologies, including baseline creation and model comparison
Hands-on expertise in Eval testing, creating structured test suites to measure accuracy, relevance, safety, and performance
Ability to define and apply evaluation metrics (precision/recall, BLEU/ROUGE, F1, hallucination rate, latency, cost per output)
Prompt engineering and prompt testing experience across zero-shot, few-shot, and system prompt scenarios
Python/other programming languages, for automation, data analysis, batch evaluation execution, and API integration
Experience with evaluation tools/frameworks (OpenAI Evals, HuggingFace evals, Promptfoo, Ragas, DeepEval, LM Eval Harness)
Ability to create datasets, test cases, benchmarks, and ground truth references for consistent scoring
Test design and test automation experience, including reproducible evaluation pipelines
Knowledge of AI safety, bias, security testing, and hallucination analysis
Preferred
RAG evaluation experience
Azure OpenAI / OpenAI / Anthropic / Google AI platforms
Performance benchmarking (speed, throughput, cost)
Domain knowledge: Office apps / enterprise systems / networking
Benefits
Discretionary Annual Incentive.
Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
Family Support: Maternal & Parental Leaves.
Insurance Options: Auto & Home Insurance, Identity Theft Protection.
Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
Time Off: Vacation, Time Off, Sick Leave & Holidays.
Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
Company
Tata Consultancy Services
Tata Consultancy Services is a business solutions company that specializes on information technology services and consulting.
Founded in 1968
10001+ employees
H1B Sponsorship
Tata Consultancy Services has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (7880)
2024 (9690)
2023 (8537)
2022 (11159)
2021 (9813)
2020 (11984)
Funding
Current Stage
Public CompanyTotal Funding
unknown2004-08-25IPO
Leadership Team
K. Krithivasan
Chief Executive Officer & Managing Director
Aarthi Subramanian
President and Chief Operating Officer
Recent News
2026-01-18
Hindu Business Line
2026-01-18
Company data provided by crunchbase