Apply on Employer Site

Akkodis Group Nordics · 3 days ago

AI EVAL Engineer

United States

Full-time

Remote

Mid, Senior Level

$50/hr - $53/hr

3+ years exp

Akkodis Group Nordics is seeking an AI EVAL Engineer for a contract role. The engineer will be responsible for designing and automating evaluation tests for AI models, defining metrics, and conducting performance analysis while ensuring AI safety and quality.

Embedded SystemsSoftware

Responsibilities

Design, implement, and automate evaluation test suites to measure LLM accuracy, relevance, safety, latency, and cost across zero-shot, few-shot, and system-prompt scenarios

Define and apply robust evaluation metrics (e.g., precision/recall, BLEU/ROUGE, F1, hallucination rate, throughput, cost-per-output) and establish reproducible baselines for model comparison

Build datasets, ground-truth references, and benchmarks, and maintain versioned test cases for consistent, repeatable scoring

Develop batch evaluation pipelines in Python (and other languages as needed) with API integrations, integrating frameworks like OpenAI Evals, HuggingFace evals, Promptfoo, Ragas, DeepEval, or LM Eval Harness

Conduct performance benchmarking and analysis across Azure OpenAI (and other providers), reporting insights on speed, scalability, and resource efficiency

Assess and mitigate AI safety, bias, and hallucination risks, while collaborating with product, research, and platform teams to improve prompts, guardrails, and overall model quality

Qualification

PythonAI evaluation frameworksAI/ML evaluation experienceAzure OpenAIBenchmarkingTest automationData ScienceCollaboration

Required

Bachelor's or master's in computer science, Data Science, AI/ML, or related field

3–5+ years in AI/ML evaluation, benchmarking, or applied ML (including LLMs and generative AI)

Strong Python skills with hands-on experience in evaluation frameworks (e.g., OpenAI Evals, Hugging Face evals, Promptfoo, Ragas, DeepEval, LM Eval Harness) and defining/applying metrics (precision/recall, BLEU/ROUGE, F1, hallucination rate, latency, cost)

Practical experience with Azure OpenAI (and/or OpenAI/Anthropic/Google AI), test automation pipelines, and benchmarking across zero-/few-shot prompts