Randstad Digital Americas ยท 11 hours ago
Backend Software engineer
Orlando, FL
Full-time
Onsite
Mid Level
$84/hr - $88/hrRandstad Digital Americas is seeking an experienced software engineer to build tools and solutions for AI evaluations and improve business rules. The role involves serving as a technical liaison with product partners, investigating issues, running experiments, and ensuring quality and reliability in a hypercare environment.
Information Technology & Services
Responsibilities
Work with product partners to design and run experiments for changes to models, prompts, content, retrieval strategies, guardrails, and other variables; track and report metrics using internal tooling and industry-standard AI observability platforms
Build or enhance tooling for content gap detection, retrieval diagnostics, prompt evaluation, and automated guardrails to minimize poor/unexpected outputs
Help define and monitor key KPIs (e.g., precision/recall, MRR/NDCG, answer accuracy, coverage, and latency) and maintain dashboards that reflect quality and reliability over time
Investigate and determine root cause for specific search and answer issues in partnership with product and operations (e.g., content gaps, retrieval/pipeline problems, prompt quality, code bugs, configuration or business rule errors)
Implement fixes directly or coordinate with the appropriate engineering and content teams to mitigate issues and prevent regressions
Identify classes of issues, propose product improvements, and design technical solutions to improve relevance, answer quality, and recommendation performance at scale
Own triage in a hypercare role after releases: ensure issues are logged, prioritized, assigned, and driven to closure with clear communication and SLAs
Document findings, playbooks, and runbooks; create repeatable processes for RCA, experimentation, and hypercare operations
Qualification
Required
Strong software engineering skills in Java and Python, with experience building back-end services, APIs, pipelines, and integrations
Hands-on experience with search technologies and Retrieval-Augmented Generation (RAG), including vector embeddings, chunking strategies, and prompt building
Practical knowledge of when and how to use LLMs to improve efficiency and quality, including prompt design/tuning, guardrails, and fallback strategies
Proven root cause analysis (RCA) capabilities: ability to trace issues across data, retrieval, prompts, models, and code; skilled at identifying patterns and proposing systemic fixes
Excellent communication, relationship-building, and cross-team coordination skills; comfortable serving as a liaison between product operations and engineering
Experience with experimentation frameworks and A/B testing, defining success metrics, and interpreting results to drive decisions
Familiarity with AI observability and monitoring tools for prompt/model performance, drift detection, and error tracking
Proficiency with CI/CD, version control, automated testing, and feature flagging
Strong bias for action; able to manage multiple priorities in a fast-moving environment and follow through from issue detection to mitigation
Benefits
Medical
Prescription
Dental
Vision
AD&D
Life insurance offerings
Short-term disability
401K plan
Company
Randstad Digital Americas
Randstad Digital is a trusted digital enablement partner that facilitates accelerated transformation for businesses by providing global talent, capacity, and solutions across specialized domains.
Founded in 1984
10001+ employees
Funding
Current Stage
Late StageLeadership Team
Graig Paglieri
CEO, Randstad Digital Americas
Pascal de Hesselle
SVP, Executive Client Partner - Technology, Media & Telecom
Company data provided by crunchbase