Jobs via Dice ยท 2 days ago
Backend Software engineer
Orlando, FL
Full-time
Onsite
Mid Level
$84/hr - $88/hrJobs via Dice is seeking an experienced software engineer to build tools and solutions for AI evaluations and content aggregation. The role involves serving as a technical liaison with operational product partners, optimizing the platform, and driving solutions for search and answer issues.
Computer Software
Responsibilities
Work with product partners to design and run experiments for changes to models, prompts, content, retrieval strategies, guardrails, and other variables; track and report metrics using internal tooling and industry-standard AI observability platforms
Build or enhance tooling for content gap detection, retrieval diagnostics, prompt evaluation, and automated guardrails to minimize poor/unexpected outputs
Help define and monitor key KPIs (e.g., precision/recall, MRR/NDCG, answer accuracy, coverage, and latency) and maintain dashboards that reflect quality and reliability over time
Investigate and determine root cause for specific search and answer issues in partnership with product and operations (e.g., content gaps, retrieval/pipeline problems, prompt quality, code bugs, configuration or business rule errors)
Implement fixes directly or coordinate with the appropriate engineering and content teams to mitigate issues and prevent regressions
Identify classes of issues, propose product improvements, and design technical solutions to improve relevance, answer quality, and recommendation performance at scale
Own triage in a hypercare role after releases: ensure issues are logged, prioritized, assigned, and driven to closure with clear communication and SLAs
Document findings, playbooks, and runbooks; create repeatable processes for RCA, experimentation, and hypercare operations
Qualification
Required
Strong software engineering skills in Java and Python, with experience building back-end services, APIs, pipelines, and integrations
Hands-on experience with search technologies and Retrieval-Augmented Generation (RAG), including vector embeddings, chunking strategies, and prompt building
Practical knowledge of when and how to use LLMs to improve efficiency and quality, including prompt design/tuning, guardrails, and fallback strategies
Proven root cause analysis (RCA) capabilities: ability to trace issues across data, retrieval, prompts, models, and code; skilled at identifying patterns and proposing systemic fixes
Excellent communication, relationship-building, and cross-team coordination skills; comfortable serving as a liaison between product operations and engineering
Experience with experimentation frameworks and A/B testing, defining success metrics, and interpreting results to drive decisions
Familiarity with AI observability and monitoring tools for prompt/model performance, drift detection, and error tracking
Proficiency with CI/CD, version control, automated testing, and feature flagging
Strong bias for action; able to manage multiple priorities in a fast-moving environment and follow through from issue detection to mitigation
Benefits
Medical
Prescription
Dental
Vision
AD&D
Life insurance offerings
Short-term disability
401K plan
Company
Jobs via Dice
Welcome to Jobs via Dice, the go-to destination for discovering the tech jobs you want.
0-1 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase