This job has closed.

Shopint · 1 month ago

Founding Eval test case Lead / eval owner/Cofounder-Track candidate (5-10hr pre week)

United States

Full-time

Remote

Lead/Staff

Shopint is an innovative company focused on enhancing AI shopping systems for better decision-making in e-commerce. They are seeking a Founding Eval Method Lead to develop evaluation methods for AI product-checking systems and guide stakeholders while working approximately 5-10 hours per week.

Computer Software

Hiring Manager

bufan shen

Responsibilities

Reviewing and improving current AI shopping eval test case cards

Tightening scenario boundaries and grounding context

Improving eval methodology

Identifying weak, subjective, or non-judgeable criteria

Helping prepare scenario performance cards for design-partner review

Joining selected design-partner / advisor / investor-style calls to explain the eval methodology

Qualification

LLM evaluationAI quality evaluationModel output evaluationHuman judgment workflowsAnnotation guideline designRubric designRegression test case designPrompt response quality evaluationReviewer calibrationGolden examplesBenchmark examplesFailure taxonomy designSafety evaluationTrust evaluationReliability evaluationQA process design for AI systems

Required

LLM evaluation

AI quality evaluation

model output evaluation

human judgment workflows

annotation guideline design

rubric design

regression test case design

prompt / response quality evaluation

reviewer calibration

golden examples or benchmark examples

failure taxonomy design

safety / trust / reliability evaluation

QA process design for AI systems

Company

Shopint

Shopint helps AI shopping teams improve product-checking quality under uncertainty, reducing overconfidence, false reliance, silent-risk exposure, and premature user action when evidence is limited or key conditions remain unverified.

2-10 employees

Funding

Current Stage

Early Stage

Company data provided by crunchbase