LLM QA Test Engineer - Gen AI &LLM Applications @ orangepeople | Jobright.ai
JOBSarrow
RecommendedLiked
0
Applied
0
External
0
LLM QA Test Engineer - Gen AI &LLM Applications jobs in United States
74 applicants
company-logo

orangepeople ยท 5 hours ago

LLM QA Test Engineer - Gen AI &LLM Applications

positionUnited States
timeContract
remoteRemote
seniorityMid Level
money$82/hr - $86/hr
date3+ years exp
ftfMaximize your interview chances
EnterpriseInformation Services
check
Senior Management
check
H1B Sponsor Likelynote
Hiring Manager
Saravanan Palani
linkedin

Insider Connection @orangepeople

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Design and execute comprehensive test strategies for Large Language Models.
Develop sophisticated prompting techniques to evaluate LLM responses.
Create test cases that validate accurate knowledge of parks, resorts, and cruise operations.
Identify potential vulnerabilities and edge cases in guest-facing LLM interactions.
Document and track model behaviors, issues, and improvements.
Collaborate with AI/ML teams to improve model performance and safety.
Participate in OrangePeople monthly team meetings, and participate in team-building efforts.
Contribute to OrangePeople technical discussions, peer reviews, etc.
Contribute content and collaborate via the OP-Wiki/Knowledge Base.
Provide status reports to OP Account Management as requested.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Quality AssuranceAI/ML systemsLLM evaluation methodologiesPrompt engineeringTest automationVersion control systemsBug-tracking toolsPythonAI safetyEthical considerationsDocumentation skillsCross-functional collaboration

Required

3+ years of experience in Quality Assurance, with at least 1 year focused on AI/ML systems.
Demonstrated expertise in LLM evaluation methodologies and prompt engineering.
Strong understanding of LLM limitations, biases, and potential failure modes.
Excellent analytical and problem-solving skills.
Strong documentation and communication abilities.
Experience with LLM testing frameworks and evaluation metrics.
Knowledge of prompt engineering best practices.
Understanding of AI safety and ethical considerations.
Familiarity with version control systems and bug-tracking tools.
Basic programming skills for test automation (Python preferred).
Creative problem-solving for edge case discovery.
Attention to detail in identifying subtle model behaviors.
Strong analytical thinking for systematic testing approaches.
Ability to think like both a guest and an adversary.
Excellence in cross-functional collaboration.

Benefits

401(k)
Dental Insurance
Health insurance
Vision insurance

Company

orangepeople

twittertwitter
company-logo
OrangePeople specializes in providing digital transformation services.
dateFounded in 2006
location
Irvine, California, USA
people-size201-500 employees
web-link
http://www.orangepeople.com/

H1B Sponsorship

orangepeople has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2023 (1)
2022 (6)
2021 (6)
2020 (11)

Funding

Current Stage
Growth Stage

Leadership Team

leader-logo
Raghav Putrevu
Founder and CEO
linkedin
Company data provided by crunchbase
logo

Orion

Your AI Copilot