Milestone Technologies, Inc. · 3 hours ago
GenAI/LLM/AI QA Test Engineer (REMOTE & W2 ONLY)
Santa Monica, CA
Contract
Remote
Mid Level
$80/hr - $88/hr
3+ years exp
Maximize your interview chancesApplication Performance ManagementConsumer Electronics
Hiring Manager
Aishwariya Ray
Insider Connection @Milestone Technologies, Inc.
Get 3x more responses when you reach out via email instead of LinkedIn.
Responsibilities
Design and execute comprehensive test strategies for Large Language Models
Develop sophisticated prompting techniques to evaluate LLM responses about Disney Experience offerings
Create test cases that validate accurate knowledge of parks, resorts, and cruise operations
Identify potential vulnerabilities and edge cases in guest-facing LLM interactions
Document and track model behaviors, issues, and improvements
Collaborate with AI/ML teams to improve model performance and safety
Qualification
Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.
Required
3+ years of experience in Quality Assurance, with at least 1 year focused on AI/ML systems
Demonstrated expertise in LLM evaluation methodologies and prompt engineering
Strong understanding of LLM limitations, biases, and potential failure modes
Excellent analytical and problem-solving skills
Strong documentation and communication abilities
Experience with LLM testing frameworks and evaluation metrics
Knowledge of prompt engineering best practices
Understanding of AI safety and ethical considerations
Familiarity with version control systems and bug tracking tools
Basic programming skills for test automation (Python preferred)
Creative problem-solving for edge case discovery
Attention to detail in identifying subtle model behaviors
Strong analytical thinking for systematic testing approaches
Ability to think like both a guest and an adversary
Excellence in cross-functional collaboration
Benefits
Comprehensive benefit options
Company
Milestone Technologies, Inc.
Milestone Technologies is a global IT Services and Digital Solutions company based in Silicon Valley that helps hundreds of leading corporations deliver technology around the globe.
1,001-5,000 employees
H1B Sponsorship
Milestone Technologies, Inc. has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2022 (2)
2021 (3)
2020 (4)
Funding
Current Stage
Late StageTotal Funding
$42.5MKey Investors
H.I.G. Capital
2022-12-13Acquired· undefined
2015-08-11Private Equity· $42.5M
Leadership Team
Sameer Kishore
President and Chief Executive Officer
Mayank K Agrawal
Chief Financial Officer
Company data provided by crunchbase
