Apply on Employer Site

Acentra Health · 16 hours ago

Artificial Intelligence-QA, Mid

United States

Full-time

Remote

Mid Level

$115K/yr - $144K/yr

3+ years exp

Acentra Health is dedicated to empowering better health outcomes through innovative technology and clinical expertise. They are seeking an Artificial Intelligence-QA Specialist to ensure the accuracy and reliability of their AI and ML products, performing various testing methodologies and collaborating with a cross-functional team to enhance product quality.

Health Care

H1B Sponsor Likely

Responsibilities

Develop and implement test strategies, detailed test plans, and test cases tailored to AI, ML, and GenAI solutions

Conduct thorough Functional, UI, and API testing, including GenAI output quality, hallucination detection, and user acceptance testing to ensure optimal application performance

Work closely with data scientists, software developers, engineers, and project managers to validate algorithms and integrate QA best practices into development processes

Implement automation testing frameworks to optimize testing efforts, including automated GenAI evaluation workflows

Verify training data integrity, perform bias detection, and conduct prompt-response validations for AI and GenAI model training

Review and verify AI-generated outputs for consistency and accuracy. Collaborate with AI/ML engineers to enhance automated evaluation systems and pipelines

Document test results, identify defects, collaborate to resolve issues, and prepare comprehensive reports with actionable insights for continuous improvement

Ensure adherence to industry standards, regulatory requirements, and Acentra Health standards around ethical and safe AI usage

Stay current with emerging AI testing methodologies and tools; contribute to evolving QA processes and strategies

Qualification

Quality AssuranceAI TestingAPI TestingPythonRESTful APIsGenAI EvaluationTest PlanningAnalytical ThinkingAttention to DetailCommunication SkillsCollaboration Skills

Required

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field; equivalent work experience will be considered

3-5 years of quality assurance experience, with a strong background in traditional QA and UAT methodologies

Proven experience in testing AI, ML, and Generative AI outputs, including Functional, UI, and API testing

Proficiency in RESTful APIs and testing tools such as Postman, specifically for testing GenAI model APIs like OpenAI and Hugging Face

Experience verifying training data integrity, bias detection, and prompt-response validation for AI models

Familiarity with data annotation tools, quality control processes, and GenAI-specific annotation for RAG and LLM fine-tuning

Demonstrated experience in testing complex software application

Proven experience designing, documenting, and executing comprehensive test plans for AI-driven products

Strong analytical thinking, attention to detail, excellent communication skills, and the ability to thrive in a fast-paced, cross-functional team environment

Strong experience with QA tools and frameworks like PyTest, Selenium, or JUnit, including Gen AI evaluation frameworks like RAGAS

Understanding of AI, ML, and GenAI concepts, algorithms, and output validation techniques

Familiarity with data annotation tools and quality control processes for GenAI-specific tasks

Solid knowledge of Python, with experience in Gen AI libraries like LangChain and LlamaIndex

Experience with GenAI model evaluation metrics such as perplexity and BLEU

Solid knowledge of software development life cycle (SDLC) and agile methodologies

Excellent analytical and problem-solving skills with keen attention to detail

Strong communication and collaboration skills, with the ability to explain complex technical issues to diverse audiences

Ability to work effectively in a fast-paced, cross-functional team environment

Preferred

Experience in healthcare technology or data analytics environments

Familiarity with regulatory and ethical standards related to AI and healthcare solutions

Benefits

Comprehensive health plans

Paid time off

Retirement savings

Corporate wellness

Educational assistance

Corporate discounts

Company

Acentra Health

Acentra Health is a provider of clinical services and technological solutions to government healthcare organizations.

Founded in 1994

Mclean, Virginia, USA

1001-5000 employees

https://acentra.com/

H1B Sponsorship

Acentra Health has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2025 (49)

2024 (49)

2023 (16)