Qualitest · 19 hours ago

#16710 - AI Test Architect

USA

Full-time

Remote

Senior Level

$170K/yr - $180K/yr

Maximize your interview chances

ConsultingEnterprise Software

Actively Hiring

Insider Connection @Qualitest

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Architect and build scalable test automation frameworks for generative AI models, leveraging tools like DeepEval, PyTest, or custom Python scripts.

Implement automated evaluation workflows for model validation, including functionality, accuracy, performance, and reliability testing.

Design continuous testing pipelines integrated with CI/CD for model deployments.

Develop modular frameworks to automate evaluation of NLP-based outputs, including coherence, relevance, toxicity, bias, and safety checks.

Optimize test execution for large-scale LLM deployments, minimizing latency and maximizing throughput.

Automate performance and load testing for high-token transactions and API-driven AI endpoints.

Leverage vector database testing to ensure quality of data retrieval and seamless model integration.

Introduce monitoring mechanisms to validate models post-deployment and detect model drift.

Collaborate with DevOps and AI/ML teams to integrate AI model testing within MLOps workflows.

Drive adversarial and stress testing automation to assess AI robustness and edge-case handling.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Test automation frameworksDeepEvalLangsmithPyTestPythonJavaScriptTypeScriptNLP applicationsCI/CD pipelinesVersion control (Git)MLOps platformsPerformance testing toolsVector databasesAI evaluation metricsAdversarial testing techniquesArchitectural skills