Keeper Security, Inc. · 5 days ago
Senior QA Analyst, AI & Machine Learning Systems
United States
Contract
Remote
Mid, Senior Level
3+ years expKeeper Security, Inc. is a leading cybersecurity company transforming cybersecurity for organizations globally. The Senior QA Analyst will provide technical leadership for testing AI and Machine Learning driven features, ensuring their quality, safety, and performance standards.
Cloud SecurityCyber SecurityIdentity ManagementSecuritySoftware
Responsibilities
Design and execute test plans for A.I and Machine Learning driven features including classification, summarization, anomaly detection, and autofill systems
Develop Python-based automation and evaluation scripts to validate model performance, reliability, and output quality
Test LLM-based features for accuracy, guardrail effectiveness, hallucination risk, and behavior under edge cases
Validate data pipelines, input/output transformations, and model integration points across Keeper’s platform
Conduct regression testing on ML models and AI workflows to ensure reliability across releases
Test enterprise-scale behavior analysis features driven by data from ARAM, PAM, and session recordings
Validate security, privacy, and zero-trust requirements for AI model interactions and data handling
Collaborate with ML engineers and cross-functional teams to identify defects and improve model behavior
Analyze logs, metrics, and model outputs to detect anomalies, performance issues, or silent failures
Qualification
Required
3+ years of experience in QA, test automation, or evaluation of complex software systems
Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs, or data pipelines
Understanding of machine learning concepts such as classification, summarization, embeddings, anomaly detection, or behavior analysis
Experience testing LLM-integrated features or AI-driven workflows
Familiarity with ML-related tooling such as Pytest, Jupyter Notebooks, Hugging Face libraries, or similar
Experience with API testing, JSON validation, and test data generation
Ability to test nondeterministic systems and evaluate model quality using structured methodologies
Strong analytical, troubleshooting, and communication skills
Experience testing cloud-integrated systems, preferably on AWS
Bachelor's degree in Computer Science, Information Systems, or equivalent practical experience
Preferred
Experience testing security-focused AI or threat analytics systems
Familiarity with ML observability tools, vector databases, or data processing frameworks
Experience testing behavior analysis, video, or image-processing pipelines
Knowledge of zero-trust principles and secure data handling
Experience working with LLM providers or AI guardrail frameworks
Benefits
Medical, Dental & Vision (inclusive of domestic partnerships)
Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life
Voluntary Short/Long Term Disability Insurance
401K (Roth/Traditional)
A generous PTO plan that celebrates your commitment and seniority (including paid Bereavement/Jury Duty, etc)
Above market annual bonuses
Company
Keeper Security, Inc.
Keeper Security is transforming cybersecurity for millions of individuals and thousands of organizations globally.
501-1000 employees
H1B Sponsorship
Keeper Security, Inc. has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2021 (1)
Funding
Current Stage
Late StageTotal Funding
$60.25MKey Investors
Summit PartnersInsight Partners
2023-05-09Private Equity
2020-08-13Private Equity· $60M
2011-01-01Seed· $0.25M
Leadership Team
Darren Guccione
Co-founder and CEO
Craig Lurey
CTO, Co-founder
Recent News
Best Information Security SIEM Tools, Software, Solutions & Vendors
2026-01-11
2025-12-25
Company data provided by crunchbase