Crowe · 6 days ago
AI Testing Engineer
Chicago, IL
Full-time
Hybrid
Senior Level, Lead/Staff
$74K/yr - $148K/yr
4+ years expCrowe is one of the largest public accounting, consulting and technology firms in the United States, and they are seeking an AI Testing Engineer to ensure the quality and reliability of enterprise AI and machine learning systems. The role involves leading testing efforts, developing automated evaluation frameworks, and collaborating with various teams to define testing strategies and assess model behavior.
AccountingAdviceConsultingFinanceFinancial ServicesInformation TechnologyProfessional ServicesTax Consulting
No H1B
Responsibilities
Designing comprehensive testing strategies for predictive models, generative AI systems, and end-to-end ML pipelines
Leading the development of automated test harnesses, evaluation suites, and validation tools integrated into CI/CD workflows
Analyzing model outputs for correctness, safety, fairness, robustness, and stability across diverse test scenarios
Building synthetic datasets, challenge sets, and adversarial test cases to uncover model weaknesses
Evaluating LLM and generative model behavior, including hallucination rates, prompt sensitivity, and retrieval accuracy
Collaborating with engineering and data science teams to define evaluation criteria, KPIs, and acceptance thresholds
Troubleshooting complex ML system issues such as performance degradation, drift, or unexpected failure patterns
Implementing post-deployment monitoring systems to continuously validate model behavior in production
Documenting testing methodologies, findings, and recommendations to inform system improvements
Guiding junior engineers and QA specialists in advanced AI testing techniques and tools
Ensuring adherence to enterprise responsible AI, safety, security, and compliance standards
Identifying reliability and trust risks and contributing to mitigation strategies
Contributing to AI platform architectural decisions to improve testability and observability
Researching and evaluating emerging AI testing methodologies, benchmarks, and tooling ecosystems
Qualification
Required
4+ years of experience in software testing, ML engineering, data science, or related roles
Strong proficiency in Python and automated testing frameworks
Deep understanding of model evaluation techniques, including precision/recall, calibration, robustness, and stability testing
Familiarity with LLM evaluation metrics, safety testing approaches, and structured test design
Demonstrated ability to diagnose complex model, data, and pipeline failures
Strong collaboration and communication skills across technical and non-technical teams
Willingness to travel occasionally for cross-functional planning and collaboration
Preferred
Bachelor's degree in Computer Science, Engineering, Data Science, or a related technical field, or equivalent experience
Experience testing AI/ML systems in cloud-based environments
Hands-on experience with cloud ML platforms such as SageMaker, Vertex AI, or Azure ML
Familiarity with containerization (Docker), Kubernetes, and distributed test execution
Experience integrating automated AI testing into CI/CD pipelines (e.g., GitHub Actions or similar tools)
Experience with monitoring and logging systems for post-deployment model validation
Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, and hallucinations
Familiarity with RAG evaluation workflows and vector databases (e.g., FAISS, Pinecone, Weaviate)
Experience with prompt engineering, adversarial prompting, and synthetic data generation
Familiarity with Hugging Face evaluation tools and testing fine-tuned models (e.g., LoRA, QLoRA)
Testing, quality engineering, or cloud certifications
Excellent analytical, documentation, and mentorship skills
Ability to collaborate effectively in hybrid or remote team environments and support extended hours during critical model releases or incidents
Benefits
Unlimited PTO
Flexible remote work policy
Comprehensive total rewards package
Company
Crowe
Crowe LLP is a public accounting, consulting, and technology firm.
5001-10000 employees
Funding
Current Stage
Late StageTotal Funding
unknown2023-08-29Acquired
Leadership Team
James L. Powers
CEO
Joy Mikolajczak Duce
Managing Principal/Partner - Human Capital Consulting
Recent News
Canada NewsWire
2026-01-02
2025-11-11
2025-10-24
Company data provided by crunchbase