OnCorps AI · 1 month ago
ML QA Analyst
Boston, Massachusetts, United States
Full-time
Hybrid
New Grad, Entry LevelOnCorps LLC is a trailblazer in the financial services sector, harnessing the power of advanced AI and machine learning technologies to revolutionize the industry. As an ML QA Analyst, you’ll work under the Data Science umbrella and own key tasks across model evaluation, training, and ongoing maintenance for the pipelines and work products built by our DS team.
AnalyticsArtificial Intelligence (AI)CollaborationData IntegrationData VisualizationEnterprise SoftwareMachine LearningPredictive Analytics
Responsibilities
Labeling: Owning data labeling for PDF / Excel-based ML ingestion engines across the company. This includes both training-based labeling and live-run corrective labeling
Agent Evaluation Review: Review evaluation jobs where agents are underperforming, digging into individual logs and traces and providing documentation of possible prompt weaknesses
Agent Regression Testing: Interacting with customer facing “chat” agents. Validating the integrity of an agent’s response to both “happy-path” and adversarial prompts pre-deployment & when model updates are in testing
Dashboarding: Owning client-specific dashboard deployments. Working with product/CS stakeholders to deploy client-specific visualizations. Testing dashboards for errors / logical gaps
Maintenance-Mode Model Upkeep: Initiating and monitoring routine model retraining for models in a steady-state & fixing job failures in production. Working with Data Science to determine appropriate thresholds for escalation
AI Table Annotation
Qualification
Required
STEM BA/BS
High attention to detail
Ability to think critically and analytically
Ability to follow and understand a complex ruleset
Comfortable working in fast-paced environments and meeting tight deadlines
U.S. based with work authorization
Preferred
Familiarity with LLMs/Agents is a plus
Knowledge of SQL is a plus
Knowledge of Python is a plus
Benefits
Innovative Environment: Work with cutting-edge AI and machine learning technologies in a company that is at the forefront of financial services innovation.
Growth Opportunities: As a rapidly growing startup, we offer significant opportunities for professional development and career advancement.
Collaborative Culture: Be part of a collaborative and inclusive team where your ideas are valued, and your contributions make a real impact.
Work-Life Balance: Enjoy the flexibility of a hybrid working environment, allowing you to balance your professional and personal life effectively.
Competitive Compensation: We offer a competitive salary and a comprehensive benefits package.
Company
OnCorps AI
At OnCorps AI, our mission is to elevate work and performance by applying AI to handle predictable and laborious calculations, decisions, and tasks.
Founded in 2011
51-200 employees
H1B Sponsorship
OnCorps AI has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (1)
2024 (1)
2021 (1)
Funding
Current Stage
Growth StageTotal Funding
$73.67MKey Investors
Long Ridge Equity PartnersGuggenheim InvestmentsMassChallenge
2025-12-11Private Equity· $55M
2022-03-21Series B· $9.6M
2021-12-17Non Equity Assistance
Leadership Team
Robert Suh
Founder and Chairman
Recent News
2025-12-11
2024-02-06
Company data provided by crunchbase