WellSaid · 8 hours ago

Senior Technical Product Manager, ML Quality

Remote - US

Full-time

Remote

Senior Level

$130K/yr - $170K/yr

5+ years exp

Maximize your interview chances

Artificial Intelligence (AI)Real Time

No H1B

U.S. Citizen Only

Insider Connection @WellSaid

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Defining QA strategy for ML models across many languages: Identifying critical metrics and testing methodologies to assess the performance of TTS models throughout the research & development lifecycle. This includes developing such strategies for any new languages we explore in the future.

Evaluating and validating TTS models: Working with the Applied ML team and selecting appropriate evaluation metrics, running rigorous tests across datasets, experiments, and final models, and interpreting results to identify areas for improvement and/or offer recommendations to the Research team.

Managing data quality: Monitoring data quality, identifying potential biases in training data, and ensuring data preprocessing is done correctly to optimize model performance.

Developing QA automation frameworks: Defining automated testing tools and pipelines to streamline the testing and evaluation processes for TTS models, especially for regression testing.

Monitoring model performance in production: Tracking key metrics post-deployment to identify potential degradation in model performance and take corrective action. This may also require defining the monitoring and tooling needed to do so.

Moderating ethical considerations: Assessing potential biases in ML models and training data, ensuring responsible development practices, and addressing concerns regarding fairness, transparency, and Responsible AI.

Researching and partnering with third parties: Defining test cases, scenarios, and evaluation batches with external evaluation services when appropriate. Recommending evaluation services to leadership with a well-defined testing strategy defending your recommendation.

Collaborate closely with data engineers and software engineers in the ML team to understand model capabilities, identify new methods or metrics for gauging model progression, and create new tools to make the testing and evaluation processes fast and easy to implement

Write design docs with clear requirements and expectations for engineers to execute against

Research and collate text and audio test cases from real customer data, product feedback, and active experiments.

Synthesize evaluation results into reports for internal and external publication and/or deliver recommendations to the research team informed by rigorous regression analysis

Partner closely with Lead Product Manager, ML on overall product roadmap, customer impact, and development recommendations.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Machine LearningTTSModel PerformanceQA StrategyData Quality ManagementAutomation FrameworksAudio EngineeringProcess-oriented OrganizationProject ManagementLeadership Qualities