This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Manager, Data Quality & Evaluation based in Australia.
This role offers the opportunity to build and lead the quality foundation behind advanced AI data programs supporting the next generation of intelligent systems.
You will define evaluation frameworks, quality standards, and operational processes that ensure reliable and scalable human data workflows.
The position combines strategic leadership, analytical problem-solving, and hands-on execution across AI evaluation, annotation, and data quality initiatives.
You will collaborate with cross-functional teams and customers to transform complex requirements into effective quality solutions.
The ideal candidate is a quality-focused leader who enjoys building systems, improving processes, and driving operational excellence in a fast-moving AI environment.
You will play a key role in strengthening trust, performance, and scalability across AI development programs.
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Manager, Data Quality & Evaluation based in Australia.
This role offers the opportunity to build and lead the quality foundation behind advanced AI data programs supporting the next generation of intelligent systems.
You will define evaluation frameworks, quality standards, and operational processes that ensure reliable and scalable human data workflows.
The position combines strategic leadership, analytical problem-solving, and hands-on execution across AI evaluation, annotation, and data quality initiatives.
You will collaborate with cross-functional teams and customers to transform complex requirements into effective quality solutions.
The ideal candidate is a quality-focused leader who enjoys building systems, improving processes, and driving operational excellence in a fast-moving AI environment.
You will play a key role in strengthening trust, performance, and scalability across AI development programs.
Accountabilities
- Design and manage quality frameworks for AI data and evaluation programs, including quality standards, acceptance criteria, review processes, and performance metrics.
- Translate customer requirements into scalable quality workflows that support data collection, annotation, evaluation, review, calibration, and reporting activities.
- Identify quality risks early and collaborate with delivery teams to resolve issues affecting timelines, customer confidence, or program outcomes.
- Build repeatable quality processes for calibration, QA sampling, adjudication, reviewer performance tracking, and customer reporting.
- Lead quality operations across multilingual evaluation, speech and audio review, transcription, annotation, human preference evaluation, expert review, coding evaluation, and AI model assessment programs.
- Develop and improve rubrics, task instructions, reviewer guidelines, calibration exercises, golden datasets, scorecards, and reporting templates.
- Monitor quality indicators such as reviewer agreement, error trends, performance metrics, and root causes of quality variation.
- Turn quality insights into actionable improvements across instructions, training programs, tooling, staffing models, and operational workflows.
- Support strategic customer programs through quality reviews, business discussions, escalations, retrospectives, and improvement plans.
- Partner with internal teams to ensure programs are designed for quality success from initial planning through production delivery.
- Build reusable quality assets and standardized approaches that improve scalability across multiple programs.
- Lead, coach, and develop Quality Managers, Quality Leads, Quality Specialists, reviewers, and other contributors involved in data quality operations.
- Promote a culture of quality ownership, accountability, continuous improvement, and operational excellence.
- 5+ years of experience in quality operations, data operations, AI data services, localization quality, annotation quality, evaluation operations, trust and safety quality, or a related field.
- Proven experience managing quality programs for complex customer accounts or large-scale operational delivery environments.
- Strong understanding of QA methodologies, calibration processes, sampling strategies, adjudication, error analysis, and performance reporting.
- Experience collaborating with cross-functional teams including operations, delivery, supply chain, sales, and customer-facing stakeholders.
- Strong analytical skills with the ability to transform quality data into operational improvements and strategic recommendations.
- Excellent written and verbal communication skills, with the ability to explain quality insights clearly to customers and senior stakeholders.
- Experience building processes in fast-moving, ambiguous environments where systems and workflows are continuously evolving.
- Strong leadership and people management skills, including coaching and developing quality specialists, reviewers, annotators, or operational teams.
- Experience with AI data workflows, RLHF, LLM evaluation, speech/audio evaluation, transcription, coding evaluation, multilingual evaluation, or expert review programs is highly valued.
- Experience designing evaluation rubrics, annotation guidelines, reviewer training materials, calibration workflows, or quality scorecards is a plus.
- Familiarity with human-in-the-loop data processes, annotation platforms, QA tools, dashboards, and distributed contributor networks is desirable.
- Knowledge of multilingual evaluation, cultural considerations, language quality risks, or specialized domain review processes is beneficial.
- Fully remote position with flexibility to work from anywhere.
- Opportunity to contribute to the development of AI systems through high-impact data quality initiatives.
- Leadership role with significant influence over quality strategy, processes, and operational standards.
- Collaboration with international teams working on advanced AI programs.
- Opportunity to build scalable systems and shape the future of AI evaluation operations.
- Professional growth opportunities within a fast-evolving technology environment.
- Dynamic workplace focused on innovation, continuous improvement, and operational excellence.