Partners with engineers and stakeholders across the Copilot organization to turn quality questions into rigorous, reproducible evaluations, building the scenarios, metrics, and scorecards that gate what ships. Designs and builds major components of an AI-first, agentic-first evaluation platform where autonomous agents are first-class operators, delivering services, pipelines, and tooling that agents can run, extend, and reason about, with the observability and guardrails that make agent-driven operation trustworthy. Contributes to the platform's hardest challenge, delivering trustworthy results across a deep chain of dependencies driven far outside their normal operating envelope, implementing graceful degradation, intelligent retry, and dependency-aware gating so reliability holds under sustained, growing load. Builds the capabilities that make agentic Copilot experiences measurable end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, including the simulation, scraping, and scoring that capture agent behavior. Writes and reviews extensible, maintainable code and mentors less-experienced engineers, contributing to modernization of the evaluation runtime and platform so it scales with fast-growing demand while improving cost and latency. Acts as a Designated Responsible Individual (DRI) for a system that gates releases, helping instrument the reliability, throughput, and dependency health of the platform and working on-call so the results teams depend on stay correct and timely. Stays current with agentic systems, large-scale AI evaluation, and autonomous operations, adopting new tools and patterns and sharing what works to raise the bar for how the team measures quality. Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to .Net, Java, JavaScript, Rust or Python These requirements include but are not limited to the following specialized security screenings: Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, .Net, Java, JavaScript, Rust or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, .Net, Java, JavaScript, Rust or Python OR equivalent experience. Experience with large-scale distributed systems and reliability engineering, especially building dependable services that operate across many interacting dependencies under heavy or unusual load. Experience building AI-first or agentic systems, including systems, APIs, or tooling intended to be operated by autonomous agents, with the observability and guardrails that make agent-driven operation trustworthy. Experience with AI/ML evaluation, measurement, or benchmarking systems, such as LLM-based grading, metric design, or large-scale data and quality pipelines.
Want jobs like this matched to you?
Swoopd scores fresh postings against your résumé so you only see the matches that matter.