We're growing our team and looking for a Senior/Principal QA Engineer to own quality across our AI voice assistant: from prompt and agent behavior testing to mobile, TV, and backend.
Requirements
- Senior / Principal, 5+ years of manual QA - a high level of ownership and self-direction: the ability to take on the project as a whole, be fully accountable for quality, and get up to speed on new things independently.
- Experience testing AI agents / assistants: checking scenario and prompt following (prompt / instruction following), response quality and relevance.
- Experience testing "under the hood" via AI observability / tracing tools (LangSmith and analogs): analyzing an agent's traces - tool calls, memory handling, the model that ran, inputs/outputs, token consumption, spotting bottlenecks.
- Experience with mobile and/or TV-app testing; experience specifically with TV devices is less critical.
- Experience testing backends and integrations: API testing (REST / gRPC), verifying the seams between services, reading logs and traces (Kibana / Grafana / Loki, etc.).
- Solid QA fundamentals: test design, test documentation, defect handling.
- Hands-on experience using AI tools in the day-to-day QA process.
Nice to have
- Experience testing speech technologies (ASR / TTS) and working with audio.
- Evals - building and running AI-assistant quality evaluations: scenario-based self-play, LLM-as-judge, manual validation as human-in-the-loop.
- CI/CD / deployment / test environments: configuring stands, building and deploying applications (GitHub / GitLab, etc.).
- Device testing experience / ADB.
- Willingness to move into test automation over time.
Responsibilities
AI assistant:
- Assessing assistant quality: prompt / instruction following, compliance with requirements; catching regressions between prompt and model versions.
- Testing the assistant "under the hood" via observability / tracing tools: correctness and parameters of tool calls (MCP / integrations with external APIs), memory handling, the bot's reasoning, traces, etc.
Devices and applications:
- End-to-end manual testing of the assistant in the mobile app and on the devices (TV, speaker).
- Testing voice interaction with the assistant, recognition (ASR) and synthesis (TTS) quality, voice UX.
- Testing the assistant's product scenarios: agentic e-commerce (voice order -> cart -> confirmation on TV via remote), media content and a movie showtimes guide, flight tickets, basic scenarios (weather, currency rates, time, timers, Q&A).
Platform:
- Testing the platform / backend part the assistant runs on - a distributed backend of many services: dialog orchestration, LLM agent, ASR / TTS, etc.
- Verifying the request's end-to-end path through these components, the correctness of integrations and the seams between services, working with backend logs and traces.
Process:
- Maintaining test cases and bug reports.
- Close collaboration with the team: TPM, Prompt Engineer, DEV, DevOps, ML.
- First-line QA of critical scenarios ahead of demos.
What we offer
- The team has built award-winning AI products for tech corporations - devices, voice assistants, products that are actually in the world
- Cutting-edge tech stack: Speech Technologies, NLP, Generative AI (LLMs, diffusion models), voice-first agentic architecture with privacy-first and on-premises deployment
- High engineering bar and real ownership - the team cares about what actually works in production, not what looks good in a demo, and you'll see the impact of your work directly
- Fast career progression - a senior-heavy team and a high volume of real problems means you grow faster than you would anywhere else
- Startup pace with enterprise stability - real clients, real revenue, no bureaucracy
- Fully remote across Europe
- 21 vacation days + public holidays + 5 sick days
- Private English lessons via Preply