QA Bench: Evaluating QA agents on web apps | Momentic

QA Bench: Mo, Codex, Claude Code and Devin against 1,396 labeled bugs.Read the report
QA Bench measures how many real bugs a QA agent finds in a running web application within a fixed amount of time, and how many of the bugs it reports are real.
QA Bench measures how many real bugs a QA agent finds in a running web application within a fixed amount of time, and how many of the bugs it reports are real. It compares Momentic's Mo with Codex (GPT-6.1 Sol and GPT-6.1 Terra), Claude Code (Opus 5.5 and Sonnet 5.5), and Devin across 20 product areas in four widely used open-source applications, Gitea, Metabase, Twenty, and Saleor, which together have over 190,000 GitHub stars. Codex and Claude Code each ran with browser automation tools, Playwright MCP and agent-browser, driving hosted Browserbase browsers rather than local ones. The reference set holds 1,396 labeled bugs.
Each run gives the agent one product area in a fresh sandbox and either 30 or 60 minutes. An LLM judge then matches the agent's bug reports against the approved reference bugs for that area and a set of claims reviewers have already rejected. We report recall, precision, a severity-weighted score, and cost per matched bug. Each count averages three runs per area (except Devin, which had only a single pass), so it reflects a typical pass through the suite rather than any single pass.
Mo is Momentic's testing agent. Momentic built, designed, ran, and graded the benchmark.
Try Mo on your staging URL
Mo files bugs with recordings and repro steps. $50 of free usage each month.
If you point a coding agent at a running web app and ask it to find bugs, it returns a list. The list doesn't tell you how much the agent missed or how many items are real. The only way to know either is to have a labeled set of bugs for that app and grade the list against it.
That matters more than it used to. Static analysis and unit tests now catch a large share of defects before code ships, so the bugs that get through are increasingly the kind you only see by using the product, such as a button that does nothing, a filter that returns the wrong rows, or a settings page that shows a stack trace. End-to-end testing is where those bugs are, and it's the part of QA that agents are now being asked to do.
Momentic's experience is that agents miss many of these. We wanted a way to measure whether Mo was getting better at this and compare it with the agents people already have. No benchmark existed for the task, so we built one.
The benchmark asks how much of a labeled bug set each agent can find in a fixed time (recall), how many of its reports are real (precision), and what it costs. The two kinds of mistakes don’t cost quite the same.
Higher recall increases monetary cost (the agent spends more time finding bugs) but produces higher-quality production code.
QA Bench is a first-party benchmark, and we run every agent under the same rules.
The rules fix the time an agent gets, not how it spends it, so cost appears next to every coverage number.
The benchmark uses four widely used open-source web applications, each pinned to an upstream commit and split into product areas. An area is a part of the product with its own scope and entry points, such as Gitea's issues, Metabase's dashboards, Twenty's workflows, or Saleor's vouchers.
Upstream repositories: Gitea , Metabase , Twenty and Saleor .
The areas differ in size. Gitea's eleven areas average 43 labeled bugs each, Metabase's four average 116, and Twenty's four average 94.
Ten agent configurations ran the benchmark.
Mo ran with no custom prompt and no setup beyond the effort preset. This tested Mo as a new user would: signing up and telling it to test something. Its medium and high presets run for about 30 and 60 minutes, serving as its two budget conditions.
Codex and Claude Code each ran two models and two browser integrations, so we can compare results across models and browser tools. Both integrations drove a hosted Browserbase browser, not a browser on the agent's machine.
Devin ran in its own native browser at default settings, not on Browserbase.
The agent gets the area's scope and entry points, then explores the area and reports the bugs it finds. It can't read the application source, the labels, or earlier reports. Each configuration ran each area three times at each budget, for a total of 60 runs per configuration per budget, with one exception:
Where a configuration is missing an area, its totals cover the areas it ran, and any comparison with Mo uses the same areas for both.
The reference set is the list of bugs that each area is known to have. Candidate bugs come from exploratory bug bashes by Mo, Claude Code, and Codex, and each candidate goes through the same four steps before it becomes a label:
Approved bugs go into the area's ground-truth.json. Rejected claims go into known-invalid.json, which the grader also uses, so an agent that reports something a reviewer has already rejected gets marked down for it. The reference set lists confirmed bugs in these application versions, not every bug they contain.
Each approved bug has a stored severity weight based on the kind of failure. Scoring uses the stored weight, not the judge's estimate.
The weights grade what the user runs into, not security impact, so a page crash the user can recover from by reloading is still a 5. Section 9 has examples at each weight.
An LLM judge, GPT-6.1 Sol at high reasoning effort, matches each submitted finding to the area's labels. It first groups duplicate reports of the same bug. Each finding is then a match (a bug in ground-truth.json) or an invalid match (a claim in known-invalid.json). Findings that match neither go to a review queue, where a reviewer labels each one as one or the other, so the final results classify every finding as one of the two.
Duplicates are counted separately, so a bug reported twice in one run is one match and one duplicate.
Recall is the share of an area's labeled bugs that a run matched: