Debates, voting, and reflection loops: composing an agentic PR reviewer that costs less than a dollar - Quarkus

Blog Debates, voting, and reflection loops: composing an agentic PR reviewer that costs less than a dollar
At the moment when I write this, the LangChain4j project has received 1,698 pull requests since the beginning of this year. Reviewing them all is a lot of work, and we wanted an agentic system to help. But a single pattern can’t do it: routing, debating, voting and reflection all need to work together, within a cost budget, in a real Quarkus application. Here’s how we built one that reviews a 2900-line pull request for $0.45.
In previous articles we discussed why no single agentic pattern can cover all use cases, introducing a generic Planner abstraction to define custom orchestration strategies, and we showed how the voting pattern and dynamic chat model selection make an agentic system more reliable and more cost-efficient. Each of those articles focused on one pattern, applied to a deliberately small example. This time we want to see what happens when many of these patterns have to work together in a real application, with real costs, real failures, and a human who wants to have the last word.
Given a pull request, the reviewer produces a report saying what is good, what is wrong and what could be improved, with every problem anchored to a file and a line of the diff. It also gives a score from 0 to 10 for usefulness, correctness, conciseness, security, test coverage, maintainability, API and documentation, and PR hygiene, plus an overall score with its confidence, and a verdict: APPROVE, COMMENT or REQUEST_CHANGES.
It only reads GitHub, with a read-only token, and never writes to it: its output is a Markdown report, the same report in JSON, and an HTML trace of the execution of its agents.
The reviewer is a Quarkus application built with the Quarkus LangChain4j extensions, version 1.15 on LangChain4j 1.21: quarkus-langchain4j-agentic for the agents, the Anthropic and OpenAI extensions for the chat models, quarkus-langchain4j-typesafe for the decision model (where TypeSafe is the company behind the Jev decision models), and quarkus-github-api to read GitHub.
The agents are plain Java interfaces written with the declarative agentic API: the extension finds them at build time, validates how they are wired together and registers each of them as a CDI bean. The reviewer can be used both from the command line and from a local web interface. Let’s see how it works.
When designing an agentic system, it is tempting to hand everything to a supervisor and let an LLM decide what to do next. For a reviewer this would be a poor choice: we want every review to fetch the same data, to run the checks that matter and to produce a report with the same structure, and we want to be able to test all of this. For this reason, the root of the reviewer is a plain sequence, and model judgment lives inside its steps, where it is actually needed.
There is nothing to implement behind this interface. Quarkus LangChain4j detects it at build time, checks that every parameter an agent reads is the output key of another agent or an input of the root, and registers the whole system as a CDI bean, so that running a review is one injection away:
A parameter named after a key that no agent produces fails the build, with the message No agent provides an output key named '…​', instead of failing a review.
Many of these subagents don’t use an LLM at all. Fetching the pull request, parsing and classifying its diff, validating the findings and rendering the report are better done by plain code, which in the declarative API is simply a static method annotated with @Agent. For instance, this non-AI agent renders the description of the pull request that all the reviewers read:
The reviewer reads GitHub through the hub4j client that the quarkus-github-api extension brings, with a read-only token, in two different ways. The first agent of the sequence reads the pull request with its diff, commits, CI checks, previous reviews and linked issue through a small GitHubReads interface: there is no point in asking an LLM what to fetch, when the answer is always the same.
Conversely, the agents that need to explore the repository, like the context gatherer collecting the code around the change, get five read-only tools: the @Tool methods of a CDI bean, reading files, searching code, reading commits and issues, that the agent declares with @ToolBox:
Quarkus LangChain4j generates the executors of these tools at build time, and lets a wrapper bean see every tool call together with the invocation context of the agent call. The reviewer uses one to enforce a budget on the number of tool calls per invocation, so that a curious agent cannot spend the whole review budget browsing the repository. Read-only access is a matter of construction: the interface has no write method, and the bean exposes exactly these five read tools.
Moreover, this agent runs only when the triage agent says that the diff alone is not enough to understand the change: it is the subagent of a @ConditionalAgent whose activation condition reads the PrSummary produced by the triage.
The heart of the review is a panel of five specialized reviewers: correctness, security, design, tests, and documentation and API. Running all of them on every pull request would be a waste: a fix in a README doesn’t need a security review, and a dependency bump doesn’t need a design review. We could ask an LLM which reviewers to run, but every routing decision would cost a full LLM call whose textual answer has to be parsed, and it would say nothing about how sure the LLM was about its choice.
This is a job for a decision model: a model that doesn’t generate text, but answers typed questions about an input, returning the probability of each possible answer. The decision router pattern uses one to choose among its subagents: given an activation threshold, it asks in a single call whether each subagent should handle the request, and then runs in parallel all those whose probability of "yes" rea