AI Root Cause Analysis: LLM-Powered Root-Cause Analysis for Distributed Systems - Quarkus

Blog AI Root Cause Analysis: LLM-Powered Root-Cause Analysis for Distributed Systems
When something goes wrong in a distributed system, engineers face the same ritual: open Grafana, dig through traces in Tempo, correlate logs in Loki, check Prometheus metrics, and mentally piece together a causal chain across services. What if an LLM could do that for you — pulling telemetry from the same tools you already use and producing a structured root-cause analysis in seconds?
That’s what AI Root Cause Analysis does. It’s an open-source Quarkus application that connects to your existing Grafana LGTM stack via MCP (Model Context Protocol), feeds correlated traces, logs, and metrics to an LLM, and returns actionable analysis — complete with root causes, severity assessments, and remediation suggestions.
AI Root Cause Analysis is built for developers who work with distributed systems but don’t want to become observability experts. You know your LGTM stack is collecting traces, logs, and metrics — but correlating them across services, spotting the root cause in a wall of spans, and distinguishing symptoms from causes takes experience and time. The AI handles that for you: it knows how to query Tempo, Loki, and Prometheus, understands what the telemetry means in context, and draws correlations that would take manual investigation to piece together.
If you’re comfortable with AI but not with the details of trace propagation, span hierarchies, or PromQL — this tool bridges that gap.
AI Root Cause Analysis is a standalone analysis engine. Point it at your LGTM stack and your running applications, and it will:
Retrieve recent traces from Tempo via MCP and identify the interesting ones (errors, high latency, unusual patterns)
Correlate logs for each trace from Loki, linking log entries to specific spans
Pull Prometheus metrics at the time window of each trace to add system-level context
Analyze everything together using an LLM, producing a structured report that distinguishes root causes from symptoms, identifies cascading failures, and suggests next steps
Optionally, it can also:
Examine source code of monitored applications via Quarkus Dev MCP, correlating analysis findings back to specific code paths
Generate Grafana dashboards tailored to the issues it found, saved directly into each application’s workspace
The key insight is the data pipeline. Raw telemetry from MCP is noisy and can easily overwhelm an LLM’s context window. A StripMcpClient decorator sits between MCP and the LLM, applying per-tool filtering functions that strip the noise while preserving signal:
Traces: Removes SDK metadata (telemetry.sdk., otel.), localhost attributes, and instrumentation library details
Logs: Strips SDK metadata, redundant fields, and duplicate information already present in trace spans
Metrics: Removes Netty internals, static/constant values, histogram bucket breakdowns, and per-region metric breakdowns — reducing 586 entries to ~35 (a 94% reduction)
This keeps the context window focused on signals that matter for root-cause analysis.
Imagine you have two microservices — a gateway and a backend — and users are reporting slow responses. You start AI Root Cause Analysis, point it at your LGTM stack, and ask it to analyze the last 5 traces. Here’s what happens:
The AI retrieves 5 recent traces from Tempo and spots that 3 of them took over 3 seconds
It pulls the correlated logs from Loki and finds Connection pool exhausted warnings on the backend service
It checks Prometheus metrics and sees JVM heap at 92% with frequent GC pauses
It produces a report: "Root cause: database connection pool exhaustion on the backend service, caused by slow queries under memory pressure. The gateway’s 3s latency is a downstream symptom, not a cause. Remediation: increase connection pool size and investigate the heap usage."
That’s the kind of multi-signal correlation that normally takes 30 minutes of dashboard-hopping — done in seconds.
If you already have an LGTM stack running, you don’t need to clone the repo — just download the uber-jar from Maven Central and run it:
Open http://localhost:8080, pick how many traces to analyze, and hit Analyze. That’s it — no build, no clone, no setup.
For development with Quarkus Dev Services (auto-starts an LGTM stack for you):
The -Dapp.ports argument (shown as 8081,8082 in the examples above) is optional — it lists the ports of your running Quarkus applications. When provided, the AI module connects to them via Dev MCP, enabling source code examination and writing generated Grafana dashboards directly into your application sources. This requires the target applications to be running in Quarkus dev mode with quarkus.dev-mcp.enabled=true set (which exposes the Dev MCP endpoint). Without it, the core analysis (traces, logs, metrics) works the same — the Examine Source button is simply disabled in the UI.
To run without application ports, simply omit -Dapp.ports from the java command or the ports argument from ./dev-ai.sh.
AI Root Cause Analysis supports multiple LLM providers via Maven/Quarkus profiles:
Each profile activates the corresponding LangChain4j extension and Quarkus configuration. Grok uses the OpenAI-compatible API at https://api.x.ai/v1.
Open http://localhost:8080 to access the analysis UI:
The interface provides:
Traces — number of recent traces to analyze (1-20)
Output Format — HTML, Markdown, Plain Text, or AsciiDoc
Examine Source — include source code examination via Dev MCP
Create Dashboard — generate a Grafana dashboard tailored to the issues found
After clicking Analyze, the UI shows a performance stats bar (duration, LLM calls, input/output tokens) and three collapsible result sections:
Analysis — the main telemetry analysis with per-trace root cause, error patterns, latency breakdown, and cross-trace correlation
Examined Sources — when enabled, the AI reads the source code of your monitored applications (via Quarkus Dev MCP) and maps the problems it found back to specific code lo