Quarkus Insights #265: Observability and Evaluation with Langfuse - Quarkus

Blog Quarkus Insights #265: Observability and Evaluation with Langfuse
This summary was generated using AI, reviewed by humans - watch the video for the full story.
Traditional software testing relies on deterministic inputs and outputs: given the same arguments, a function should return the exact same result, and HTTP 200 responses signal that all systems are healthy. Large language models and generative AI break these assumptions by introducing non-determinism, where code can compile cleanly, tests pass, and operational dashboards glow green while the system produces completely incorrect or hallucinated answers. In episode 265, regular host Eric Deandrea swaps seats to become the featured guest, joining hosts Holly Cummins and Dimitris Andreadis to demonstrate how to bring observability, continuous evaluation, and regression testing to AI applications using Langfuse and LangChain4j.
Holly opened the episode with notable updates from across the community:
Quarkiverse release surge: A newly deployed release announcement automation recorded 27 new Quarkiverse extension releases in a single week.
Quarkus 4 compatibility: Quarkiverse extension maintainers continue rolling out updates to ensure day-one compatibility with Quarkus 4.
Quarkus:Dev conference announced: Following a community vote, the upcoming virtual conference has been named Quarkus:Dev and is scheduled for February 2nd to 3rd. Community members are encouraged to prepare story proposals and participate in the upcoming call for presentations.
Time zone transitions: European daylight saving changes resulted in an hour shift between European and American broadcast schedules, giving Eric an extra hour to fine-tune his live demo.
In traditional enterprise architectures, components such as databases, message queues, and downstream microservices have well-understood failure modes. Engineers use unit tests, integration tests, and resilience patterns (circuit breakers, retries, bulkheads) to verify that the system works as designed.
Generative AI models introduce a fundamentally different integration point: they are non-deterministic. A system interacting with an LLM can execute flawlessly from an infrastructure perspective, yet return an answer that is useless, inaccurate, or commercially damaging.
Eric highlighted high-profile real-world examples:
A car dealership chatbot that agreed to sell a vehicle for one dollar, which was subsequently upheld in court as a binding agreement.
An airline customer service chatbot that fabricated an incorrect bereavement refund policy, resulting in legal liability for the carrier.
In both instances, the applications did not crash. Health checks stayed green, HTTP 200 OK statuses were returned, and operational metrics showed normal latency. The system worked, but it did not work well.
To demonstrate how subtle AI bugs can be, Eric presented a common scenario in LangChain4j: modifying a prompt template string block by removing the line containing the user’s question. The Java code compiles without issue, unit tests expecting non-empty strings pass, and the LLM returns a fluent response. However, the output is completely detached from what the user asked. Simple keyword searches or regular expressions cannot reliably catch these regressions without creating brittle tests that fail randomly across model version bumps.
Testing AI applications requires shifting from asking "Did the code execute?" to "How good was the result?" Building on concepts articulated by Angie Jones, Eric illustrated how the traditional testing pyramid evolves for AI systems:
Predictable baseline (Unit testing and resilience): At the foundation, developers still write standard unit tests to verify application logic and error handling. Mocking model providers with tools such as WireMock or Toxiproxy allows teams to verify how the application handles HTTP 500 errors, rate limits, or network timeouts.
Reproducibility and interaction flows: The middle tier focuses on recording production interactions and replaying them. In agentic architectures, this layer verifies whether agents interact properly, hand off tasks correctly, and select appropriate tools.
Performance and degradation tracking: Model behavior drifts over time. Tracking token usage, latency, and response quality across rolling averages ensures that infrastructure upgrades or prompt changes do not quietly degrade performance.
Qualitative judgment (The top tier): At the apex sits qualitative evaluation, including LLM-as-a-judge scoring, rubric evaluations, semantic similarity checks, and human-in-the-loop annotations.
Higher levels of the pyramid provide deeper confidence in system quality, but they also incur higher latency and compute costs. Sustainable testing requires balancing fast, deterministic assertions at the base with targeted evaluations higher up.
Two years ago, tracking LLM performance in production required building custom logging harnesses. Today, dedicated AI observability and evaluation platforms have matured rapidly, accompanied by significant industry consolidation:
Arize Phoenix was acquired by Dynatrace.
Traceloop was acquired by ServiceNow.
Langfuse was acquired by ClickHouse.
LangSmith remains deeply integrated into the LangChain ecosystem.
These platforms specialize in capturing AI interactions, tracking token economics, organizing evaluation datasets, and measuring model performance across time.
Deploying Langfuse for local development or production involves multiple components: a web frontend, a background worker, a PostgreSQL database, a Redis cache, and a ClickHouse analytics database. In production environments, this stack is typically deployed using Helm on platforms such as Red Hat OpenShift or via Docker.
The Quarkus Langfuse extension simplifies this operational overhead through Quarkus Dev Services:
When running in dev mode (quarkus dev), Quarkus automatically provisions and starts all required containers in the background.
The extension autom