LLM Observability Tools That Actually Work in Production: Why Agentic AI Needs Trace-Level Monitoring Beyond APM
LLM observability tools like Langfuse beat APM for agentic AI. Learn why trace-level monitoring catches what Datadog misses in production.
TL;DR: LLM observability tools like Langfuse and Honeycomb are the correct primary instrumentation layer for production agentic AI because APM cannot capture trace-level data, token counts, prompt versions, chain-of-thought reasoning steps, and tool-call sequences, that explain why a model behaved a specific way. Traditional APM remains a valid complement for infrastructure metrics like latency and error rates, but it was never built to debug non-deterministic, multi-step agent behavior.
Key takeaways
- Trace-level visibility is non-negotiable: Teams without step-by-step semantic traces are the last to understand why an agent failed.
- Per-step cost attribution changes the math: LLM-native tools pin exact spend to each reasoning hop so you can kill expensive steps.
- Langfuse fills the gaps APM leaves open: Prompt versions, evaluation scores, and full conversation context do not surface in general-purpose monitoring.
- APM and LLM observability are complements: Infrastructure signals belong in Datadog; semantic traces belong in Langfuse or LangSmith.
- Agentic AI is production infrastructure now: Autonomous agents calling external APIs make observability gaps a live business risk.
- Named alternatives exist beyond the two leaders: Arize Phoenix, Helicone, Portkey, and Lunary each serve teams with different stack constraints.
Introduction
Agentic AI systems are carrying real production load, calling external APIs, chaining reasoning steps, executing consequential actions inside live customer workflows. Traditional APM (Datadog, New Relic, classic Honeycomb) was built on one assumption: latency and error rate are sufficient proxies for system health. For agentic AI, that assumption creates a dangerous blind spot. Practitioners are actively asking how to instrument costs, token usage, function calling, and full-text search across LLM interactions, none of which traditional APM surfaces natively.
What does Datadog APM actually miss when monitoring LLM agents in production?
Datadog APM tracks latency, error rates, and HTTP status codes, none of which tell you whether your LLM agent's output was correct, costly, or safe.
APM cannot capture any of the five data categories that explain what actually happened:
- Prompt text and response content
- Model outputs at each reasoning step
- Tool call parameters and return values
- Per-step token consumption and cost attribution
- Evaluation scores and quality signals
APM tells you the pipes didn't leak. It cannot tell you what flowed through them. Datadog is categorized separately from LLM-native platforms in major comparisons for exactly this reason.
How do Langfuse and Honeycomb close the gap that APM leaves open?
Langfuse and Honeycomb close the APM gap by capturing the semantic content of each reasoning step, not just whether it completed, but what the model said, what it cost, and whether the output was correct.
Langfuse: semantic trace layer
Langfuse is an open-source LLM observability platform under the MIT license that captures every request an LLM application handles and turns that data into something a team can debug and evaluate. It handles prompt version tracking, evaluation score ingestion, full conversation context, and per-step cost attribution. When a prompt regression ships, teams can filter traces by version, compare evaluation scores, and identify which tool calls degraded, rather than waiting for a customer to report the failure.
Which LLM observability tools belong in a production stack, and what does each one cover?
The right stack pairs a semantic trace platform like Langfuse or LangSmith with an infrastructure query tool (leaving Datadog APM for the non-AI service layer) because no single tool captures everything an agentic system produces.
Table 1: LLM Observability Stack Capability Matrix, 2025
Capabilities reflect publicly documented platform behavior as of June 2025. Verify current feature status directly with each vendor before making a procurement decision.
| Capability | Datadog APM | Honeycomb | Langfuse (OSS) | LangSmith |
|---|---|---|---|---|
| HTTP latency and error rate | Native | Native | Not in scope | Not in scope |
| Full prompt and response text | Not captured | Manual only | Native | Native |
| Per-step token cost attribution | Not captured | Manual only | Native | Native |
| Tool call parameters and returns | Not captured | Manual only | Native | Native |
| Prompt version tracking | Not captured | Not captured | Native | Native |
| Evaluation and quality scores | Not captured | Not captured | Native | Native |
| High-cardinality event querying | Limited | Best-in-class | Limited | Limited |
| Open-source / self-hostable | No | No | Yes, MIT license | No |
LangSmith is a strong alternative for teams already in the LangChain ecosystem. Practical rule of thumb: Datadog for the non-AI service mesh, Langfuse or LangSmith for the semantic trace layer, Honeycomb optional if your team already relies on it for broader infrastructure.
Additional platforms worth evaluating
- Helicone, open-source (Apache-2.0), proxy-based instrumentation with cost tracking
- Portkey, open-source (MIT), gateway-oriented with multi-provider routing and observability
- Lunary, open-source (Apache-2.0), conversation-focused with evaluation and analytics
Run a short evaluation against a real agent trace before committing.
When should a team use both APM and LLM-native observability, and when is one enough?
Use both layers whenever your agent calls external APIs, APM covers the infrastructure contract and LLM-native tools cover the semantic contract, and neither substitutes for the other.
Scenario 1, APM and Langfuse (most common): An agent embedded in a customer workflow with real-world side effects. APM monitors infrastructure health; Langfuse monitors whether the agent reasons correctly.
Scenario 2, Langfuse only: An internal-only LLM app, no downstream side effects, small team. Reasonable when the worst failure mode is a bad answer, not a bad action.
Scenario 3, Honeycomb and Langfuse, skip Datadog: A team already running Honeycomb for the non-AI stack. Honeycomb handles infrastructure signals; Langfuse handles semantic traces.
Frequently asked questions
What is LLM observability and how is it different from regular monitoring? LLM observability is the practice of gathering data about an LLM-based system in production to understand, evaluate, and optimize it, including the prompt, the model's response, tool calls made, and whether the output was correct. Regular monitoring tells you a request completed; LLM observability tells you whether it produced a good result.
Why can't I just log LLM inputs and outputs to Datadog and call it done? Logging raw LLM I/O to Datadog omits prompt version management, structured evaluation score ingestion, per-step cost rollups, and tool-call parameter tracking that purpose-built platforms provide natively. For a prototype this may be acceptable, but you'll find yourself rebuilding what LLM-native tools handle out of the box.
What are the best open-source LLM observability platforms available as of June 2025? Langfuse is widely cited and available under the MIT license. Other active options include Arize Phoenix (ELv2), Helicone (Apache-2.0), Portkey (MIT), and Lunary (Apache-2.0). The right choice depends on your existing stack and the debugging workflows you prioritize.
How do forward-deployed engineers instrument per-step token cost in multi-agent reasoning chains? Each reasoning step gets wrapped as a named span with token counts and model ID attached as span attributes; each sub-agent gets its own trace context that rolls up to the parent, giving visibility into both total and step-level cost. Achieving equivalent visibility with standard APM requires significant custom instrumentation work.
Conclusion
APM answers one question: did the system respond? Agentic AI introduces a second question APM cannot answer: did the agent reason correctly? A trace can pass every APM health check while the agent executes a hallucinated tool call that breaks a live customer workflow.
The LLM observability tooling has matured enough that running blind on the semantic layer is now a deliberate tradeoff, not an infrastructure constraint. The named platforms in this guide (Langfuse, LangSmith, Arize Phoenix, Helicone, Portkey, and Lunary) each address parts of the gap APM leaves open.
Your next step: Instrument one production agent with Langfuse this sprint. Attach prompt version, tool call parameters, and per-step token cost to every span. Run it two weeks alongside your existing APM. The difference in what each surface shows you will make the case for the full stack change.
Learn from me
Forward Deployed Engineering Bootcamp for Full-Stack Developers, my Maven cohort. Build and ship complete AI products end to end, from React and Node.js frontends to deployed models with caching and observability. Join the next cohort →
Hire us
Traversaal.ai. We're a team of forward deployed engineers solving the toughest AI problems for Fortune 100 companies: document intelligence, agentic data platforms, and real-time web intelligence, deployed in production. Work with our team to deploy your next agentic ecosystem. Talk to Traversaal.ai →
Join us
Want to solve these problems with us? We're always looking for forward deployed engineers who want to ship production AI. jobs@traversaal.ai
