development

Best LLM Observability & Eval Tools

Every serious way to observe what your LLM application is actually doing in 2026 — open-source platforms, commercial eval tools, APM incumbents, AI gateways and cloud-native tracing — compared on licence, price, OpenTelemetry support and what they capture by default.

Updated August 2026

Quick answer: For most teams the best default is Langfuse — MIT-licensed, genuinely self-hostable, OpenTelemetry-native, and now owned by ClickHouse, which acquired it on 16 January 2026 alongside a $400 million Series D, with the MIT licence and self-hosting explicitly retained. If your priority is evaluation rather than tracing, Braintrust is the strongest commercial option and raised $80 million at an $800 million valuation in February 2026. If you already run Datadog, Sentry, Grafana or Honeycomb, use their LLM features before buying anything new — the marginal value of a second tool is lower than the vendors selling it suggest. If you serve traffic through Cloudflare, its AI Gateway gives you dashboard analytics, caching, rate limiting and persistent logging free on every plan, which makes it the cheapest useful starting point in the entire category — though Guardrails is billed separately as Workers AI inference.

The caveat that governs every recommendation below: the OpenTelemetry GenAI semantic conventions — the standard all of these tools claim to support — are still marked Development, not Stable, and in June 2026 they were moved out of the core specification repository entirely. “OpenTelemetry-compatible” does not currently mean two tools agree on a schema.

This page covers the whole landscape: open-source platforms, commercial evaluation tools, APM incumbents, AI gateways, and the cloud providers’ native offerings. It is written for developers choosing a tool, not for buyers reading a feature grid.

What LLM observability actually is

Traditional application monitoring answers three questions: is the service up, how fast is it, and are there errors. None of those catches the failure modes that matter in an LLM application.

The best non-vendor framing comes from a CNCF post published on 4 August 2026, which sets out three pillars for agent systems:

1. Traces — the session timeline. Every model call, tool invocation and sub-agent delegation becomes a span, nested so a sub-agent’s work appears as children of its parent.
2. Costs — the unit economics. As the post puts it: “An agent that loops… burns tokens geometrically. Without cost monitoring, you discover this when the invoice arrives, not when the loop starts.”
3. Audit — an append-only, timestamped record of every tool call and governance decision, with PII redacted before logging.

The same post draws the distinction most teams get wrong: “Traces are for debugging, metrics are for alerting. Don’t try to alert on traces (too detailed) or debug with metrics (too aggregated).”

Why APM breaks here. Agents do not crash with stack traces. They loop, they hallucinate, and they produce plausible output that is subtly wrong. A conventional monitor reports a successful 200 response on a request that cost forty dollars and answered incorrectly. That is the gap this entire tool category exists to fill.

The pillars in practice. Beyond traces, costs and audit, a complete stack usually covers offline evaluation against test sets, online evaluation of production traffic, prompt versioning, dataset management, guardrails, and latency metrics including time to first token. No single tool is best at all of these, which is why the recommended stacks at the end of this page combine two.

The comparison table

Ranked by how well each serves a team that has an LLM application in production today. Prices are the vendor’s own published figures; the final column records whether we read the figure on the vendor’s own pricing page.

The open-source tier

Langfuse — the best default

Langfuse is the tool to pick if you have no strong reason to pick another one.

Licence. The repository is MIT, with the exception of the ee/ enterprise folders, stated plainly in the project’s own README. That makes tracing, evaluations, prompt management, datasets and the playground free to self-host without a commercial licence. Features reserved for paying self-hosters are the enterprise administration set — SCIM, audit logs, retention policies.

The ownership change, and why it is reassuring rather than alarming. ClickHouse announced on 16 January 2026 that it had acquired Langfuse, in the same announcement as a $400 million Series D led by Dragoneer that valued ClickHouse at $15 billion. ClickHouse’s own post commits to keeping the MIT licence and self-hosting as a first-class option. The technical logic is unusually clean: Langfuse already ran entirely on ClickHouse in both its cloud and self-hosted deployments, so this is an infrastructure company buying the application already built on it, not an unrelated acquirer.

Pricing model. Langfuse meters a single billable unit and charges no per-seat fee. A unit is a trace, an observation or a score, and your bill is the sum of all three. The free Hobby tier includes 50,000 units per month. This is a materially better model for a growing team than per-seat pricing, because adding a colleague costs nothing.

One honest gap in our own research: we confirmed the tier names, the billable-unit definition and the free allowance from Langfuse’s own documentation, but could not load the pricing page itself to verify the exact dollar figures for the Core and Pro tiers. We have not published those figures rather than repeat numbers from aggregators.

OpenTelemetry. Native, and it accepts traces from existing OTel instrumentation. Integrations cover the full framework field including LangChain, LlamaIndex, LiteLLM, the Vercel AI SDK, Amazon Bedrock, DSPy, CrewAI and Dify.

Arize Phoenix — excellent, and not open source

Phoenix is the strongest local-first debugging experience in the category, and the licence deserves a closer read than it usually gets.

Phoenix is licensed under the Elastic License 2.0, which is not an OSI-approved open-source licence. In practice this matters less than it sounds: Arize’s own licence page states that self-hosting on your own infrastructure is free, fully permitted and has no feature gates. What ELv2 prohibits is offering Phoenix to third parties as a hosted service. Unless you are building a competing product, this is not a constraint you will hit — but call it Elastic-licensed rather than open source, because the distinction is real.

Arize AX, the managed product, is priced at $0 for 25,000 spans per month with 1GB of ingestion and 15-day retention, and $50 per month for 50,000 spans, 10GB and 30-day retention, with an Enterprise tier adding self-hosting, SOC 2, HIPAA and custom data regions. Read on 16 August 2026.

Arize also maintains OpenInference, its own Apache-2.0 tracing specification, which predates the maturity of the official OTel GenAI conventions and was purpose-built for agent and retrieval shapes. Arize’s stated direction is to normalise OpenInference traces into OTel-compatible spans at ingestion as the official spec stabilises.

Comet Opik — the most permissive licence

Opik is Apache-2.0 and the entire platform is self-hostable — server, web application, tracing, datasets, experiments, evaluations, prompt management, online evaluation and the agent optimiser. There is no enterprise-folder carve-out of the kind Langfuse uses. The repository shows more than 21,000 stars.

Cloud pricing, read from Comet’s own page on 16 August 2026: Free at 25,000 spans per month with 60-day retention and up to 10 team members, Pro at $19 per month including 100,000 spans then $5 per additional 100,000, with up to 50 team members and the same 60-day retention, and a custom Enterprise tier. Comet also offers a free Pro tier to verified students and educators.

Choose Opik over Langfuse if licence purity matters to your legal team, or if you want the whole product self-hosted with no enterprise carve-out at all.

MLflow — the most standards-committed

MLflow deserves more attention than it gets in LLM discussions, because it makes the strongest OpenTelemetry claim of any tool here: it both ingests and exports traces in the OpenTelemetry GenAI semantic convention format, and the MLflow server exposes an OTLP endpoint at /v1/traces that accepts traces from any OTel instrumentation, not just Python. Apache-2.0, Linux Foundation-governed.

Choose MLflow if avoiding vendor lock-in is your primary constraint, or if you already run it for classical machine learning.

Traceloop OpenLLMetry and Laminar — the instrumentation layer

OpenLLMetry is Apache-2.0 and is not an observability backend at all — it is a set of OpenTelemetry instrumentation libraries whose semantic conventions have been contributed upstream to OpenTelemetry itself. It exports to more than twenty destinations including Datadog, Dynatrace, Grafana, Honeycomb, New Relic, SigNoz, Splunk and Sentry. If you want instrumentation that does not tie you to a backend, this is it.

Laminar is Apache-2.0, written in Rust, OTel-native, and self-hostable with a single Docker Compose command. It is small (roughly 3,000 stars, YC S24) but includes a genuinely useful SQL editor over all platform data.

Two open-source tools to approach with caution

Lunary — the open-source claim could not be verified. Lunary’s pricing page still states that you can self-host the Community Edition for free. However, as of 16 August 2026 the core lunary monorepo is absent from the company’s GitHub organisation, which lists only peripheral repositories — the JavaScript SDK, an archived Python SDK, and forks. This is consistent with a report from December 2025 that the core repository had begun returning a 404. We are not calling this a licence change, because we cannot see one. We are saying that the self-hosting option the pricing page advertises could not be located, and you should verify it yourself before depending on it.

Langtrace — maintenance risk. Dual-licensed AGPL-3.0 for the application and Apache-2.0 for the SDKs, which is a meaningful distinction if you plan to modify and network-serve it. The larger concern is activity: the most recent release visible is 4.0.11 from April 2025, with no 2026 releases. The cloud product is still free with a stated intention to monetise later.

One licence note worth internalising: this tier spans MIT, Apache-2.0, Elastic License 2.0 and AGPL-3.0, and those are four genuinely different legal positions. AGPL in particular has copyleft implications for a modified, network-served deployment that MIT and Apache-2.0 do not.

The commercial evaluation tier

LangSmith

The most complete commercial platform, and the obvious choice if you build on LangChain or LangGraph. Pricing from LangChain’s own page, read 16 August 2026: Developer at $0 for one seat with up to 5,000 base traces per month, Plus at $39 per seat per month with up to 10,000 base traces, and custom Enterprise with self-hosted and hybrid deployment.

Usage beyond the included allowance is metered in two normalised units — a LangChain Compute Unit at $1.50 and a LangChain Storage Unit at $1.00. Base traces carry 14-day retention; extended traces carry 400 days.

A conflict worth knowing: several third-party pricing round-ups quote flat overage rates of $2.50 per 1,000 base traces and $5.00 per 1,000 extended traces. Those figures do not appear on the current official pricing page, which describes overage solely through the compute and storage unit model. Treat the flat per-trace numbers as probably superseded.

Self-hosting is Enterprise-only. OpenTelemetry support is native and requires the LangSmith Python SDK at version 0.4.1 or above.

Braintrust — the eval-first pick

If evaluation is the reason you are shopping, start here. Braintrust organises around Observe, Evaluate and Discover rather than around tracing, and includes a built-in agent that can generate test cases and run evaluations autonomously.

Pricing from Braintrust’s own page, read 16 August 2026: Starter free with $10 of model credits, 1GB of processed data, 10,000 scores per month and 14-day retention; Pro at $249 per month with $249 of model credits, 5GB then $3 per GB, 50,000 scores then $1.50 per 1,000, and 30-day retention; Enterprise adding SAML, custom roles, a BAA and self-hosted deployment.

The company raised an $80 million Series B led by Iconiq at an $800 million post-money valuation, announced 17 February 2026, with Andreessen Horowitz, Greylock and Elad Gil participating.

Galileo, Maxim AI and HoneyHive

Galileo is built around small, fast evaluator models — its Luna-2 family — for scoring production traffic at low latency, which is the right architecture if you want online evaluation on a meaningful share of real traffic. Free at 5,000 traces per month with unlimited users; Pro at $100 per month billed yearly for 50,000 traces; Enterprise adds real-time guardrails and on-premises deployment.

Maxim AI publishes the most granular self-serve ladder in the category: free for 3 seats and 10,000 logs, Professional at $29 per seat per month for 100,000 logs, Business at $49 per seat per month for 500,000 logs and 30-day retention, then Enterprise for in-VPC deployment. It also ships Bifrost, an open-source Go gateway, separately.

HoneyHive is OpenTelemetry-native with a free tier of 10,000 events per month across 5 users, and jumps straight to Enterprise after that — but its Enterprise tier supports fully air-gapped Kubernetes deployment, which is rare and matters in regulated environments.

Weights & Biases Weave

Weave is usage-based, metered on ingestion volume and storage rather than seats, with additional cloud storage at $0.03 per GB per month. Its agent tracing is built on the OpenTelemetry GenAI semantic conventions, so OTel-emitting frameworks work without custom instrumentation. It is also the designated migration target for Humanloop customers — see the vendor-risk section below.

Cloudflare AI Gateway, in depth

If your traffic already goes through Cloudflare, this is the highest value-per-effort option in the entire category, because the core is free on every plan.

AI Gateway is an edge proxy between your application and model providers — OpenAI, Anthropic, Google, Workers AI, xAI, Groq, Replicate and others. You change a base URL and get observability without touching application code.

What is free. Cloudflare names dashboard analytics, caching and rate limiting as core features “available to use on all plans”, and persistent logging is free too within the storage caps below, per Cloudflare’s pricing page (updated 19 May 2026). What varies by plan is log storage: Workers Free stores 100,000 logs in total across all gateways; Workers Paid stores 10 million logs per gateway. When you hit the cap, new logs stop saving until you delete old ones or enable automatic log deletion, which purges oldest-first. Note this is a storage ceiling, not a monthly quota — a distinction several third-party summaries get wrong.

What each log contains. Prompt, response, provider, model, timestamp, status, token usage, estimated cost, duration and client user-agent.

The privacy control that makes it usable. Two separate headers do different jobs. cf-aig-collect-log: false disables logging for a request entirely. cf-aig-collect-log-payload: false keeps the metadata — tokens, cost, model, duration — while suppressing the prompt and response bodies. That second header is the one most teams want and most miss, because it gives you complete cost observability with no prompt retention.

Caching. Exact-match only, on text and image responses. The default cache key is a SHA-256 hash of provider, endpoint, model, auth header and full request body, overridable via cf-aig-cache-key. TTL is configurable from 60 seconds to one month. Semantic caching is not available — it is described as planned. If you expected fuzzy matching on similar prompts, you will not find it here yet.

Spend limits. Dollar-budget rules scoped by model, provider or custom metadata such as per-user or per-team, up to 20 rules per gateway. On breach the default is a 429, or you can configure a fallback to a cheaper model through Dynamic Routing. Cloudflare states plainly that cost tracking is best-effort estimation from token counts and model pricing, and is not guaranteed to match your provider’s actual bill — which is a more honest framing than most of this category offers.

Guardrails. Real-time moderation of both prompts and responses using Meta’s Llama Guard 3 8B running on Workers AI, applied uniformly across every provider you proxy. Billed as Workers AI token inference, so cost scales with the length of what is being evaluated rather than a flat fee.

Data loss prevention. Free on all plans, with two predefined profiles covering financial information and social, insurance and national ID numbers. The full profile set requires a Zero Trust subscription. Note that DLP flags or blocks — it does not redact in place.

OpenTelemetry. AI Gateway exports OTLP spans following the GenAI semantic conventions, with attributes including gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.cost, and — importantly — gen_ai.prompt_json and gen_ai.completion_json, which carry the full prompt and completion. Trace-context headers let you stitch gateway spans into your application’s existing traces. Be deliberate here: if you enable OTel export, full prompt and completion content leaves Cloudflare for your chosen backend by default.

Unified billing. You can pay for third-party model usage on your Cloudflare invoice for a 5 per cent fee on credits purchased — a $100 credit costs $105 — with inference itself passed through at no markup.

Related 2026 launch. Cloudflare’s managed retrieval product, formerly AutoRAG, was relaunched as AI Search and announced on 16 April 2026, in open beta. It is a sibling product, not a feature of AI Gateway.

The honest limitation: AI Gateway is observability at the proxy layer. It sees requests to model providers. It does not see your retrieval step, your tool execution, or your agent’s control flow unless those also make proxied model calls. For agent debugging you will want a tracing tool alongside it — which is exactly what the OTel export is for.

The other gateways

LiteLLM is the self-hosted alternative, and it has the most granular redaction controls of anything reviewed here: a global turn_off_message_logging switch, a per-request x-litellm-enable-message-redaction header, a redact_user_api_key_info option, and a per-call no-log flag that can itself be disabled globally. Its newer OpenTelemetry v2 integration follows the GenAI conventions and is off by default, and explicitly does not capture prompts and responses unless you opt in — the inverse of Cloudflare’s default. Enterprise pricing is quote-only and stated to be based on gateway request capacity rather than tokens.

Vercel AI Gateway takes no markup and no platform fee on tokens, and includes request-level observability, spend and time-to-first-token metrics free. Its extras are individually metered: Custom Reporting at $0.075 per 1,000 writes and $5 per 1,000 queries, Provider Allowlist at $0.10 per 1,000 successful requests, and Zero Data Retention at $0.10 per 1,000 requests. Trace Drains — the OTel export — is Pro and Enterprise only and carries zero included allowance on Pro, meaning billing starts from the first trace. Read 16 August 2026, page dated 1 August 2026.

OpenRouter charges 5.5 per cent on card credit purchases and 5.0 per cent on crypto, with pass-through model pricing. Its distinctive feature is data-policy control: you can restrict routing to zero-retention providers only. It is a router with privacy controls rather than an observability product, and retention is provider-dependent rather than uniformly guaranteed.

Bifrost, from Maxim AI, is an open-source Go gateway with built-in observability using SQLite by default and PostgreSQL for production. Prompt and completion content is captured by default; disable_content_logging keeps usage metadata while dropping content.

Kong AI Gateway targets enterprise governance across LLM, MCP and agent-to-agent traffic. We could not extract reliable pricing from Kong’s own page and are not publishing third-party figures.

The APM incumbents

The strategic point first: if you already pay one of these vendors, evaluate their LLM feature before buying a dedicated tool. A single trace spanning your HTTP handler, your database query and your model call is more useful for debugging than two disconnected tools, and you are already paying for the ingestion.

Datadog renamed its product to Agent Observability and has supported the OTel GenAI semantic conventions since December 2025, with automatic cost calculation across a large model catalogue and PII coverage through Sensitive Data Scanner. We could not find official per-request pricing on Datadog’s own pages and are not publishing the third-party figures in circulation.

Sentry has the most privacy-conservative defaults in this entire page: prompts and completions are off by default and require setting send_default_pii=True plus per-integration flags. LLM monitoring is bundled into all plans including the free Developer tier with no separate surcharge.

Honeycomb is the most standards-pure option. Its Agent Timeline feature, shipped around May 2026, is built entirely on the OTel GenAI conventions with no proprietary SDK required — which is unsurprising given Honeycomb engineers are OpenTelemetry maintainers. Its pricing is event-based: a free tier of up to 20 million events per month, and a Pro tier starting at $150 per month for up to 750 million events, read 16 August 2026.

SigNoz is fully OTel-native and publishes clear numbers: $49 per month base, traces and logs at $0.30 per GB, metrics at $0.10 per million samples. Note its own comparison material concedes that evaluations and prompt management come via integrations rather than natively.

New Relic is the outlier on defaults in the opposite direction from Sentry: prompts and completions are captured in full by default once AI monitoring is enabled, with a kill switch at ai_monitoring.record_content.enabled. There is no LLM-specific SKU — it bills through standard ingest at $0.40 per GB, with 100GB free monthly.

Grafana Cloud offers a free tier of 30,000 generations and 25 million tokens monthly, with an Enterprise minimum commitment reported at $25,000 per year. Dynatrace ships dt-evals, an open-source CLI with more than ten LLM-as-judge evaluators, and accepts three ingestion paths — OpenLLMetry, OTel GenAI conventions and OpenInference. Splunk deserves credit for stating explicitly in its documentation that its cost figure is an estimate and does not reflect actual billing. Coralogix monitors coding agents including Claude Code and Codex CLI over native OTLP, which is an unusual and genuinely useful niche. Middleware correlates GPU utilisation, memory, temperature and power with LLM trace load, which nobody else here emphasises.

The cloud providers’ native tooling

Amazon Bedrock. Model invocation logging is opt-in and disabled by default, writing to S3 or CloudWatch Logs, with full request and response JSON inline up to 100KB and larger payloads referenced in S3. CloudWatch GenAI Observability reached general availability on 13 October 2025, covering token usage, latency percentiles, error rates and cost attribution. Bedrock Guardrails is the one piece of cloud-native pricing we could confirm officially: content filters and denied topics at $0.15 per 1,000 text units, sensitive information filters at $0.10 per 1,000 with regex-based filters free, contextual grounding at $0.10 per 1,000, and human evaluation tasks at $0.21 each.

Google. Vertex AI is rebranding to the Gemini Enterprise Agent Platform through 2026 and documentation URLs are actively redirecting. Cloud Trace is OpenTelemetry-based and explicitly documented as conforming to the GenAI semantic conventions — the clearest conformance statement of the three hyperscalers. Notably, Google steers developers away from storing prompts and responses in log entries, because Cloud Logging caps entries at 256 KiB. Its guardrail product, Model Armor, is stateless by design and does not store content unless Cloud Logging is explicitly enabled.

Microsoft. Azure AI Foundry is rebranding to Microsoft Foundry. Tracing is GA for prompt and hosted agents but still preview for workflow and external agents — read the banner on the page you are relying on. It stores to Application Insights over OpenTelemetry using the GenAI conventions, and Microsoft co-developed a multi-agent semantic convention extension with Cisco Outshift covering spans such as invoke_agent and agent_to_agent_interaction. Inputs, outputs, tool calls, tokens and cost are captured by default as part of tracing; redaction is a manual practice, not automatic.

The standard nobody should assume is settled

This is the most important technical section on this page, because it undermines the single claim every vendor here makes.

The OpenTelemetry GenAI semantic conventions define how a model call should be represented as a span. Every tool above claims support. Here is the actual state of the specification:

• Nothing GenAI-specific is marked Stable. As of July 2026, no GenAI span, event, metric or attribute in the dedicated repository has reached stable status — they remain in Development. The stable attributes you see referenced, such as error.type and server.address, are inherited from general HTTP conventions, not GenAI ones.
• The conventions moved out of the core repository. Release v1.42.0 of the main semantic-conventions repository, on 12 June 2026, deprecated all gen_ai.* content and relocated it to a new dedicated repository. Release v1.43.0 on 3 July 2026 ships none of it. The new repository had no releases or tags at the time of writing.
• The schema has changed repeatedly. Token attributes were renamed in v1.27.0. gen_ai.system became gen_ai.provider.name in v1.37.0. A gen_ai.evaluation.result event was added in v1.38.0, which is the first standards-level acknowledgement that evaluation scores belong in traces. Retrieval spans and cache-token attributes arrived in v1.40.0. invoke_agent was split into client and internal spans in v1.41.0.

The practical consequence. Frameworks emit different generations of this schema right now: some are frozen at an older revision, some emit current and legacy fields simultaneously, and the OpenAI Agents SDK does not emit these conventions natively at all. “OpenTelemetry-compatible” is not yet a guarantee that two tools will agree on what your spans mean.

One design decision you should adopt regardless of tooling. In the specification, prompt and completion content — gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions — is marked opt-in and off by default, carrying an explicit warning that it is likely to contain sensitive information. The GenAI SIG discussion behind that design — raised as an issue and closed through a subsequent pull request rather than settled in the thread itself — gives two reasons: privacy, because prompts “can contain health concerns, ssns, addresses, names, etc.”; and size, because non-GenAI-specific backends “are not optimized for this and it’s expensive to store such data in hot storage”. Both reasons apply to you.

What each tool captures by default

This is the buying decision most teams never consciously make, and it varies enormously.

Read that table before you choose. Turning on an LLM observability tool is, in several of these cases, a decision to begin retaining every prompt your users write — which may include whatever they paste into your product. That is a data protection decision, not a monitoring one, and it should be made deliberately rather than inherited from a default.

Evaluation, and how much to trust an LLM judge

Most teams trace before they evaluate, and the survey data says most never really get to evaluation at all.

LangChain’s State of Agent Engineering survey — 1,340 respondents, fielded 18 November to 2 December 2025 and published 12 June 2026 — found that 89 per cent of teams have implemented some observability and 62 per cent have detailed tracing, but only 52.4 per cent run offline evaluations and just 37.3 per cent run online evaluations on production traffic. Around 29.5 per cent report not evaluating at all. Among teams already in production, online evaluation rises to 44.8 per cent. Human review at 59.8 per cent and LLM-as-judge at 53.3 per cent are the dominant methods. Note the obvious incentive: LangChain sells an evaluation product, and the survey is its own.

The judge reliability question has a real answer now, and it is more nuanced than either camp claims.

The largest study to date is Norman, Rivera and Hughes, “Reliability without Validity”, submitted 17 June 2026 — 21 judge models from 9 providers, across three benchmarks, 118 runs, roughly 541,000 individual judgments. Four findings matter for practice:

1. Raw agreement figures are inflated. The gap between exact-match agreement and Cohen’s kappa, which corrects for chance, runs 33 to 41 percentage points on MT-Bench. A judge advertised at high agreement with humans may be far weaker once chance is accounted for.
2. Judge rankings are benchmark-dependent. The same judges shift by up to 14 positions depending on which benchmark ranks them. There is no context-free “best judge”.
3. Consistency does not imply fairness. The study identifies a consistency-bias paradox: two production-deployed judges showed test-retest reliability above 0.95 while carrying position bias above 0.10. A judge can be perfectly repeatable and systematically wrong in the same direction every time.
4. Verbosity bias was small in their cohort, below 0.011 under a single pairwise rubric — which contradicts the common assumption that judges always reward length, and suggests the effect is rubric-dependent.

On position bias specifically, Shi and colleagues tested 15 judges across more than 150,000 evaluation instances and found it is not random, varies by judge and task, and is strongest when the two candidate answers are close in quality — precisely the comparisons you most need to get right. Earlier work by Chen and colleagues, published at EMNLP 2024, found both human and LLM judges vulnerable to misinformation, authority and other biases, which is a useful corrective: the comparison is not against a perfect human baseline.

What to do with this. Use LLM judges, because the alternative is not evaluating. But swap answer positions and average the results, use a different model family as judge than the one you are evaluating, write rubrics that explicitly instruct against length preference, and validate your judge against a human-labelled set before you trust its absolute scores. Treat judge output as a regression signal, not as ground truth.

Vendor risk is unusually high in this category

Three dated facts that should shape a multi-year commitment.

Humanloop is gone. Anthropic acqui-hired the Humanloop founding team in August 2025, and the platform shut down for existing customers on 8 September 2025. Anthropic confirmed it did not acquire Humanloop’s assets or intellectual property — it hired the people. Customers lost access to prompt workflows, evaluations and observability logs. Weights & Biases published the official migration guide.

Helicone is in maintenance mode. Mintlify acquired Helicone on 3 March 2026 and the founders joined Mintlify. Active feature development has ended; security patches, bug fixes and new model support continue. Third parties are publishing migration guides. Helicone’s pricing page remains live at $79 per month for Pro and $799 for Team, and the open-source core is still Apache-2.0 — but this is not a platform to build a new dependency on.

OpenAI’s Evals product has a shutdown date. Per OpenAI’s own deprecation notices, Evals and Agent Builder become read-only on 31 October 2026 and shut down entirely on 30 November 2026. OpenAI’s own cookbook now recommends migrating to the open-source Promptfoo framework. The Traces dashboard for the Agents SDK is a separate product and continues. If you built evaluation on OpenAI’s native tooling, you have a deadline.

Meanwhile, money is flowing to evaluation specifically — Braintrust at $80 million in February 2026, and Patronus AI at $50 million in June 2026 with Datadog participating as a strategic investor. The category is consolidating around evaluation as the durable product, with tracing increasingly treated as a commodity that the APM incumbents can absorb.

Best for each situation

Best overall default: Langfuse. MIT-licensed, self-hostable, OTel-native, no seat fees, and its new owner already supplied its storage engine.

Best if evaluation is the actual problem: Braintrust, or Galileo if you need low-latency scoring on a large share of production traffic.

Best if you are already on Cloudflare: AI Gateway, immediately. The core is free, it takes a base-URL change, and you can export to a tracing tool later.

Best if you already pay for APM: whichever you already have. Sentry if privacy defaults matter, Honeycomb if standards purity matters, SigNoz if you want published per-GB pricing.

Best for the strictest licence requirements: Comet Opik, Apache-2.0 with the whole platform self-hostable and no enterprise carve-out.

Best for avoiding lock-in entirely: OpenLLMetry for instrumentation plus MLflow or any OTLP backend. You keep the ability to change your mind.

Best for regulated or air-gapped environments: HoneyHive’s Enterprise tier for fully air-gapped Kubernetes, or self-hosted Langfuse or Opik.

Best if you build on LangChain or LangGraph: LangSmith, for the integration depth alone.

Avoid starting new work on: Helicone, which is in maintenance mode; OpenAI’s Evals product, which shuts down on 30 November 2026; and Lunary until its self-hosting claim can be verified.

Two stacks that work

The pragmatic startup stack. Cloudflare AI Gateway at the proxy layer for free cost, caching and rate-limit control, exporting OTel traces to self-hosted Langfuse for tracing, prompt versioning and evaluations. Total software cost: your own infrastructure. This covers the CNCF three pillars without a vendor invoice.

The scaling company stack. Your existing APM — Datadog, Sentry, Honeycomb — for infrastructure traces and alerting, with LLM spans in the same trace, plus Braintrust for evaluation, datasets and experiments. The division of labour is deliberate: your APM owns alerting and system-level debugging, the evaluation tool owns quality regression.

What both have in common: one tool for debugging, one for quality, and prompt content captured deliberately rather than by default.

Frequently asked questions

What is LLM observability?

LLM observability is the practice of instrumenting an application that calls language models so you can see what it actually did, what it cost, and whether the output was any good. It goes beyond conventional application monitoring because an LLM application fails differently: it returns a successful HTTP response containing a wrong answer, or loops through tool calls burning tokens without crashing. A working setup captures traces of every model call and tool invocation, tracks token usage and cost per session, and evaluates output quality either offline against test sets or online against production traffic. The CNCF’s framing for agent systems is three pillars: traces for debugging, costs for unit economics, and an append-only audit record.

What are the best LLM observability tools in 2026?

Langfuse is the best default for most teams, because it is MIT-licensed, genuinely self-hostable, OpenTelemetry-native and charges per unit rather than per seat. Braintrust is the strongest choice if evaluation rather than tracing is your priority, and raised $80 million at an $800 million valuation in February 2026. Comet Opik is the pick for the most permissive licence, being Apache-2.0 with the entire platform self-hostable. Arize Phoenix is excellent for local debugging but is Elastic-licensed rather than open source. If you already pay for Datadog, Sentry, Honeycomb or Grafana, use their LLM features before adding a tool.

What are the best LLM evaluation tools?

Braintrust and Galileo are the strongest dedicated commercial options, with Braintrust organised around experiments and scoring, and Galileo built on small fast evaluator models for low-latency scoring of production traffic. DeepEval is the leading open-source evaluation framework, Apache-2.0 with more than 30 research-backed metrics, with Confident AI as its commercial layer. Ragas remains the standard choice for retrieval-augmented generation specifically. Promptfoo matters more than its profile suggests, because OpenAI’s own migration cookbook recommends it as the successor to OpenAI’s Evals product, which shuts down on 30 November 2026.

Is Langfuse still open source after the ClickHouse acquisition?

Yes. ClickHouse announced the acquisition on 16 January 2026, alongside a $400 million Series D that valued ClickHouse at $15 billion, and committed in its own announcement to keeping the MIT licence and self-hosting as a first-class option. The Langfuse repository remains MIT-licensed with the exception of its enterprise folders, which was already the case before the acquisition. The technical rationale reduces the risk further: Langfuse already ran on ClickHouse in both its cloud and self-hosted deployments, so the acquirer bought an application built on its own database rather than a product it needs to re-platform.

Does Cloudflare AI Gateway cost anything?

The core features cost nothing on any plan. Dashboard analytics, caching, rate limiting and persistent logging are free, including on the Workers Free plan. What varies is log storage capacity: Workers Free holds 100,000 logs in total across all gateways, while Workers Paid holds 10 million logs per gateway, after which new logs stop saving until old ones are deleted. Two features carry costs: Logpush bulk export is Workers Paid only with 10 million requests included monthly then $0.05 per million, and Guardrails is billed as Workers AI inference because it runs a moderation model. Paying for third-party model usage through Cloudflare’s unified billing adds a 5 per cent fee on credits purchased, with no markup on inference itself.

Do I need a dedicated LLM observability tool if I already use Datadog or Sentry?

Usually not as a first step. Every major APM vendor has shipped LLM features, and a single trace that spans your HTTP handler, your database call and your model call is more useful for debugging than two disconnected tools. Sentry bundles LLM monitoring into all plans including its free tier with no surcharge, and Honeycomb built its agent tracing entirely on the OpenTelemetry GenAI conventions with no proprietary SDK. The gap in the incumbents is evaluation: they give you tracing and cost, but dataset management, experiments and systematic quality scoring are weaker or absent. The common pattern is to keep your APM for tracing and alerting, and add a dedicated tool only when evaluation becomes the bottleneck.

Are the OpenTelemetry GenAI semantic conventions stable?

No. As of July 2026 no GenAI-specific span, event, metric or attribute has reached Stable status; they all remain in Development. In June 2026 the conventions were deprecated in the main OpenTelemetry semantic-conventions repository at release v1.42.0 and moved to a dedicated repository, which had no releases or tags at the time of writing. The schema has also changed repeatedly, including renaming token attributes, replacing gen_ai.system with gen_ai.provider.name, and splitting invoke_agent into client and internal spans. In practice frameworks emit different generations of the schema simultaneously, so “OpenTelemetry-compatible” does not yet guarantee that two tools interpret your spans identically.

Do these tools store my users’ prompts?

Several do by default, and the defaults vary sharply, so check before enabling anything. The OpenTelemetry specification marks prompt and completion capture as opt-in and off by default, with an explicit warning that the content is likely to contain sensitive data. Sentry follows that conservative default and requires you to set send_default_pii explicitly, as does LiteLLM’s newer OpenTelemetry integration. Cloudflare AI Gateway, Microsoft Foundry, New Relic and Bifrost capture content by default once enabled. Most offer a middle setting worth knowing about: Cloudflare’s cf-aig-collect-log-payload: false keeps tokens, cost and latency while dropping the prompt and response bodies, which gives you full cost observability with no prompt retention.

How reliable is LLM-as-a-judge for evaluation?

Reliable enough to use as a regression signal, not reliable enough to treat as ground truth. The largest study to date, published in June 2026, covered 21 judge models across roughly 541,000 individual judgments and found that exact-match agreement overstates reliability by 33 to 41 percentage points once corrected for chance agreement, and that judge rankings move by up to 14 positions depending on which benchmark is used. It also identified a consistency-bias paradox, where judges with test-retest reliability above 0.95 still showed substantial position bias — meaning a judge can be perfectly repeatable and consistently wrong. Separate research across more than 150,000 evaluation instances found position bias is worst precisely when two answers are close in quality. Mitigate by swapping answer order and averaging, judging with a different model family than the one under test, and validating against a human-labelled set.

What is the cheapest way to start monitoring an LLM application?

If your traffic already passes through Cloudflare, AI Gateway costs nothing for logging, caching, rate limiting and analytics on any plan, and requires only a base-URL change rather than code instrumentation. If it does not, self-hosted Langfuse or Comet Opik costs only the infrastructure you run it on, and both are free of feature gates on the core product. On managed free tiers, Langfuse includes 50,000 units per month, Comet Opik and Arize AX each include 25,000 spans, and Galileo includes 5,000 traces with unlimited users. Sentry is worth checking first if you already use it, since LLM monitoring is bundled into every plan including the free tier.

Which LLM observability tools can I self-host?

Comet Opik offers the most complete self-hosting position, Apache-2.0 with server, web application, tracing, datasets, experiments, evaluations and prompt management all included and no enterprise carve-out. Langfuse is MIT-licensed and self-hostable, with enterprise administration features such as SCIM and audit logs reserved for paying self-hosters. Arize Phoenix is free to self-host with no feature gates but is licensed under Elastic License 2.0, which prohibits offering it to third parties as a service. MLflow, Laminar and LiteLLM are all self-hostable and open-source. Among commercial platforms, self-hosting is generally enterprise-tier only — this is true of LangSmith, Braintrust, Galileo and Maxim AI — while HoneyHive’s enterprise tier supports fully air-gapped Kubernetes deployment.

What happened to Humanloop?

Anthropic acqui-hired the Humanloop founding team in August 2025 and the platform shut down for existing customers on 8 September 2025. Anthropic confirmed it hired the people rather than acquiring the assets or intellectual property, which meant customers lost access to their prompt workflows, evaluations, observability logs and API integrations on that date. Weights & Biases published the official migration guide positioning Weave as the replacement, and PromptLayer and Agenta published their own. It is the clearest cautionary case in this category, and the reason self-hostable options carry a premium in a buying decision: a hosted vendor can be acquired for its team and switched off within a month.

Related pages

Build with AI: Best AI for coding · Best AI models · Best LLM APIs · Best AI agent frameworks

Run it yourself: Best open-source LLMs · Best local LLMs

Understand the field: What is agentic AI · Best AI agents · Best AI for code review

Prices, licences and features were read from each vendor’s own pages on 16 August 2026; the “Price verified” column says which ones we could confirm there. Free tiers and rate cards in this category change often, so check anything you are budgeting against.