LangSmith: Seeing What Your LLM Application Did

A traditional application that misbehaves leaves a trail: a stack trace, an error log, a request ID you can grep for. An LLM application that misbehaves usually leaves nothing but a bad answer. The retrieval step might have pulled the wrong document, the prompt template might have rendered with a missing variable, the model might have ignored an instruction it was given three tool calls ago — and from the outside, all of that looks identical. The user just got a wrong or weird response, and you have no record of why.

LangSmith exists to close that gap. It's LangChain's observability and evaluation platform for LLM applications, and despite the name overlap, you don't need to be using LangChain or LangGraph to benefit from it — it instruments OpenAI, Anthropic, CrewAI, the Vercel AI SDK, Pydantic AI, and plain custom Python or TypeScript code just as directly. The pitch is simple: log every step an LLM application takes as a structured trace, then use those traces for three different jobs — debugging today's failure, building a dataset that catches tomorrow's regression, and watching production quality drift before your users complain about it.

What a Trace Actually Captures

The unit of observability in LangSmith is the trace — a single end-to-end record of one run through your application, made up of nested spans. A span might be an LLM call, a retriever lookup, a tool invocation, or any function you've chosen to instrument. Each span records its inputs, its outputs, and its timing, and because spans nest, a trace reconstructs the entire call tree: which retriever ran before which LLM call, which tool the model decided to invoke, and what that tool actually returned.

That nesting is the detail that makes traces more useful than ordinary logs. A log line tells you a function ran. A trace tells you what it ran with, what it produced, where it sat inside the larger request, and how long each part took relative to the others — which is usually exactly the information you need to tell "the retriever returned nothing relevant" apart from "the model ignored good context it was given."

Setting Up Tracing

For a LangChain or LangGraph application, tracing turns on with environment variables and no code changes:

bash

pip install -U langsmith openai

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-langsmith-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"

LANGSMITH_PROJECT is optional — set it to route traces into a named project, or leave it unset and LangSmith creates a default one. Accounts outside the US region also need LANGSMITH_ENDPOINT pointed at their regional API.

For applications built without LangChain, there are two ways to instrument code, and most real projects end up using both.

Automatic wrapping for supported SDKs, which logs every call without touching your call sites:

python

from openai import OpenAI
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())  # every completion call is now logged as a span

The @traceable decorator for everything else — your retrieval function, a tool, or the top-level handler that ties a request together:

python

from langsmith import traceable

@traceable(run_type="tool")
def get_context(question: str) -> str:
    # retrieval logic
    return "...relevant context..."

@traceable
def assistant(question: str) -> str:
    context = get_context(question)          # nested span
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"Use this context: {context}"},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

Because get_context and assistant are both decorated, LangSmith automatically nests the tool span inside the assistant span — you get the full call tree without manually threading a trace ID through your code. The run_type argument ("tool", "retriever", "llm", "chain", and so on) is what lets the LangSmith UI render each span with the right icon and grouping instead of treating everything as an undifferentiated function call.

Note: For LangChain and LangGraph applications specifically, @traceable and the wrapper functions aren't even necessary — every node, chain, and model call is traced automatically the moment LANGSMITH_TRACING is set. That's the practical payoff of the two projects sharing a vendor: LangGraph's explicit node structure (covered in a companion piece on LangGraph's state and control flow) maps directly onto LangSmith's trace tree, one node per span, with no extra instrumentation work.

From One Bad Trace to a Reliable Test Suite

A trace answers "what happened in this one run." That's necessary for debugging, but it doesn't stop the same failure from recurring next week. LangSmith's evaluation tooling is the piece that turns an isolated incident into a regression test.

The workflow has three parts:

Datasets are collections of input/expected-output pairs. You build them by hand-curating representative cases, by exporting a batch of real production traces (including the ones that just failed), or by generating synthetic examples.

Evaluators score your application's output against each dataset example. LangSmith supports several kinds: human review through an annotation queue, code-based heuristics (does the output match a regex, is it valid JSON, is it under a length limit), LLM-as-judge evaluators that use a second model to grade correctness or tone, and pairwise comparisons between two versions of a prompt or pipeline.

Experiments are the result of running your application against a dataset with a chosen set of evaluators — with configurable repetition counts, concurrency, and caching so you can compare experiment N against experiment N-1 and see, example by example, whether a prompt change made things better or worse.

That's the offline half: test before you ship. The online half runs the same evaluators against live production traffic, on a configurable sampling rate, using reference-free evaluators since there's no pre-written "correct answer" for a real user's question. When an online evaluator flags a bad response, that trace can be pushed straight into a dataset — closing the loop from "we noticed a failure in production" to "we now have a permanent regression test for it," without anyone manually re-typing the failing case.

Example: Say a support bot starts giving overly formal answers after a system-prompt tweak. Without evaluation, you'd notice this anecdotally, days later, from user complaints. With an online evaluator scoring tone against a rubric, the drift shows up on a dashboard the same day, and the specific offending traces are one click away from becoming test cases that stop it from reappearing.

Prompt Engineering: Playground and Prompt Hub

LangSmith also treats prompts as versioned artifacts rather than strings buried in source code. The Playground is an interactive editor for iterating on a prompt against a live model — set the input variables, pick a provider (OpenAI, Anthropic, Gemini, and others are supported), and see the actual completion before committing to anything.

Saved prompts live in the Prompt Hub, where every edit is recorded as a commit, similar to version control. You can tag a specific commit — production, v2-shorter, whatever fits your workflow — and pull it into your application at runtime with the SDK:

python

from langsmith import Client

client = Client()
prompt = client.pull_prompt("support-bot-system-prompt", label="production")

push_prompt() does the reverse, publishing a locally edited prompt back to the Hub. The point of this isn't novelty — it's that a prompt change becomes a reviewable, revertible artifact instead of a silent edit inside a Python string, and because the Playground and Prompt Hub sit next to evaluation, you can run a new prompt version against a full dataset before deciding whether the commit is worth tagging production.

Watching Production, Not Just Debugging It

Traces and evaluations both matter more once an application is live and you're not looking at every request by hand. LangSmith's monitoring layer covers that gap with dashboards and alerts built on trace and evaluation data, automation rules and webhooks that can trigger on specific trace conditions (route a low-score trace to an annotation queue, for instance), and an "Engine" feature aimed at automatically surfacing and diagnosing recurring issues in production traces rather than waiting for a human to spot a pattern.

Feedback collection is part of the same loop: an annotation queue lets a reviewer score real traces after the fact, and inline user feedback (a thumbs-up/down captured from your own application's UI, attached back to the originating trace) gives you a signal that doesn't depend on an LLM-as-judge evaluator at all.

Pricing and Deployment Options

LangSmith's pricing model, as published on LangChain's site, is a mix of seat-based and usage-based charges:

Usage beyond the included allowance is billed through two consumption units: LangChain Compute Units (LCU, $1.50 each) for compute-heavy operations across the Engine, Fleet, deployments, and sandboxes, and LangChain Storage Units (LSU, $1.00 each) for traces and stored data. Traces themselves come in two retention tiers — base traces, kept for 14 days, and extended traces, kept for 180 days by default (Enterprise customers can adjust this) — and extended retention costs more per trace than base retention, so it's worth deciding deliberately which traces are worth keeping for six months versus two weeks rather than defaulting everything to extended.

Deployment options scale with the plan: fully managed cloud hosting in the US or EU, a hybrid model where LangChain runs the control plane while your data infrastructure stays in your own environment, and full self-hosting inside your VPC for Enterprise customers who can't send trace data outside their perimeter at all. LangChain also publishes a startup program offering discounted rates and credits (reportedly up to $10,000) for VC-backed companies — worth checking directly if that applies to you, since program terms change independently of the core pricing page.

Common Mistakes to Avoid

Best Practices Worth Adopting Early

Instrument before you need to debug, not after — retrofitting @traceable calls onto a production incident is a worse week than adding them during initial development. Give every span a clear name and run_type so a trace reads like a call tree instead of a flat list of anonymous function calls. Build your first evaluation dataset from real traces rather than hypothetical examples; the failures you've actually seen are more valuable test cases than the ones you imagine. Decide your trace retention strategy deliberately, project by project, instead of leaving everything on the default. And if you're running LangGraph, don't duplicate effort building custom logging — the automatic tracing integration is the more direct path and gives you the graph's node structure for free.

Is LangSmith the Right Fit?

LangSmith isn't the only LLM observability platform — tools like Langfuse cover overlapping ground, and some teams build a thinner custom solution on top of OpenTelemetry instead. Where LangSmith earns its place is in the combination: tracing, evaluation, prompt versioning, and production monitoring under one data model, with genuinely first-class (not bolted-on) support for LangChain and LangGraph if that's your stack, and framework-agnostic SDKs if it isn't. Teams already committed to self-hosting everything, or with a hard requirement that no trace data ever leaves their own infrastructure outside of an Enterprise contract, should weigh the deployment tiers carefully before committing. For most teams shipping an LLM application to real users, though, the alternative to a platform like this isn't a better solution — it's finding out what went wrong from a support ticket instead of a trace.

Conclusion

The hardest part of running an LLM application in production usually isn't the model call — it's everything wrapped around it: retrieval, tool use, prompt assembly, multi-step agent logic, and the fact that any one of those can quietly go wrong while still returning a plausible-looking response. LangSmith's value is that it makes that wrapper inspectable, turns individual failures into datasets that prevent recurrence, and keeps watching after you've shipped instead of only during development. None of that replaces good engineering judgment about what to test and what to monitor — but it does mean that judgment has something to act on besides a bad response and a shrug.

Sources consulted:

• LangSmith Observability — Docs by LangChain

• LangSmith Observability Quickstart

• LangSmith Evaluation Overview

• LangSmith Prompt Engineering Quickstart

• Manage Billing in Your Account — LangSmith Docs

• LangSmith Pricing — LangChain

• LangSmith: Agent & LLM Observability Platform