Skip to main content

Pydantic AI

Pydantic AI has OpenTelemetry instrumentation built in. Call Agent.instrument_all once at startup and every agent run emits GenAI spans for the run, each model request and each tool call, plus a token usage metric. It writes to whichever tracer and meter providers are installed, so it works with the plain OpenTelemetry SDK and needs no Logfire account. For agents that run as Temporal workflows, see Pydantic AI on Temporal.

TL;DR

Set up an OpenTelemetry tracer and meter provider, then call Agent.instrument_all(InstrumentationSettings(include_content=...)). Each run emits invoke_agent <agent>, chat <model> and execute_tool <tool> spans and records gen_ai.client.token.usage. Pass conversation_id and metadata to agent.run to tie spans to your own IDs and prompt versions.

Note: For framework-agnostic agent patterns, see AI Agent Observability. For other Python agent frameworks, see LangChain and LangGraph.

Running this in production

Storing and querying these traces at production volume is what base14 Scout does. Check out Scout LLM Observability.

Who This Guide Is For​

  • Python developers running Pydantic AI agents who want traces in an OpenTelemetry backend rather than Logfire.
  • Teams running local models through Ollama who want the same spans and token metrics as with a hosted provider.
  • Teams that version their prompts and want the version on every agent span.

Overview​

  • Turn on Pydantic AI instrumentation over the plain OpenTelemetry SDK.
  • Control prompt and completion capture with an environment variable.
  • Read the invoke_agent, chat and execute_tool span tree.
  • Use the gen_ai.client.token.usage metric.
  • Recognize an output validation retry in a trace.
  • Carry a prompt version in run metadata.
  • Turn thinking off for Ollama models.

Signals​

SignalWhat Pydantic AI emitsWhat the example adds
Tracesinvoke_agent, chat and execute_tool spans with GenAI attributes, through the global tracer provider.error.type, base14.gen_ai.cost and base14.prompt.version in a span exporter. FastAPI and psycopg spans, Temporal spans and five hand-written wait spans.
Metricsgen_ai.client.token.usage per request and token type. operation.cost only for models in its price table, and gen_ai.client.operation.time_to_first_chunk only for streamed responses.base14.kyc.* counters and histograms from workflow code. http.server.* from the FastAPI instrumentation and the Temporal SDK metrics.
LogsNone. Pydantic AI writes no log records.OTLP logs through a replay-safe logger provider, so every line carries the trace and span ID.

Prerequisites​

  • Python 3.10 or later. The example uses 3.14.
  • Ollama with a model that supports tool calls pulled. The Quick Start uses qwen3.5:9B. No provider API key is needed for Ollama.
  • An OpenTelemetry Collector. See Docker Compose Setup.
  • A base14 Scout account, optional. The example runs without one.

Compatibility Matrix​

ComponentVersion in the example
pydantic-ai-slim2.49.0, with the openai extra
opentelemetry-sdk, opentelemetry-api1.44.0
opentelemetry-exporter-otlp-proto-http1.44.0
opentelemetry-instrumentation-fastapi, -psycopg, -logging0.65b0
fastapi0.141.1
Python3.14
Ollama0.40.0, with gemma4:e2b and qwen3.5:9B
OpenTelemetry Collector Contrib0.161.0
Exampleai-kyc-onboarding

Last verified 2026-10-07 with Pydantic AI 2.49.0, against open-telemetry/semantic-conventions-genai at commit 4f85037 (2026-10-06). The GenAI conventions are in Development status, so attribute names can change between Pydantic AI releases. Pin pydantic-ai-slim to an exact version and re-check the spans after each upgrade.

The example holds opentelemetry-sdk and opentelemetry-api at 1.44.0. The replay-safe logger in temporalio 1.33.0 lacks the enabled() method that the 1.45.0 API makes abstract, so the worker cannot build its logger provider on 1.45.0. Move the SDK pin when temporalio ships the method. See Pydantic AI on Temporal.

Installation​

Terminal
uv add "pydantic-ai-slim[openai]==2.49.0" \
  opentelemetry-sdk==1.44.0 \
  opentelemetry-exporter-otlp-proto-http==1.44.0

The openai extra covers OllamaProvider, which talks to Ollama's OpenAI-compatible endpoint. The example also installs the temporal extra, which you only need for durable agents.

Quick Start​

This file is a minimal starting point, not part of the example. It sets up tracing and metrics, turns on Pydantic AI instrumentation and runs one agent with one tool on Ollama. Save it as quickstart.py:

quickstart.py
import asyncio

from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from pydantic_ai import Agent, InstrumentationSettings
from pydantic_ai.models.openai import OpenAIChatModel, OpenAIChatModelSettings
from pydantic_ai.providers.ollama import OllamaProvider

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tracer_provider)

meter_provider = MeterProvider(metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter())])
metrics.set_meter_provider(meter_provider)

Agent.instrument_all(InstrumentationSettings(include_content=True))

model = OpenAIChatModel(
    "qwen3.5:9B",
    provider=OllamaProvider(base_url="http://localhost:11434/v1"),
    settings=OpenAIChatModelSettings(openai_reasoning_effort="none"),
)
agent = Agent(
    model,
    name="order-agent",
    instructions="Look up the order with the order_status tool, then answer in one sentence.",
)


@agent.tool_plain
def order_status(order_id: str) -> str:
    """Return the shipping status of an order."""
    return "shipped" if order_id == "A-100" else "not found"


async def main() -> None:
    result = await agent.run(
        "Where is order A-100?",
        conversation_id="order-A-100",
        metadata={"prompt_version": "v1"},
    )
    print(result.output)


asyncio.run(main())
tracer_provider.shutdown()
meter_provider.shutdown()

Pull the model, point the file at your collector and run it:

Terminal
ollama pull qwen3.5:9B
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=order-agent
python quickstart.py

It prints the answer, for example Order A-100 has been shipped. The exporters post to /v1/traces and /v1/metrics under OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_SERVICE_NAME sets service.name. Your backend then shows one trace for the run, with the service name you set:

The Quick Start trace
invoke_agent order-agent
|-- chat qwen3.5:9B                 asks for the tool call
|-- execute_tool order_status
`-- chat qwen3.5:9B                 writes the answer

Every span carries gen_ai.conversation.id=order-A-100, and invoke_agent carries the metadata JSON. The metric gen_ai.client.token.usage arrives with the same service name. If no trace shows up, check the endpoint and see Troubleshooting.

Configuration​

  • TracerProvider with BatchSpanProcessor(OTLPSpanExporter()) exports spans in batches on a background thread, off the request path.
  • MeterProvider with PeriodicExportingMetricReader(OTLPMetricExporter()) exports the token metric. Without a meter provider, Pydantic AI records no metric.
  • Agent.instrument_all applies to every agent that does not set its own instrumentation. Call it before the first run.
  • InstrumentationSettings uses the global tracer and meter providers unless you pass tracer_provider or meter_provider. include_content turns prompt and completion capture on or off.
  • shutdown() on both providers flushes what is still buffered. A long-running service calls it on exit.

The example sets include_content from an environment variable, as shown in Content Capture:

src/kyc_onboarding/telemetry.py (excerpt)
def configure_instrumentation() -> None:
    Agent.instrument_all(InstrumentationSettings(include_content=_capture_content_enabled()))

It also wraps the OTLP span exporter in its own CostAndErrorAttributingSpanExporter. See Adding Cost and Error Type. For the tracer provider a Temporal worker needs, see Pydantic AI on Temporal.

What Pydantic AI Emits​

One assessment run, with the tool calls it is required to make:

One assessment agent run
invoke_agent kyc-assessment
|-- chat qwen3.5:9B
|-- execute_tool compare_identity
|-- execute_tool screen_sanctions
|-- execute_tool check_expiry
|-- chat qwen3.5:9B
`-- execute_tool decision_from_answer     the answer, checked by the output function
SpanOne per
invoke_agent <agent name>Agent run.
chat <model>Model request.
execute_tool <tool name>Tool call, and the output function when the agent has one.

The extraction agent has no tools, so its run is invoke_agent kyc-extraction with one chat gemma4:e2b child, or more when the output is retried.

The GenAI attributes follow the OpenTelemetry GenAI semantic conventions, which are in Development status as of September 2026, so names can change. The main attributes:

AttributeOnValue
gen_ai.conversation.idEvery GenAI spanThe conversation_id passed to agent.run.
gen_ai.agent.name, gen_ai.agent.descriptioninvoke_agentFrom the agent's name and description.
gen_ai.provider.namechatollama with OllamaProvider.
gen_ai.request.model, gen_ai.response.modelchatRequested and returned model.
gen_ai.request.temperature, gen_ai.request.max_tokenschatFrom the model settings.
gen_ai.usage.input_tokens, gen_ai.usage.output_tokenschatTokens for one request.
gen_ai.aggregated_usage.*invoke_agentTokens for the whole run.
gen_ai.response.finish_reasonschatWhy the model stopped.
gen_ai.tool.name, gen_ai.tool.call.arguments, gen_ai.tool.call.resultexecute_toolTool name, arguments and result.
server.address, server.portchatThe Ollama endpoint.
metadatainvoke_agentThe run metadata, as one JSON string.

Run totals use gen_ai.aggregated_usage.*, not gen_ai.usage.*, so a backend that sums gen_ai.usage.input_tokens across spans does not count each request twice. gen_ai.aggregated_usage.* is Pydantic AI's own namespace, not part of the conventions.

Pass the ID you want to search by as conversation_id. The example passes the case ID on every run, so one filter on gen_ai.conversation.id finds every model call for a case:

src/kyc_onboarding/workflows.py (excerpt)
result = await agents.extraction.run(
    prompt,
    usage=self._usage,
    usage_limits=self._usage_limits(EXTRACTION_RUN_REQUEST_LIMIT),
    metadata={PROMPT_VERSION_METADATA_KEY: self._input.extraction_prompt_version},
    conversation_id=self._case.case_id,
)

Token Usage Metrics​

Pydantic AI records gen_ai.client.token.usage, a histogram in {token}, once per model request and token type. Each data point carries gen_ai.token.type (input or output) with the model and provider attributes of the request. It records through the global meter provider unless you pass meter_provider to InstrumentationSettings, so a process with no meter provider records nothing.

Content Capture​

With include_content=True, Pydantic AI records system instructions, prompts, completions, tool arguments and tool results on its spans, in gen_ai.system_instructions, gen_ai.input.messages, gen_ai.output.messages, gen_ai.tool.call.arguments and gen_ai.tool.call.result.

Pydantic AI does not read OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT itself. The example maps it onto include_content:

src/kyc_onboarding/telemetry.py (excerpt)
def _capture_content_enabled() -> bool:
    return (
        os.environ.get("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "true").lower()
        != "false"
    )

The example defaults to true because its documents are synthetic. With include_content=False, the message attributes are still present but hold no prompt or completion text. Token counts and the other attributes are recorded either way.

Output Validation Retries​

When a response fails validation, Pydantic AI sends the model a retry prompt and makes another request. Each request is its own chat span under the same invoke_agent. A retried extraction run looks like this:

An extraction run with one output retry
invoke_agent kyc-extraction
|-- chat gemma4:e2b      the response was plain text, not the output tool call
`-- chat gemma4:e2b      input carries the retry prompt

The first chat span has no error status. The retry shows only as the extra span, and the second span's gen_ai.input.messages holds the retry prompt when content capture is on. retries={"output": 2} allows two retries. After that the run raises UnexpectedModelBehavior, which the example's workflow turns into an escalation with reason invalid_output.

The assessment agent's output function raises ModelRetry when a required tool has not returned yet. The model gets the message and makes another request, which also shows as an extra chat span:

src/kyc_onboarding/agents/assessment.py (excerpt)
missing = missing_tool_calls(ctx)
if missing:
    raise ModelRetry(
        "Before deciding, call the tools that have not returned a result yet: "
        f"{', '.join(missing)}."
    )

To count retries, count the chat children of an invoke_agent span. A run with no retries and no tools has one.

Prompt Version in Run Metadata​

Pass the prompt version in metadata on each run. Pydantic AI records the whole metadata dict as one JSON attribute, metadata, on invoke_agent. The example loads versioned prompts from prompts/extraction_v1.yaml and prompts/assessment_v3.yaml, chosen by EXTRACTION_PROMPT_VERSION and ASSESSMENT_PROMPT_VERSION, and passes the version on every run:

metadata={PROMPT_VERSION_METADATA_KEY: self._input.assessment_prompt_version}

A JSON string is awkward to filter on, so the example's span exporter copies the version out into a flat base14.prompt.version attribute on invoke_agent:

src/kyc_onboarding/telemetry.py (excerpt)
def _prompt_version_attributes(
    attributes: Mapping[str, AttributeValue],
) -> dict[str, AttributeValue]:
    if attributes.get(GEN_AI_OPERATION_ATTRIBUTE) != INVOKE_AGENT_OPERATION_NAME:
        return {}
    try:
        metadata = json.loads(str(attributes.get(RUN_METADATA_ATTRIBUTE)))
    except ValueError:
        return {}
    if not isinstance(metadata, dict) or PROMPT_VERSION_METADATA_KEY not in metadata:
        return {}
    return {PROMPT_VERSION_ATTRIBUTE: str(metadata[PROMPT_VERSION_METADATA_KEY])}

Adding Cost and Error Type with a Span Exporter​

Pydantic AI prices a request only when its price table knows the provider and model. Ollama models are not in it, so on Ollama there is no cost. To add cost, or any attribute derived from Pydantic AI's own, wrap the span exporter. A finished span cannot be changed. The wrapper rebuilds each GenAI span it changes before export and passes the rest through unchanged.

A wrapper can add:

  • A cost attribute on chat, from gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and a price table keyed by gen_ai.request.model.
  • A simulated flag when the model has no row in the table, so a zero cost from a local model is not read as a real price.
  • A flat prompt version on invoke_agent, copied out of metadata as above.
  • error.type on chat and invoke_agent spans with error status, from the span's first recorded exception.

The example's CostAndErrorAttributingSpanExporter does all four. It writes base14.gen_ai.cost and base14.gen_ai.cost.simulated from the prices in _shared/pricing.json, and a run that hits a usage limit gets error.type=pydantic_ai.exceptions.UsageLimitExceeded. Use your own prefix for application attributes. gen_ai. belongs to the semantic conventions.

Ollama Model Settings​

Reasoning models on Ollama think before they answer unless told not to. openai_reasoning_effort="none" in the model settings tells the model not to produce reasoning tokens, as in the Quick Start. Three more settings in the model profile matter on Ollama:

  • supports_json_object_output=False stops PromptedOutput from sending a response_format. With one set, Ollama never returns a tool call.
  • openai_chat_supports_max_completion_tokens=False sends the cap as max_tokens, which Ollama reads. It ignores max_completion_tokens.
  • json_schema_transformer=OpenAIJsonSchemaTransformer keeps format in the output schema. OllamaProvider picks a profile by model name. For Gemma models, that profile's JSON schema transformer moves format: date into the field description, and the model may then leave date fields empty.

Known Gaps​

As of Pydantic AI 2.49.0, verified 2026-10-07:

  • No error.type. A failed chat or invoke_agent span gets error status and a recorded exception only, so Scout cannot group failures by type. The example adds it in a span exporter. See Adding Cost and Error Type.
  • Cost only for priced models. operation.cost on chat spans and the operation.cost histogram appear only when Pydantic AI's price table knows the provider and model. On Ollama nothing is recorded. The example adds base14.gen_ai.cost with a simulated flag.
  • No request duration metric. Pydantic AI records gen_ai.client.token.usage and, for streamed responses only, gen_ai.client.operation.time_to_first_chunk. Read request latency from chat span durations.
  • No log records. Pydantic AI writes nothing to the OpenTelemetry logs API. Add the LoggingHandler, or in a Temporal worker the replay-safe logger provider, yourself.
  • The capture and semconv variables are not read. Map OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT onto include_content as in Content Capture. OTEL_SEMCONV_STABILITY_OPT_IN has no effect on Pydantic AI's spans.
  • metadata is one JSON string on invoke_agent. A filter on a value inside it needs a flat copy, which the example's exporter makes for base14.prompt.version.
  • Run totals use Pydantic AI's own namespace. gen_ai.aggregated_usage.* on invoke_agent is not in the conventions, so a dashboard built on gen_ai.usage.* sees per-request counts only.
  • An output retry has no error status. The first chat span of a retried run ends clean; the retry shows only as an extra chat span. See Output Validation Retries.
  • The span shape depends on the format version. InstrumentationSettings(version=...) defaults to 5. Version 6 puts tool results under the tool role instead of user, and versions 2 to 4 are deprecated. Check the format version as well as the package version after an upgrade.
  • On Ollama, a response_format blocks tool calls and the Gemma profile drops format: date. Set the model profile as in Ollama Model Settings.

What to Look For in Scout​

Follow one case across its agent runs​

Filter spans on gen_ai.conversation.id with the case ID. An auto-approved case shows two invoke_agent kyc-extraction runs with one chat gemma4:e2b each, then invoke_agent kyc-assessment with its chat qwen3.5:9B and execute_tool spans.

Find failed runs and why​

Filter spans on status = Error and group by error.type. In the example's tight_budget scenario, invoke_agent kyc-assessment carries error.type=pydantic_ai.exceptions.UsageLimitExceeded, and the kyc.assess span above it carries base14.kyc.escalation_reason=budget. The case still closes, approved by a reviewer.

Count output retries​

Count the chat children of each invoke_agent span. A run with no tools and no retries has one. In the bad_output scenario one extraction run has two, and the second span's gen_ai.input.messages holds the retry prompt.

Read tokens by model​

Sum gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on chat spans grouped by gen_ai.request.model, or chart the gen_ai.client.token.usage histogram by gen_ai.request.model and gen_ai.token.type. For one run's total, read gen_ai.aggregated_usage.* on its invoke_agent span.

Compare prompt versions​

Group invoke_agent spans by base14.prompt.version. The example runs extraction on v1 and assessment on v3, so a decision rate by version is one grouping on kyc.assess and its child invoke_agent spans.

Running Your Application​

Run quickstart.py as in Quick Start. For the full example, see Running Your Application on the Temporal page.

Troubleshooting​

No GenAI spans​

Agent.instrument_all was not called, or the agent sets its own instrumentation, which takes precedence. Call it once at startup.

Spans appear but no token metric​

No meter provider with an exporting reader is installed. Set a MeterProvider with a PeriodicExportingMetricReader, as in Configuration.

Date fields come back empty​

One cause: a Gemma model on OllamaProvider, whose profile moves format: date out of the schema. Set json_schema_transformer=OpenAIJsonSchemaTransformer on an OpenAIModelProfile, as in the Agent and Model tab of Configuration.

The model never calls a tool​

A response_format is being sent with the tool definitions, and Ollama does not return tool calls under one. Avoid NativeOutput, and set supports_json_object_output=False on the model profile when you use PromptedOutput.

gen_ai.provider.name is not ollama​

The model uses the plain OpenAI provider with a custom base URL. Use OllamaProvider.

Security Considerations​

  • Content capture records applicant data. KYC prompts carry names, dates of birth and addresses. Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false for real data, or scrub in the collector with the attributes or transform processor.
  • Tool results are content too. With capture on, gen_ai.tool.call.result on execute_tool screen_sanctions carries the nearest sanctions list entry's name, even for a clear result.
  • Keep identifiers, not names, in your own attributes. The example puts the sanctions result and score on spans, never the matched name.

Performance Considerations​

  • Each model request adds one chat span and each tool call one execute_tool span. Output retries add spans.
  • Content capture copies the whole conversation into each chat span, and a tool loop resends the conversation on every request.
  • An exporter wrapper like the one in Adding Cost and Error Type does one price lookup and one JSON parse per GenAI span it changes.

FAQ​

Does Pydantic AI need Logfire for OpenTelemetry tracing?​

No, Pydantic AI does not need Logfire. Agent.instrument_all(InstrumentationSettings()) writes to the global OpenTelemetry tracer and meter providers, so any OTLP exporter and backend works.

Which spans does a Pydantic AI agent run produce?​

One invoke_agent <agent> span per run, one chat <model> span per model request and one execute_tool <tool> span per tool call. An output function also shows as an execute_tool span.

How do I turn off prompt and completion capture in Pydantic AI?​

Pass include_content=False to InstrumentationSettings. Pydantic AI does not read OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, so map the variable yourself as the example does.

Does Pydantic AI record token usage metrics?​

Yes, Pydantic AI records the gen_ai.client.token.usage histogram once per model request, split by gen_ai.token.type.

How do I see an output validation retry in a Pydantic AI trace?​

Count the chat spans under the invoke_agent span. A retry adds one, and its input messages carry the retry prompt. The first attempt has no error status.

How do I tag Pydantic AI spans with a prompt version?​

Pass the prompt version in metadata on agent.run. Pydantic AI records metadata as one JSON attribute on invoke_agent. To filter on it, copy the value into a flat attribute in a span exporter wrapper.

How do I group all Pydantic AI model calls for one request?​

Pass your ID as conversation_id to agent.run. It is recorded as gen_ai.conversation.id on every GenAI span of the run.

Does Pydantic AI record cost?​

Only for models in its price table. Pydantic AI then writes operation.cost on the chat span and records an operation.cost histogram. Ollama models have no price, so neither appears. To add a cost attribute for them, wrap the span exporter and compute it from the token counts and your own price table, as in Adding Cost and Error Type.

How do I run Pydantic AI agents durably with tracing?​

Give each agent the TemporalDurability capability and use Temporal's OpenTelemetryPlugin with a replay-safe tracer provider. See Pydantic AI on Temporal.

What's Next?​

Scout Platform Features​

Complete Example​

ai-kyc-onboarding is a KYC (know your customer) onboarding service with two agents on local Ollama models. An extraction agent reads an identity document into typed fields, and an assessment agent checks expiry, identity and sanctions with three tools, then approves, asks again or escalates to a reviewer. Both agents run inside Temporal workflows.

ai-kyc-onboarding/
|-- prompts/                     versioned extraction and assessment prompts
`-- src/kyc_onboarding/
    |-- telemetry.py             providers, instrument_all, cost and error attributes
    `-- agents/
        |-- extraction.py        Ollama model, extraction agent
        |-- assessment.py        assessment agent, output function
        |-- tools.py             check_expiry, compare_identity, screen_sanctions
        `-- prompts.py           prompt loading, metadata key

Source: python/ai-kyc-onboarding.

References​

Last updated on
Was this page helpful?