One trace ID across Spring Boot, FastAPI and LangChain on Kubernetes with OpenTelemetry and Grafana

A 500 from order-service can come from a timeout in a service further down the call chain. Each service wraps the error and throws it again before it reaches the client. Without a shared request ID, you have to match timestamps across several log streams and guess which lines belong to the same request.

The setup in this article gives every request one ID and sends it to every service the request reaches. Logs, metrics and traces all carry that ID, so a log line of a failed request links to its trace.

The stack is Java and Python on Kubernetes:

• Spring Boot
• FastAPI
• LangChain
• OpenTelemetry
• Grafana LGTM (Loki, Grafana, Tempo, Prometheus)

The demo project has three services and a Makefile that starts the whole environment on Minikube with one command.

Logs, metrics and traces

Logs record events inside one service: requests, errors, calls to other services, retries. Each service writes its own logs, so the logs of one service cannot show the full path of a request across services.

Metrics show the behavior of the system over time: throughput, latency, error rate and saturation. They tell you that something is wrong and where to start, but not which request caused it.

Traces follow one request through all the services. A distributed trace records each service call, each HTTP call and each internal span, with timing and attributes.

In this stack, OpenTelemetry creates a trace_id for every incoming request. The ID is:

• sent to every downstream HTTP call in the W3C traceparent header,
• added to every log line written during the request,
• stored on every span of the trace in Tempo.

This Loki query:

{namespace="apps"} | json | traceId="4bf92f35..."

returns every log line of that request from every service. A click on the traceId field opens the trace in Tempo.

Why LGTM and OpenTelemetry

The Elastic Stack is a common alternative. I chose the Grafana LGTM stack for two reasons.

The license is the first reason. In 2021 Elasticsearch moved to the SSPL license, which the OSI does not accept as open source and which adds limits for self-hosting. Loki, Tempo and Grafana use AGPLv3, and Prometheus and the OpenTelemetry Collector use Apache 2.0. The OSI accepts both licenses.

Resource usage is the second reason. Loki indexes only the labels and not the full log text, so it needs less CPU, memory and storage. When most queries filter on fields like traceId or level, a label index is enough.

The stack has four parts:

• OpenTelemetry Collector: receives OTLP spans from all services, groups them in batches and sends them to Tempo. It runs as a Kubernetes Deployment in the observability namespace.
• Grafana Tempo: stores the traces and answers trace queries. It accepts OTLP over gRPC on port 4317.
• Grafana Loki and Promtail: Promtail runs as a DaemonSet, reads the pod logs and sends them to Loki. Grafana turns the traceId field of each log line into a link to Tempo.
• Prometheus and Grafana: Prometheus scrapes /actuator/prometheus on Spring Boot and /metrics on FastAPI. Pod annotations tell it which pods to scrape. Grafana is the UI for all three backends.

Architecture

The services send data only to the OpenTelemetry Collector. Prometheus and Promtail pull the metrics and logs themselves, so the pods need no sidecar.

The Spring Boot OTLP exporter uses HTTP/protobuf on port 4318. The Python services use gRPC on port 4317. The Collector accepts both protocols, so each runtime can use its default.

The reference project

The cluster setup follows the Minikube profiles note. The project has its own Minikube profile, its own kubectl context and its own kubeconfig file. make cluster-start creates the observability-demo profile and exports its context to ~/.kube/observability-demo.yaml.

# Create the cluster, install the stack, build and deploy
make all

# Forward all UIs and service ports to localhost
make port-forward        # Grafana :3000  Prometheus :9090  services :8080–8082

# Generate traffic to produce traces, logs, and metrics
make seed                # 10 orders through the full chain

# Open k9s on this cluster
make k9s
# k9s --context observability-demo
# k9s --kubeconfig ~/.kube/observability-demo.yaml

# Per-session isolation
export KUBECONFIG=~/.kube/observability-demo.yaml

Every kubectl command in the Makefile uses --context=observability-demo, and every helm command uses --kube-context=observability-demo. The Makefile works on this cluster whatever context is active in ~/.kube/config.

Instrumentation: Spring Boot

Spring Boot 3 with Micrometer Tracing does most of the work. The service needs three dependencies:

<!-- Bridges Micrometer's tracing API to the OpenTelemetry SDK -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>

<!-- Exports spans via OTLP HTTP/protobuf to the OTEL Collector -->
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

<!-- Exposes /actuator/prometheus for Prometheus scraping -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

In application.yml, set the Collector address and sample 100% of the requests:

spring:
  application:
    name: order-service
  jackson:
    property-naming-strategy: SNAKE_CASE

management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://${OTEL_COLLECTOR_HOST:otel-collector.observability.svc.cluster.local}:4318/v1/traces
  endpoints:
    web:
      exposure:
        include: health,prometheus

Port 4318 is OTLP over HTTP/protobuf, which the Spring Boot exporter uses. If you set the gRPC port 4317 by mistake, the service shows no exception and no timeout, and no spans arrive in Tempo.

Micrometer Tracing writes traceId and spanId to the MDC. logstash-logback-encoder adds them to every JSON log line:

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <customFields>{"service":"${appName}"}</customFields>
        <includeMdcKeyName>traceId</includeMdcKeyName>
        <includeMdcKeyName>spanId</includeMdcKeyName>
    </encoder>
</appender>

Every log line written during a request now has a "traceId":"4bf92f35..." field. The application needs no other change.

Spring also propagates the trace over HTTP. When order-service calls inventory-service with a RestClient built by Spring, Spring adds the W3C traceparent header to the request.

Instrumentation: FastAPI

Python needs more setup. Five packages cover tracing, metrics and structured logging:

opentelemetry-sdk>=1.25.0
opentelemetry-instrumentation-fastapi>=0.46b0
opentelemetry-exporter-otlp-proto-grpc>=1.25.0
prometheus-fastapi-instrumentator>=6.1.0
structlog>=24.2.0

At startup, the service creates the tracer provider and instruments the FastAPI app:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

resource = Resource.create({"service.name": SERVICE_NAME})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint=f"{OTEL_COLLECTOR_HOST}:4317", insecure=True)
    )
)
trace.set_tracer_provider(tracer_provider)

app = FastAPI()
FastAPIInstrumentor().instrument_app(app)

The Python exporter uses gRPC on port 4317.

To add the trace ID to the logs, structlog needs one custom processor. It reads the active span when the log line is written:

import structlog
from opentelemetry import trace

def add_otel_context(logger, method, event_dict):
    span_context = trace.get_current_span().get_span_context()
    if span_context.is_valid:
        event_dict["traceId"] = format(span_context.trace_id, "032x")
        event_dict["spanId"]  = format(span_context.span_id,  "016x")
    return event_dict

structlog.configure(
    processors=[
        structlog.stdlib.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        add_otel_context,
        structlog.processors.JSONRenderer(),
    ],
    ...
)

Every service must write the field with the same name, traceId, in camelCase. Spring Boot writes traceId, and the Grafana link to Tempo looks for traceId. If one service writes trace_id or TraceId, the query for one request misses its log lines and the link to the trace does not appear.

A custom span needs a few lines:

tracer = trace.get_tracer(SERVICE_NAME)

@app.get("/inventory/{product_id}")
async def get_inventory(product_id: str):
    with tracer.start_as_current_span("inventory.lookup") as span:
        span.set_attribute("product.id", product_id)
        # ... logic

In Tempo, this span appears under the span of the HTTP request, with its own duration and attributes.

Instrumentation: LangChain

By default, OpenTelemetry traces do not show LangChain operations such as chain calls, LLM calls, tool calls and agent steps. OpenInference, from Arize AI, adds OpenTelemetry instrumentation for LangChain:

openinference-instrumentation-langchain>=0.1.19

Call the instrumentor after you set the global tracer provider:

from openinference.instrumentation.langchain import LangChainInstrumentor

trace.set_tracer_provider(tracer_provider)     # must come first
LangChainInstrumentor().instrument()

No other change is needed. Every chain call now creates spans under the span of the request. In Tempo, a /recommendations call looks like this:

HTTP POST /recommendations  (345ms)
  └─ langchain.recommend  (340ms)        ← custom span from the handler
       └─ langchain.chain  (335ms)       ← OpenInference
            ├─ langchain.llm  (310ms)    ← LLM call: prompt + completion + tokens
            └─ langchain.output_parser   (1ms)

With a real LLM, such as OpenAI or Anthropic, the langchain.llm span contains the prompt, the completion and the token counts. The demo project uses FakeListLLM, which creates the same spans, so the setup is the same with any model.

The /recommendations response has an agent_trace_id field with the current trace ID. A client can put it in a bug report, and support can open the trace with it:

def _current_trace_id() -> str:
    span_context = trace.get_current_span().get_span_context()
    return format(span_context.trace_id, "032x") if span_context.is_valid else "0" * 32

How the correlation ID propagates

order-service creates a trace_id for every HTTP request it receives. When it calls inventory-service, the request carries this header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  │                               │               └─ flags
              │  └─ trace_id (128-bit hex)       └─ parent span_id (64-bit)
              └─ version

The receiving service reads the header, takes the trace_id and adds its spans to the same trace. The LangChain spans from OpenInference are children of the FastAPI span. So one trace_id covers the whole request, from the first service to the last LLM call.

structlog reads the trace_id from the active span every time the code writes a log line. Every logger.info(...) in a request handler includes it, so the code never passes the ID as an argument. In Spring Boot the MDC does the same.

Navigating Grafana

Run make seed to create some traces and logs, then make port-forward, and open http://localhost:3000 with user admin and password admin.

Follow one request through all the services:

1

Explore, Loki: click the compass icon in the left sidebar and select the Loki data source. Type {namespace="apps", app="order-service"} in the query field and run the query. Grafana shows the logs of the order service.

2

Open a log line: click a log line to see its fields. The JSON has a traceId field. Click the Tempo link next to it to open the trace.

3

Read the trace in Tempo: the waterfall view shows all the spans of the request. The header shows the trace ID, the start time, the total duration and the number of services. Each bar is one span, and its width is its duration.

4

Inspect a span: click a row in the waterfall. The panel below shows the span attributes: HTTP method, status code, URL, duration and the custom attributes from the instrumentation.

5

Find slow requests: in Explore, select Tempo, open the TraceQL tab and run {name="http post /orders"}. The table lists every order trace with its duration, so slow requests are easy to find.

6

Check the Prometheus scraping: in Explore, select the Prometheus data source and run up{kubernetes_namespace="apps"}. The result has 3 series, one for each service, and each has the value 1 (up).

Useful Prometheus queries:

# Request rate per service (last 5 minutes)
sum(rate(http_server_requests_seconds_count{job="kubernetes-pods-annotation"}[5m])) by (app)

# P95 latency across all services
histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le, app))

# Error rate (non-2xx responses)
sum(rate(http_server_requests_seconds_count{status!~"2.."}[5m])) by (app)

Useful Loki queries:

# All logs for a specific trace
{namespace="apps"} | json | traceId="<paste-trace-id>"

# Errors across all services
{namespace="apps"} | json | level="error"

# LangChain recommendation calls with output
{app="recommendation-agent"} | json | message="recommendation_response"

Quick reference

This setup covers one Kubernetes cluster. That is enough for local development and for many production systems. Tracing across clusters needs more work: the traceparent header must pass through the network boundaries of a service mesh, several Collectors must route spans to the backends, and Grafana needs the mesh telemetry as a fourth data source.

The configuration in each service is small. The rule that matters most is to use the same field name, traceId, in every service.