← Back to Blog

Distributed Tracing Explained: Following a Request Across Microservices

By Yasiru Samarasekara · 6 Sept 2026

A user submits a request. The order service accepts it, calls the payment service, and asks the inventory service to reserve stock. A few seconds later, the user receives an error.

Every service has logs. But which log entries belong to this particular request?

flowchart LR
    client["Client"]
    order["Order Service"]
    payment["Payment Service"]
    inventory["Inventory Service"]

    client -->|"  HTTP  "| order
    order -->|"  HTTP  "| payment
    order -->|"  HTTP  "| inventory

With one request at a time, the answer might be obvious. With hundreds of concurrent requests spread across several service instances, it becomes much harder.

This is the problem distributed tracing helps solve. Before looking at libraries or tracing tools, it is useful to understand what information needs to travel with a request and why.

Logs Tell Us What Happened Locally

Imagine opening the logs from three services:

10:15:30.101 Order Service      Processing order
10:15:30.104 Order Service      Processing order
10:15:30.109 Payment Service    Processing payment
10:15:30.112 Inventory Service  Reserving inventory
10:15:30.115 Payment Service    Payment failed
10:15:30.118 Payment Service    Processing payment

Which order caused the failed payment? Was the inventory reservation part of that same request, or a different one?

Timestamps help narrow the search, but they do not establish a relationship. Requests overlap, clocks can differ between machines, and log collection may introduce delays. A thread name only describes execution inside one process; the next service uses its own threads.

Collecting all these logs in one place makes them easier to search. It does not automatically connect them.

We need something more explicit: context that travels with the operation.

Correlation IDs: The First Improvement

A simple approach is to assign an identifier when a request enters the system, pass it to downstream services, and include it in their logs.

This is often called a correlation ID.

Order Service    correlationId=REQ-123 Processing order
Payment Service  correlationId=REQ-123 Processing payment
Payment Service  correlationId=REQ-123 Payment failed

Now we can search for REQ-123 and find related entries, even when other requests are interleaved.

That is already a substantial improvement. But a shared identifier only groups events. It does not describe which operation called which, how long each step took, or whether two downstream calls ran sequentially or in parallel.

Distributed tracing extends this idea by giving the request a shared identity and describing the work performed within it.

Traces and Spans

A trace represents a connected journey through an instrumented system. Its trace ID, usually written as traceId in application logs, identifies that journey.

A span represents one operation within the trace. It has its own spanId, a name, timing information, and potentially attributes, events, and an error status. A child span records its parent’s span ID, which establishes the relationship between operations.

For example, handling an incoming HTTP request is one operation. Making an outgoing HTTP call is another. Handling that call in the receiving service is another again.

%%{init: {
  "flowchart": {
    "wrappingWidth": 280
  }
}}%%
flowchart TD
    order["Order: handle incoming request<br/>Span A"]
    paymentClient["Order: HTTP call to Payment<br/>Span B"]
    paymentServer["Payment: handle incoming request<br/>Span C"]
    inventoryClient["Order: HTTP call to Inventory<br/>Span D"]
    inventoryServer["Inventory: handle incoming request<br/>Span E"]

    order --> paymentClient
    paymentClient --> paymentServer
    order --> inventoryClient
    inventoryClient --> inventoryServer

All five spans belong to the same trace. The letters are simplified labels; real span IDs are generated identifiers.

Notice that a span is not the same thing as a service. The order service owns several spans here, and it will create new ones for subsequent requests. Database operations can also appear as spans when the relevant instrumentation is present.

The OpenTelemetry trace model describes this structure in more detail.

Same Trace, Different Operations

Suppose the two services produce these logs:

Order Service    traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=1111111111111111 Creating order
Payment Service  traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=3333333333333333 Processing payment

The matching traceId connects the entries. The different spanId values identify the operations active when those entries were written.

The HTTP client span between them might have ID 2222222222222222. It exists even if our application does not write a log while that span is active.

Searching by trace ID finds related logs. Searching by span ID narrows the investigation to one operation. The parent-child relationships live in the span data; a list of log messages alone does not necessarily reveal the complete call tree.

This distinction is useful when debugging a slow request. A log can explain why payment processing failed, while span timing can help establish how much time was spent reaching the payment service and waiting for its response.

How Does the Next Service Know the Trace ID?

Services run in separate processes. A Java variable in the order service is not visible to the payment service.

The context must cross the same boundary as the request.

For HTTP, it travels in headers. W3C Trace Context defines a standard format called traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-2222222222222222-01

For this version of the format:

The receiving service uses the sending span as the parent of its new server span. It retains the trace ID and generates its own span ID. The header does not contain the entire call history.

An optional tracestate header carries additional tracing-system information. Neither header is a place for business identifiers.

The sampled flag participates in recording decisions. It does not guarantee that a trace will be available in storage, and it does not control whether application log statements execute.

Inject, Transport, Extract

Context propagation has three basic steps:

1. Inject: encode the current context into an outgoing carrier, such as HTTP headers.
2. Transport: send that carrier with the request or message.
3. Extract: read the context at the receiving side and make it the basis for local processing.

sequenceDiagram
    participant O as Order Service
    participant C as Instrumented HTTP Client
    participant P as Payment Service

    Note over O: Server span A is current
    O->>C: Call Payment
    Note over C: Start client span B<br/>Inject B's context
    C->>P: HTTP request + traceparent
    Note over P: Extract context<br/>Start server span C, parent B
    P-->>C: HTTP response
    Note over C: End span B
    C-->>O: Return result
    Note over O: Span A is current again

This also explains why blindly copying an incoming traceparent into an outgoing request is incorrect. The outgoing call has its own operation and context. Instrumentation creates that span and injects the appropriate header.

Our application should use an instrumented client and let the framework manage this sequence. OpenTelemetry’s context propagation guide explains the same process across language boundaries.

Connecting Trace Context to Logs

Tracing context becomes especially useful when it appears automatically in ordinary application logs.

In Java logging, MDC, or Mapped Diagnostic Context, provides named values that a logging pattern can read. With SLF4J and Logback, a pattern such as %X{traceId} prints the MDC value named traceId.

The tracing integration updates the MDC as spans become current and restores it when their scopes close. Application code can then remain straightforward:

log.info("Processing payment");

The resulting line can include the active trace and span without passing those identifiers into every method.

MDC is local logging context. Putting a value in MDC does not put it in an HTTP header or a Kafka record. Conversely, receiving a header does not automatically make every field in it available to the logger.

Network propagation and log correlation are separate connections that both need to be configured.

There is also an execution boundary to consider. MDC is normally associated with a thread. Work submitted to an arbitrary executor does not automatically inherit it. Thread pools also reuse threads, so context needs a defined scope and reliable cleanup. See Logback’s MDC documentation for the logging model.

Technical Context and Business Context

A trace identifies a technical execution. A business operation can outlive that execution.

Imagine a transaction that starts with an HTTP request, waits for a payment callback, and later runs through a retry job. These may form several traces, even though they belong to the same business operation.

We will use transactionId to connect that business history.

For example:

transactionId=TXN-2048 traceId=aaa... Initial submission
transactionId=TXN-2048 traceId=bbb... Payment callback
transactionId=TXN-2048 traceId=ccc... Reconciliation job

Here, transactionId is an application-defined identifier. It does not imply a database transaction, an atomic operation spanning services, or a Kafka transaction.

The application should persist it wherever the business workflow needs it. Tracing context is not a substitute for that durable record. Likewise, assigning a transaction ID does not make retries idempotent; duplicate prevention still requires application logic and storage guarantees.

Baggage Carries Additional Context

Baggage is a mechanism for carrying contextual key-value pairs alongside tracing context. With W3C baggage, a request might contain:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-2222222222222222-01
baggage: transactionId=TXN-2048

The first header describes technical ancestry. The second carries additional context that downstream code may need.

An application can use baggage for convenient propagation and separately select which fields become MDC entries. Baggage is not automatically a span attribute or a metric label.

It is also not a secure envelope. Headers can be supplied by callers and may be forwarded to other systems. Keep baggage small, limit its destinations, validate accepted values, and avoid secrets or personal data. Never use a baggage field as proof of identity or permission.

In particular, selecting transactionId for MDC correlation is not an allowlist for all network baggage. At a public boundary, apply an explicit policy for which incoming and outgoing fields are permitted. The OpenTelemetry baggage guidance covers these propagation and security considerations.

For business-critical data, retain transactionId in the application’s stored records or message schema as well. Baggage makes the current execution easier to follow; it should not become the only copy of business context.

Asynchronous Work Still Needs Context

HTTP is only one kind of boundary. In the Kafka article, we looked at producers appending events and consumers processing them independently.

That independence means the consumer may run on another machine, on another thread, and much later than the producer.

flowchart LR
    order["Order Service<br/>Producer context"]
    kafka["Kafka Record<br/>Payload + context headers"]
    consumer["Notification Service<br/>Consumer context"]

    order -->|"&nbsp;&nbsp;Inject and publish&nbsp;&nbsp;"| kafka
    kafka -->|"&nbsp;&nbsp;Read and extract&nbsp;&nbsp;"| consumer

The carrier changes from HTTP headers to Kafka record headers. The underlying idea remains the same: producer instrumentation injects context, and consumer instrumentation extracts it before processing.

Kafka transports the record. It does not automatically instrument the application code that publishes or consumes it.

A simple consumer can continue the producer’s trace through a child span. More complex messaging flows may use span links, which express relationships without forcing all work under one parent. A batch, for example, may contain records from several unrelated traces.

Replay introduces another useful distinction. Yesterday’s event can retain its transactionId, while today’s reprocessing may deliberately start a new trace and link to the original context. Whether to continue or link traces is an instrumentation and workflow choice; do not keep one span open for the entire business lifecycle.

The Kafka article’s replay and idempotency considerations still apply. Observability helps explain repeated processing; it does not prevent repeated side effects.

Logs, Metrics, and Traces Answer Different Questions

These signals complement each other:

An investigation might begin with a metric alert, find a failed transaction in logs, and then follow its trace ID across services. A tracing backend can add a useful timing view, but correlated logs are valuable on their own.

Keep identifiers such as transactionId and traceId out of metric labels. Their many distinct values create high cardinality: instead of a manageable set of time series, the metrics system accumulates series for individual executions.

Trace sampling and log retention are separate policies. A trace ID can appear in logs even when its spans are not retained by a tracing backend. Missing span data does not necessarily mean propagation failed.

Putting This Into Practice With Spring Boot

The examples use four independent repositories, each running one Spring Boot application:

Run the REST pair or the Kafka pair independently. Each repository below includes its own complete build file, configuration, and Java entry point. Use the Groovy DSL and a Gradle Wrapper version of 9.1.0 or later with JDK 25 in each repository.

REST Example: Client and Server

Order Service accepts a request from curl and acts as the REST client when calling Payment Service:

flowchart LR
    client["curl"]
    order["Order Service<br/>localhost:8080"]
    payment["Payment Service<br/>localhost:8081"]

    client -->|"&nbsp;&nbsp;POST /orders&nbsp;&nbsp;"| order
    order -->|"&nbsp;&nbsp;POST /payments&nbsp;&nbsp;"| payment

The example targets Spring Boot 4.1.1 and Java 25 LTS. These choices were checked against the official documentation on 6 September 2026. The Spring Boot project page lists 4.1.1 as stable, and its system requirements support Java 17 through 26. Java 25 is the latest LTS release listed in the Java support roadmap; Java 26 is a non-LTS release.

Spring’s programming model uses Micrometer Observation for instrumented operations and Micrometer Tracing for tracing. OpenTelemetry can supply the tracing implementation underneath. Spring explicitly recommends the Micrometer APIs for application instrumentation in its observability documentation.

flowchart TD
    spring["Spring MVC and RestClient instrumentation"]
    observation["Micrometer Observation"]
    tracing["Micrometer Tracing"]
    otel["OpenTelemetry"]
    logs["MDC and application logs"]
    export["Optional OTLP export"]

    spring --> observation --> tracing --> otel
    tracing --> logs
    otel -.-> export

We will let Spring instrument HTTP. The only tracing API used in our controller will establish the transaction’s baggage scope.

Each build imports Boot’s dependency platform to keep versions compatible. The OpenTelemetry starter supplies the Micrometer-to-OpenTelemetry bridge.

Both services include complete tracing and logging configuration below. W3C propagation carries baggage; baggage.correlation.fields makes transactionId available to MDC. Sampling is set to 100% for the demo, and OTLP export is disabled so no collector is required. See Spring’s tracing guide and configuration reference.

REST Client Repository: Order Service

Create a separate repository named order-service with these files.

order-service/settings.gradle

rootProject.name = 'order-service'

order-service/build.gradle

plugins {
    id 'java'
    id 'org.springframework.boot' version '4.1.1'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(25)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation platform(org.springframework.boot.gradle.plugin.SpringBootPlugin.BOM_COORDINATES)
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-opentelemetry'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
    implementation 'org.springframework.boot:spring-boot-starter-restclient'
}

order-service/src/main/resources/application.yaml

spring:
  application:
    name: order-service

server:
  port: 8080

payment:
  base-url: http://localhost:8081


management:
  otlp:
    metrics:
      export:
        enabled: false
  tracing:
    propagation:
      type: w3c
    sampling:
      probability: 1.0
    baggage:
      correlation:
        fields:
          - transactionId
    export:
      otlp:
        enabled: false

logging:
  include-application-name: false
  pattern:
    correlation: "[service=${spring.application.name} traceId=%X{traceId:-} spanId=%X{spanId:-} transactionId=%X{transactionId:-}] "

order-service/src/main/java/com/example/order/OrderApplication.java

package com.example.order;

import java.util.Map;
import java.util.UUID;

import io.micrometer.tracing.BaggageInScope;
import io.micrometer.tracing.Tracer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestClient;

@SpringBootApplication
public class OrderApplication {

    public static void main(String[] args) {
        SpringApplication.run(OrderApplication.class, args);
    }

    @Bean
    RestClient paymentClient(
            RestClient.Builder builder,
            @Value("${payment.base-url}") String baseUrl) {
        return builder.baseUrl(baseUrl).build();
    }
}

@RestController
class OrderController {

    private static final Logger log =
            LoggerFactory.getLogger(OrderController.class);

    private final RestClient paymentClient;
    private final Tracer tracer;

    OrderController(RestClient paymentClient, Tracer tracer) {
        this.paymentClient = paymentClient;
        this.tracer = tracer;
    }

    @PostMapping("/orders")
    Map<String, String> createOrder() {
        String transactionId = "TXN-" + UUID.randomUUID();

        try (BaggageInScope ignored =
                tracer.createBaggageInScope("transactionId", transactionId)) {
            log.info("Creating order");

            paymentClient.post()
                    .uri("/payments")
                    .retrieve()
                    .toBodilessEntity();

            log.info("Order processing completed");
            return Map.of("transactionId", transactionId,
                    "status", "completed");
        }
    }
}

This endpoint creates a new business transaction, so it generates transactionId. In an existing workflow, load the identifier from the authoritative business record instead. A production implementation would persist the transaction before triggering downstream work.

The baggage scope covers both log statements and the payment call. Closing it restores the previous context, including when the call throws. It does not end Spring’s enclosing server span.

The injected RestClient.Builder is essential: it contains Boot’s instrumentation configuration. Constructing a client independently with RestClient.create() bypasses that setup. Our controller never reads or copies traceparent; the client handles propagation.

Logs before this business scope, including startup messages, will not have this transaction ID. That is expected. Context exists only where it has been established.

REST Server Repository: Payment Service

Create a separate repository named payment-service with these files.

payment-service/settings.gradle

rootProject.name = 'payment-service'

payment-service/build.gradle

plugins {
    id 'java'
    id 'org.springframework.boot' version '4.1.1'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(25)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation platform(org.springframework.boot.gradle.plugin.SpringBootPlugin.BOM_COORDINATES)
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-opentelemetry'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
}

payment-service/src/main/resources/application.yaml

spring:
  application:
    name: payment-service

server:
  port: 8081

management:
  otlp:
    metrics:
      export:
        enabled: false
  tracing:
    propagation:
      type: w3c
    sampling:
      probability: 1.0
    baggage:
      correlation:
        fields:
          - transactionId
    export:
      otlp:
        enabled: false

logging:
  include-application-name: false
  pattern:
    correlation: "[service=${spring.application.name} traceId=%X{traceId:-} spanId=%X{spanId:-} transactionId=%X{transactionId:-}] "

payment-service/src/main/java/com/example/payment/PaymentApplication.java

package com.example.payment;

import io.micrometer.tracing.Baggage;
import io.micrometer.tracing.Tracer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;

@SpringBootApplication
public class PaymentApplication {

    public static void main(String[] args) {
        SpringApplication.run(PaymentApplication.class, args);
    }
}

@RestController
class PaymentController {

    private static final Logger log =
            LoggerFactory.getLogger(PaymentController.class);

    private final Tracer tracer;

    PaymentController(Tracer tracer) {
        this.tracer = tracer;
    }

    @PostMapping("/payments")
    @ResponseStatus(HttpStatus.NO_CONTENT)
    void processPayment() {
        Baggage baggage = tracer.getBaggage("transactionId");
        String transactionId = baggage == null ? null : baggage.get();

        if (transactionId == null ||
                !transactionId.matches("TXN-[0-9a-fA-F-]{36}")) {
            throw new ResponseStatusException(
                    HttpStatus.BAD_REQUEST,
                    "A valid transactionId is required");
        }

        log.info("Processing payment");
        // Simulate payment processing; no real payment is charged.
        log.info("Payment completed");
    }
}

The payment endpoint reads already-extracted baggage and checks its expected demo format. It does not generate a replacement ID or manually populate MDC. The shared correlation configuration handles that.

This is an internal demonstration endpoint, not a payment API contract. In production, authenticate service calls and validate business data independently. The controller check happens after context extraction, so enforce baggage size and field restrictions at ingress as well, before infrastructure logging can observe untrusted values.

Follow the Request Through the Logs

With JDK 25 installed, run ./gradlew bootRun (gradlew.bat bootRun on Windows) in the order-service and payment-service repositories in separate terminals, using the Gradle Wrapper included in each project. Then send a request:

curl -X POST http://localhost:8080/orders

The response contains the generated transactionId. Representative application log excerpts look like this; IDs vary on every run, and timestamps and logger names are omitted here:

[service=order-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=1111111111111111 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Creating order
[service=payment-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=3333333333333333 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Processing payment
[service=payment-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=3333333333333333 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Payment completed
[service=order-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=1111111111111111 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Order processing completed

There are three things to check:

• traceId stays the same across both services.
• spanId differs between the order and payment server operations. Order’s original span becomes current again after the client call returns.
• transactionId stays the same throughout the business scope and matches the response.

Send another request and confirm it gets a new trace and transaction ID. Send requests concurrently and confirm their contexts do not mix. Calling Payment directly without transaction baggage should return HTTP 400.

These are illustrative excerpts, not captured output from a run of these applications. The dependencies, APIs, and properties were checked against official documentation; this article’s sample has not been executed on JDK 25 here.

Kafka Example: Producer and Consumer

This example uses two new repositories: kafka-producer-service and kafka-consumer-service. Start only these two services and a Kafka broker for this example.

A request to the producer’s POST /events endpoint starts the flow. It generates a business transaction ID and publishes an order-created event. The consumer processes that event independently. The producer’s HTTP endpoint is simply a convenient way to trigger the demonstration.

With observations enabled, KafkaTemplate injects the producer span’s trace context and W3C baggage into record headers. The listener container extracts them and opens a processing span before invoking the consumer. Consumer logs therefore have the same traceId and transactionId, with their own spanId. Spring manages the headers and scopes automatically.

Assume a Kafka broker is running at localhost:9092 with an order-created topic already created. Both repositories use Spring Boot 4.1.1 and Java 25. Their complete files follow; no files or services from the REST example are required. Boot supplies the template and listener factory, with tracing enabled through Spring Kafka observations.

Kafka Producer Repository

Create a separate repository named kafka-producer-service with these files.

kafka-producer-service/settings.gradle

rootProject.name = 'kafka-producer-service'

kafka-producer-service/build.gradle

plugins {
    id 'java'
    id 'org.springframework.boot' version '4.1.1'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(25)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation platform(org.springframework.boot.gradle.plugin.SpringBootPlugin.BOM_COORDINATES)
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-opentelemetry'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
    implementation 'org.springframework.boot:spring-boot-starter-kafka'
}

kafka-producer-service/src/main/resources/application.yaml

spring:
  application:
    name: kafka-producer-service
  kafka:
    bootstrap-servers: localhost:9092
    template:
      observation-enabled: true
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

server:
  port: 8082

management:
  otlp:
    metrics:
      export:
        enabled: false
  tracing:
    propagation:
      type: w3c
    sampling:
      probability: 1.0
    baggage:
      correlation:
        fields:
          - transactionId
    export:
      otlp:
        enabled: false

logging:
  include-application-name: false
  pattern:
    correlation: "[service=${spring.application.name} traceId=%X{traceId:-} spanId=%X{spanId:-} transactionId=%X{transactionId:-}] "

kafka-producer-service/src/main/java/com/example/producer/ProducerApplication.java

package com.example.producer;

import java.util.Map;
import java.util.UUID;

import io.micrometer.tracing.BaggageInScope;
import io.micrometer.tracing.Tracer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
public class ProducerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProducerApplication.class, args);
    }
}

@RestController
class EventController {
    private static final Logger log = LoggerFactory.getLogger(EventController.class);
    private final KafkaTemplate<String, String> kafka;
    private final Tracer tracer;

    EventController(KafkaTemplate<String, String> kafka, Tracer tracer) {
        this.kafka = kafka;
        this.tracer = tracer;
    }

    @PostMapping("/events")
    Map<String, String> publish() {
        String transactionId = "TXN-" + UUID.randomUUID();
        try (BaggageInScope ignored =
                tracer.createBaggageInScope("transactionId", transactionId)) {
            log.info("Publishing order-created");
            kafka.send("order-created", transactionId, transactionId).join();
            log.info("Order-created acknowledged by Kafka");
            return Map.of("transactionId", transactionId, "status", "published");
        }
    }
}

The baggage scope covers the send and both log statements. For this minimal event, the transaction ID is both the key and the value; a fuller event schema would retain it in a transactionId field. This durable business value is separate from the baggage carried automatically in headers.

The demo waits for Kafka’s acknowledgement with join(), so a send failure reaches the request and both log statements remain on the request thread. These logs show the enclosing HTTP span; Spring manages the producer span internally. Acknowledgement does not mean the consumer has finished.

Kafka Consumer Repository

Create a separate repository named kafka-consumer-service with these files.

kafka-consumer-service/settings.gradle

rootProject.name = 'kafka-consumer-service'

kafka-consumer-service/build.gradle

plugins {
    id 'java'
    id 'org.springframework.boot' version '4.1.1'
}

group = 'com.example'
version = '0.0.1-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(25)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation platform(org.springframework.boot.gradle.plugin.SpringBootPlugin.BOM_COORDINATES)
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-opentelemetry'
    implementation 'org.springframework.boot:spring-boot-starter-kafka'
}

kafka-consumer-service/src/main/resources/application.yaml

spring:
  application:
    name: kafka-consumer-service
  main:
    web-application-type: none
  kafka:
    bootstrap-servers: localhost:9092
    listener:
      observation-enabled: true
      type: single
    consumer:
      group-id: notifications
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

management:
  otlp:
    metrics:
      export:
        enabled: false
  tracing:
    propagation:
      type: w3c
    sampling:
      probability: 1.0
    baggage:
      correlation:
        fields:
          - transactionId
    export:
      otlp:
        enabled: false

logging:
  include-application-name: false
  pattern:
    correlation: "[service=${spring.application.name} traceId=%X{traceId:-} spanId=%X{spanId:-} transactionId=%X{transactionId:-}] "

kafka-consumer-service/src/main/java/com/example/consumer/ConsumerApplication.java

package com.example.consumer;

import io.micrometer.tracing.Baggage;
import io.micrometer.tracing.Tracer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@SpringBootApplication
public class ConsumerApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConsumerApplication.class, args);
    }
}

@Component
class OrderCreatedListener {

    private static final Logger log =
            LoggerFactory.getLogger(OrderCreatedListener.class);
    private final Tracer tracer;

    OrderCreatedListener(Tracer tracer) {
        this.tracer = tracer;
    }

    @KafkaListener(topics = "order-created")
    void onOrderCreated(String transactionId) {
        Baggage baggage = tracer.getBaggage("transactionId");
        String propagatedId = baggage == null ? null : baggage.get();

        if (!transactionId.matches("TXN-[0-9a-fA-F-]{36}") ||
                !transactionId.equals(propagatedId)) {
            throw new IllegalArgumentException(
                    "Event transactionId must match propagated baggage");
        }

        log.info("Processing order-created");
        // Simulate notification processing; no message is sent.
        log.info("Notification processing completed");
    }
}

This service runs a single-record Kafka listener without an HTTP server. earliest lets a new group read existing records when it has no committed offset.

The listener reads baggage that Spring has already extracted. Its logging configuration supplies the MDC fields automatically. The demo rejects missing or mismatched baggage rather than replacing the event’s business ID. A replay workflow can retain that event ID while deliberately starting a new trace.

Spring closes the listener scope after processing, restoring the previous thread context. Keep the work inside the listener for this example; handing it to another executor requires context propagation across that additional boundary.

Run the Kafka Pair and Follow the Logs

With the broker running and the topic created, run ./gradlew bootRun in the kafka-consumer-service repository and in the kafka-producer-service repository in separate terminals. Then publish an event:

curl -X POST http://localhost:8082/events

The response contains the transaction ID. Representative log excerpts are:

[service=kafka-producer-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=1111111111111111 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Publishing order-created
[service=kafka-producer-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=1111111111111111 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Order-created acknowledged by Kafka
[service=kafka-consumer-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=5555555555555555 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Processing order-created
[service=kafka-consumer-service traceId=4bf92f3577b34da6a3ce929d0e0e4736 spanId=5555555555555555 transactionId=TXN-550e8400-e29b-41d4-a716-446655440000] Notification processing completed

The trace and transaction IDs match across the services. The consumer has its own span, whose parent is the framework’s producer span, not the HTTP span shown in the producer application’s logs. Log ordering across services can vary because consumption is asynchronous. These excerpts are illustrative, not output captured from an executed Kafka demo.

When Context Goes Missing

When related logs stop carrying the expected IDs, look for the point where context stopped following the work. The shape of the logs often indicates where to investigate:

Across services, these failures usually point to an inject, transport, or extract boundary: the sender did not write the context, the headers were lost in transit, or the receiver did not make the extracted context current. Compare the context before sending, the headers arriving at the receiver, and the IDs logged inside the receiving operation to narrow the search.

Within a service, an execution-context boundary needs the same attention. Moving work to another thread or reactive stage requires capturing and restoring context for that work, then restoring the previous scope afterwards. If baggage arrives correctly but transactionId is absent only from logs, investigate the baggage-to-MDC connection described earlier.

Avoiding Duplicate Instrumentation

Context can reach the next service correctly while the spans describing the journey are still confusing. One cause is overlapping instrumentation: a Java agent and the framework may both instrument the same outgoing HTTP call or Kafka operation.

If both create a span for that operation, one real call can appear twice. Depending on how their scopes interact, the spans may be nested unexpectedly or record different parents, making the call tree harder to interpret. The HTTP client span and the receiving server span in our example are expected; two competing spans for the same client operation are a different issue.

Choose one deliberate instrumentation strategy for each boundary. This article uses Spring’s framework-managed observations and Micrometer Tracing for HTTP and Kafka. If you introduce an agent, check which boundaries it covers and disable overlapping instrumentation where necessary, then verify the resulting spans and log correlation.

Exporting Traces to a Backend

The examples so far focus on following an operation through correlated logs. A tracing backend such as Jaeger, Tempo, or a managed platform is an optional addition: it stores and visualises span data so we can inspect timings and relationships across services.

There are two separate flows:

The next service receives context from the request or record; it does not ask the tracing backend which trace to join. This is why the examples can propagate context and produce correlated logs with export disabled.

To add a backend to the Spring Boot 4.1.1 example, enable OTLP trace export using the trace-specific switch described in the configuration section. Configure the exporter endpoint and related settings under management.opentelemetry.tracing.export.otlp.*, as documented in Spring Boot’s OpenTelemetry with OTLP guidance. Use the endpoint and protocol supported by your collector or backend.

Changing the tracing backend should only require changes to the export setup. Business code should continue logging events in the same way, and the application’s transactionId creation, persistence, baggage, and MDC handling should remain independent of where spans are stored.

Following the Operation

A useful place to start is one failing transaction. Find its transactionId, identify the relevant trace, and follow that trace through the services involved. Within each service, span IDs tell you which operation was active when a log was written.

That workflow depends on a few connected pieces: instrumentation creates spans, propagation carries context across boundaries, and logging exposes the current context at the point where work happens.

Once those pieces fit together, a message such as Payment failed becomes much more useful. It belongs to a specific operation, in a specific distributed execution, for a specific business transaction.

References

• W3C — Trace Context
• OpenTelemetry — Traces
• OpenTelemetry — Context propagation
• OpenTelemetry — Baggage
• Logback — Mapped Diagnostic Context
• Spring Boot — System requirements
• Spring Boot — Observability
• Spring Boot — Tracing
• Spring Boot — Common application properties
• Micrometer — Tracing API and baggage scopes
• Micrometer — Context Propagation
• Spring Kafka — Monitoring and observations