QUICK START:HTMLCSSJavaScriptTypeScriptPythonSQLReactNext.jsNode.jsLinux & UbuntuKotlinSwiftC# / .NETJavaGoRustC++DSASystem DesignDevOpsCybersecurityAI / ML
Advanced 28 min readModule: Module 16: Distributed Observability: OpenTelemetry & Tempo

Distributed Observability: OpenTelemetry & Grafana Tempo

Build unified observability pipelines across distributed microservices: the 3 Pillars of Observability (Metrics, Logs, Traces), OpenTelemetry (OTel) Collector architecture, W3C TraceContext distributed propagation (`traceparent`), and Grafana Tempo distributed tracing.

What You Will Learn in This Lesson

  • The 3 Pillars of Observability and why OpenTelemetry (OTel) is the vendor-neutral industry standard
  • Distributed Trace Context Propagation across HTTP/gRPC boundaries using W3C `traceparent` headers
  • The OpenTelemetry Collector pipeline: Receivers, Processors (Batch, Tail-sampling), and Exporters
  • Visualizing multi-service latency bottlenecks and database query spans in Grafana Tempo and Jaeger

Introduction & Core Concept

When an e-commerce checkout request takes 4.5 seconds and passes through 12 microservices (API Gateway, Auth, Inventory, Payment, Shipping, Notification), looking at isolated server logs is useless. Distributed Tracing assigns a unique `TraceID` to the user's initial click and propagates it across every HTTP header, RPC call, message queue, and database query, generating a unified visual Gantt chart of the exact latency contributed by every span.
WHY DOES THIS MATTER IN THE REAL WORLD?

OpenTelemetry provides vendor neutrality: instrument your code once with OTel SDKs, and export traces, metrics, and logs interchangeably to Prometheus, Tempo, Datadog, or Honeycomb without vendor lock-in.

Syntax & Structure

javascript
// W3C Trace Context Header
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

OpenTelemetry Distributed Trace Context Propagation in JavaScript / Node.js

javascript
javascript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
// OpenTelemetry Distributed Trace Context Propagation Engine
class DistributedTraceContext {
constructor() {
this.traceId = this._generateHex(16); // 128-bit Trace ID (Shared across all microservices!)
this.spanId = this._generateHex(8); // 64-bit Current Span ID
}
_generateHex(bytes) {
let result = '';
for (let i = 0; i < bytes; i++) {
result += Math.floor(Math.random() * 256).toString(16).padStart(2, '0');
}
return result;
}
// Format into standard W3C 'traceparent' header (RFC 00-traceId-spanId-flags)
toW3CHeader() {
return `00-${this.traceId}-${this.spanId}-01`;
}
// Create child span for downstream RPC call
createChildSpan() {
const child = new DistributedTraceContext();
child.traceId = this.traceId; // Retains global trace ID!
child.parentSpanId = this.spanId;
return child;
}
}
// 1. Inbound Request at API Gateway
const gatewaySpan = new DistributedTraceContext();
console.log("=== OpenTelemetry Distributed Tracing Engine ===");
console.log("[API GATEWAY] Generated Trace ID: ", gatewaySpan.traceId);
console.log("[API GATEWAY] Outgoing W3C Header: ", gatewaySpan.toW3CHeader());
// 2. Downstream Payment Microservice receives header & continues trace
const paymentSpan = gatewaySpan.createChildSpan();
console.log("\n[PAYMENT SERVICE] Ingested Trace ID: ", paymentSpan.traceId);
console.log("[PAYMENT SERVICE] New Child Span ID: ", paymentSpan.spanId);
console.log("[PAYMENT SERVICE] Parent Span ID: ", paymentSpan.parentSpanId);
console.log("\n✅ Distributed context propagated seamlessly across network boundaries!");

Line-by-Line Technical Breakdown

1Tail-Based Sampling: Traditional head-sampling decides whether to drop a trace at the start before knowing if it will fail. The OTel Collector with Tail-Based Sampling buffers complete traces in memory, guaranteeing that any trace containing an HTTP 500 error or latency > 2000ms is preserved and exported.

Try It Yourself (Interactive Editor)

Modify the code in real-time and click Run to test live browser output and console logs.

Intelligent Code Runner & Live Sandbox[JAVASCRIPT]
JAVASCRIPT SOURCE EDITOR
Interactive Live Code

Common Mistakes & How to Avoid Them

#1: Using proprietary vendor SDKs (Datadog, New Relic) directly throughout business logic, creating permanent vendor lock-in.

Use OpenTelemetry API/SDKs in application code. You can switch backend observability platforms in minutes via the OTel Collector configuration.

Incorrect / Antipattern
import datadog from 'dd-trace'; // Tight coupling to proprietary SDK
Correct / Professional Solution
import { trace } from '@opentelemetry/api'; // Vendor-neutral OpenTelemetry standard

Industry Best Practices & Professional Standards

  • Instrument code with vendor-neutral OpenTelemetry APIs (`@opentelemetry/api`).
  • Deploy OpenTelemetry Collector as a DaemonSet to receive OTLP telemetry from all pods.
  • Use Grafana Tempo for cost-effective distributed trace storage on object storage (S3/GCS).

Lesson Summary & Core Takeaways

  • Distributed tracing solves microservice latency debugging by connecting multi-service spans.
  • OpenTelemetry (OTel) is the industry standard for metrics, logs, and traces.
  • W3C TraceContext header (`traceparent`) propagates causal context across distributed systems.