When you run a backend service locally, debugging is easy. You can print logs, step through the code with a debugger, and see exactly what is happening. In production, things are very different. Your service might be running across dozens of containers, across multiple machines, with hundreds or thousands of requests happening at the same time.

If you do not have a medium membership click here for a free link
Part 9: Pub/Sub and Reliable HTTP Callbacks

When something breaks in production you usually cannot attach a debugger. You need to observe the system from the outside and figure out what went wrong.

This is what observability is about.

Observability is the ability to understand the internal state of a system using the data it produces.

There are three main pillars of observability:

  • Logs
  • Metrics
  • Traces

Each of these gives a different perspective on what the system is doing.

Logs

Logs are the most basic form of observability. They are simply structured records of events happening inside the application.

For example:

2026-03-05T10:12:45Z INFO request received user_id=123 path=/login
2026-03-05T10:12:45Z ERROR database timeout query=SELECT * FROM users

Logs answer questions like:

  • What happened?
  • When did it happen?
  • What input caused it?

In Go, you can log using the standard log library.

package main

import (
 "log"
)

func main() {
 log.Println("service started")

 userID := 123
 log.Printf("processing request for user %d", userID)
}

In production systems logs are usually structured JSON so they can be indexed by systems like Elasticsearch, Loki, or Datadog.

Example structured log:

{
  "level": "error",
  "message": "database timeout",
  "query": "SELECT * FROM users",
  "timestamp": "2026-03-05T10:12:45Z"
}

Libraries like zerolog or Uber's zap are commonly used in Go services because they are fast and produce structured logs.

Metrics

Metrics are just the numerical measurements collected over time about the application or service.

Examples of things measured can be:

  • request latency
  • number of HTTP requests
  • database query duration
  • memory usage
  • number of active goroutines

Metrics are useful because they allow you to monitor the health of your system and trigger alerts. You can use metrics to set alerts when for example, requests per second suddenly drops or latency increases above 500ms.

A very common monitoring stack is Prometheus + Grafana.

Prometheus periodically scrapes metrics from your service through an HTTP endpoint /metrics. Prometheus is also a time series database. All these metrics get stored there and then can be queried in visualization tools like Grafana.

In Go you can expose metrics using the Prometheus client library.

In your project do,

go get github.com/prometheus/client_golang/prometheus

and then in your code, you can initialize Prometheus metric, like this counter which counts the number of HTTP requests received by the service.

package main

import (
 "net/http"

 "github.com/prometheus/client_golang/prometheus"
 "github.com/prometheus/client_golang/prometheus/promhttp"
)

var requestCounter = prometheus.NewCounter(
 prometheus.CounterOpts{
  Name: "http_requests_total",
  Help: "Total number of HTTP requests",
 },
)

func handler(w http.ResponseWriter, r *http.Request) {
 requestCounter.Inc()
 w.Write([]byte("hello"))
}

func main() {
 prometheus.MustRegister(requestCounter)

 http.HandleFunc("/", handler)
 http.Handle("/metrics", promhttp.Handler())

 http.ListenAndServe(":8080", nil)
}

Now Prometheus can scrape metrics from:

http://localhost:8080/metrics

Now we can use these metrics to build dashboards in Grafana and set alerts without digging through logs.

Traces

Logs tell you what happened and metrics tell you how often it happens, but neither shows you how a single request moves through the system.

Modern backend systems are rarely a single service. A single user request might pass through multiple services before returning a response.

For example, imagine a simple request to fetch a user profile:

If the request takes 2 seconds, which component caused the delay?

This is where distributed tracing becomes useful. A distributed trace records the entire journey of a request across multiple services. Every step of the request is recorded as a span.

A span represents a single unit of work. For example:

HTTP Request (2.1s)
 ├─ Auth Service (200ms)
 ├─ User Service (1.8s)
 │    └─ Database Query (1.7s)
 └─ Cache Lookup (20ms)

From this trace we immediately see that the database query inside the user service is the bottleneck. Without tracing, this kind of insight is extremely difficult to get in distributed systems.

Trace IDs and Span IDs

To track a request across services, each request is assigned a trace ID. All the work done for that request shares the same trace ID.

Within the trace, each operation has its own span ID.

Trace ID: 9f2d1a4c

Span 1: API Gateway
Span 2: Auth Service
Span 3: User Service
Span 4: Database Query

When one service calls another service, it passes the trace ID along with the request (usually through HTTP headers). This allows the tracing system to reconstruct the entire call chain.

Example Trace Flow

Suppose a user makes a request:

GET /users/123

The trace might look like this:

Trace: 7ab91f
API Gateway (120ms)
   ↓
Auth Service (40ms)
   ↓
User Service (900ms)
   ↓
PostgreSQL Query (870ms)

From this trace it is clear that the database query is responsible for most of the request time.

Tracing tools like Jaeger and AWS XRAY allow you to visualize this request as a timeline and make a graph of all the services the request passed through, making it easy to see the exact flow and where time is being spent.

OpenTelemetry

Now that we understand logs, metrics, and traces, the next question is: how do we collect all this data from our application? Every monitoring tool has its own SDK or library

If you want traces in Jaeger, you use the Jaeger client.
If you want metrics in Prometheus, you use the Prometheus client.
If you want logs in Loki, you use another library.

This creates a problem. Applications become tightly coupled to specific monitoring vendors. Switching tools means rewriting instrumentation across the entire codebase.

OpenTelemetry solves this problem.

OpenTelemetry is an open standard for collecting telemetry data from applications. It provides SDKs for logs, metrics, and traces, and sends that data to different backends without changing your application code.

Instead of writing instrumentation specific to one tool, you instrument your application using OpenTelemetry and then configure where the data should go.

The application only talks to OpenTelemetry. The collector is responsible for exporting the data to different monitoring systems. This separation makes observability infrastructure much easier to change and scale.

OpenTelemetry Components

OpenTelemetry has three main components.

  1. Instrumentation

Instrumentation is the code added to your application to generate telemetry data.

For example:

  • starting a span when a request begins
  • recording request latency
  • logging an error

Instrumentation can be manual or automatic. Manual instrumentation looks like this:

ctx, span := tracer.Start(ctx, "get-user-profile")
defer span.End()

This records a span called get-user-profile as part of the request trace.

OpenTelemetry also provides automatic instrumentation for common frameworks like HTTP servers, gRPC, and database drivers so that traces can be generated without much manual code.

2. SDK

The SDK collects telemetry data inside your application and prepares it for export.

It handles things like:

  • batching data
  • sampling traces
  • exporting metrics
  • attaching metadata such as service name or environment

The SDK runs inside your application process.

3. Collector

The OpenTelemetry Collector is a separate service that receives telemetry data from applications. It acts as a central pipeline for observability data. Applications send telemetry using the OTLP protocol, and the collector then exports it to different systems.

The Collector can route data to multiple destinations. This architecture prevents every service from needing to talk directly to multiple monitoring systems.

Hands-on: Using OpenTelemetry to collect Logs, Metrics and Traces

In this hands-on, we will use the OpenTelemetry SDK and send logs to Loki, metrics to Prometheus and build a dashboard in Grafana and visualize the traces in Jaeger.

If you want to follow along, to get started, download this docker-compose file.

Configuring the OpenTelemetry Collector

Now we need to tell the OpenTelemetry Collector where it needs to send the collected information to. For this, we can define pipelines in a otel-config.yaml file in the same directory as the docker-compose file.

This configuration defines three telemetry pipelines.

Trace Pipeline

Go Service → OTLP → Collector → Jaeger

Traces generated by the service are forwarded to Jaeger.

Metrics Pipeline

Go Service → OTLP → Collector → Prometheus

Metrics are exported in a format that Prometheus can scrape.

Log Pipeline

Go Service → OTLP → Collector → Loki

Logs are forwarded to Loki for indexing and querying.

The collector also runs a batch processor, which groups telemetry data together before exporting it. This reduces network overhead and improves performance.

Prometheus Configuration

We also need to configure one more thing. OpenTelemetry Collector does not push the metrics to Prometheus. Prometheus scrapes metrics from the Collector, not the other way around. We need to tell it exactly where to scrape from.

For this, we can define pipelines in a prometheus.yaml file in the same directory as the docker-compose file.

Prometheus will scrape metrics from the collector every 5 seconds.

Starting the Observability Stack

Now run this docker-compose file using

docker compose up

This will start the OpenTelemetry Collector, Prometheus, Grafana, Loki and Jaeger locally. After this you’ll have these running at:

Grafana UI     http://localhost:3000
Prometheus     http://localhost:9090
Jaeger UI      http://localhost:16686
Loki API       http://localhost:3100

Instrumenting a Go Service with OpenTelemetry

Now we will build a simple Go backend service and add telemetry to it. Start by installing OpenTelemetry dependencies:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric
go get go.opentelemetry.io/otel/exporters/otlp/otlplog

Create main.go

This service produces three types of telemetry

Metrics
Every request increments:

http_requests_total

This metric is exported to Prometheus.

Traces
Each request generates a span:

hello-handler

This span is part of a trace and is sent to Jaeger.
The span also records metadata:

endpoint=/hello

Logs
The service also produces standard logs:

handling request

These logs can be forwarded through the collector to Loki.

Viewing the Telemetry Data

Once the service is running and the observability stack is started, we can begin generating telemetry data and viewing it in the monitoring tools.

Start the Go service:

go run main.go

Now send a few requests to the service.

curl http://localhost:8080/hello

Each request will generate:

  • one log entry
  • one metric increment
  • one trace with a span

Now we can visualize each of these.

Viewing Traces in Jaeger

Open the Jaeger UI.

http://localhost:16686

In the search panel:

Service: go-service

Click Find Traces.

You should see traces generated from the requests we made.

Viewing Metrics in Grafana

Next we can inspect the metrics collected from the service using Prometheus in Grafana.

Open Grafana:

http://localhost:3000

Default credentials:

username: admin
password: admin

Now add Prometheus as a data source.

Settings → Data Sources → Add Data Source → Prometheus

URL:

http://prometheus:9090

Once the data source is configured you can create dashboards.

For example a simple panel could display:

http_requests_total

This produces a live graph showing requests hitting the service.

Viewing Logs in Loki

Logs are indexed by Grafana Loki and can also be viewed inside Grafana.

In Grafana:

Explore → Data Source → Loki

Then run a query like:

{service="go-service"}

You should see log entries similar to:

Putting Everything Together

At this point we now have full observability for the Go service. Logs help us understand what happened. Metrics help us understand how often something happens and whether the system is healthy. Traces help us understand how requests move through the system.

Together these three signals allow us to debug complex distributed systems much more effectively.

Until next time!