Generate traces and metrics with Go

This document describes how to modify a Go app to collect trace and metric data using the open source OpenTelemetry framework, and how to write structured JSON logs to standard out. This document also provides information about a sample app that you can install and run. The app is configured to generate metrics, traces, and logs.

About context

OpenTelemetry's Context is a mechanism for carrying execution-scoped values across APIs within a process. An important use of context is to carry the current active span so it can be modified, or referenced as the parent of any new spans when they are created. To summarize:

• Context refers to the mechanism to propagate execution-scoped values, including the current active span, across APIs within a process.

• Span Context is an immutable object on every span that includes the trace ID, the span ID, and flags and state for the trace.

• Propagation is the is the mechanism that moves context between services and processes.

The Go standard library's context.Context also carries scoped values across API boundaries. Typically, handler functions in a server receive an incoming Context and pass it through the call chain to any clients making outgoing requests.

Go's standard library context.Context is used as the implementation of OpenTelemetry Context in Go.

Before you begin

Enable the Cloud Logging API, Cloud Monitoring API, and Cloud Trace API APIs.

Enable the APIs

Instrument your app to collect traces, metrics, and logs

To instrument your app to collect trace and metric data, and to write structured JSON to standard out, perform the following steps as described in subsequent sections of this document:

1. Configure the main function
2. Configure OpenTelemetry
3. Configure structured logging
4. Add instrumentation to the HTTP server
5. Link trace spans with logs and metrics
6. Add instrumentation to the HTTP client
7. Write structured logs

Configure the main function

To configure the app to write structured logs and to collect metrics and trace data by using OpenTelemetry, update the main function to configure the Go structured logging package, slog, and to configure OpenTelemetry.

The following code sample illustrates a main function that calls two helper functions, setupLogging() and setupOpenTelemetry(). These helper functions configure the logging package and OpenTelemetry:

func main() {
	ctx := context.Background()

	// Setup logging
	setupLogging()

	// Setup metrics, tracing, and context propagation
	shutdown, err := setupOpenTelemetry(ctx)
	if err != nil {
		slog.ErrorContext(ctx, "error setting up OpenTelemetry", slog.Any("error", err))
		os.Exit(1)
	}

	// Run the http server, and shutdown and flush telemetry after it exits.
	slog.InfoContext(ctx, "server starting...")
	if err = errors.Join(runServer(), shutdown(ctx)); err != nil {
		slog.ErrorContext(ctx, "server exited with error", slog.Any("error", err))
		os.Exit(1)
	}
}

After you configure the logging package, to link your logs to your trace data, you must pass the Go Context to the logger. For more information, see the Write structured logs section of this document.

Configure OpenTelemetry

To collect and export traces and metrics by using the OTLP protocol, configure the global TracerProvider and MeterProvider instances. The following code sample illustrates the setupOpenTelemetry function, which is called from the main function:

func setupOpenTelemetry(ctx context.Context) (shutdown func(context.Context) error, err error) {
	var shutdownFuncs []func(context.Context) error

	// shutdown combines shutdown functions from multiple OpenTelemetry
	// components into a single function.
	shutdown = func(ctx context.Context) error {
		var err error
		for _, fn := range shutdownFuncs {
			err = errors.Join(err, fn(ctx))
		}
		shutdownFuncs = nil
		return err
	}

	// Configure Context Propagation to use the default W3C traceparent format
	otel.SetTextMapPropagator(autoprop.NewTextMapPropagator())

	// Configure Trace Export to send spans as OTLP
	texporter, err := autoexport.NewSpanExporter(ctx)
	if err != nil {
		err = errors.Join(err, shutdown(ctx))
		return
	}
	tp := trace.NewTracerProvider(trace.WithBatcher(texporter))
	shutdownFuncs = append(shutdownFuncs, tp.Shutdown)
	otel.SetTracerProvider(tp)

	// Configure Metric Export to send metrics as OTLP
	mreader, err := autoexport.NewMetricReader(ctx)
	if err != nil {
		err = errors.Join(err, shutdown(ctx))
		return
	}
	mp := metric.NewMeterProvider(
		metric.WithReader(mreader),
	)
	shutdownFuncs = append(shutdownFuncs, mp.Shutdown)
	otel.SetMeterProvider(mp)

	return shutdown, nil
}

The previous code sample configures the global TextMapPropagator to use the W3C Trace Context format for propagating trace context. This configuration ensures that spans have the correct parent-child relationship within a trace.

To ensure that all pending telemetry is flushed and that connections are closed gracefully, the setupOpenTelemetry function returns a function named shutdown, which performs those actions.

Configure structured logging

To include the trace information as part of the JSON-formatted logs written to standard output, configure the Go structured logging package, slog. The following code sample illustrates the setupLogging function, which is called from the main function:

func setupLogging() {
	// Use json as our base logging format.
	jsonHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: replacer})
	// Add span context attributes when Context is passed to logging calls.
	instrumentedHandler := handlerWithSpanContext(jsonHandler)
	// Set this handler as the global slog handler.
	slog.SetDefault(slog.New(instrumentedHandler))
}

The previous code calls the handlerWithSpanContext function, which extracts information from the Context instance and adds that information as attributes to a log. These attributes can then be used to correlate a log with a trace:

• logging.googleapis.com/trace: Resource name of the trace associated with the log entry.
• logging.googleapis.com/spanId: The span ID with the trace that is associated with the log entry.
• logging.googleapis.com/trace_sampled: The value of this field must be true or false.

For more information about these fields, see the LogEntry structure.

func handlerWithSpanContext(handler slog.Handler) *spanContextLogHandler {
	return &spanContextLogHandler{Handler: handler}
}

// spanContextLogHandler is an slog.Handler which adds attributes from the
// span context.
type spanContextLogHandler struct {
	slog.Handler
}

// Handle overrides slog.Handler's Handle method. This adds attributes from the
// span context to the slog.Record.
func (t *spanContextLogHandler) Handle(ctx context.Context, record slog.Record) error {
	// Get the SpanContext from the golang Context.
	if s := trace.SpanContextFromContext(ctx); s.IsValid() {
		// Add trace context attributes following Cloud Logging structured log format described
		// in https://cloud.google.com/logging/docs/structured-logging#special-payload-fields
		record.AddAttrs(
			slog.Any("logging.googleapis.com/trace", s.TraceID()),
		)
		record.AddAttrs(
			slog.Any("logging.googleapis.com/spanId", s.SpanID()),
		)
		record.AddAttrs(
			slog.Bool("logging.googleapis.com/trace_sampled", s.TraceFlags().IsSampled()),
		)
	}
	return t.Handler.Handle(ctx, record)
}

func replacer(groups []string, a slog.Attr) slog.Attr {
	// Rename attribute keys to match Cloud Logging structured log format
	switch a.Key {
	case slog.LevelKey:
		a.Key = "severity"
		// Map slog.Level string values to Cloud Logging LogSeverity
		// https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity
		if level := a.Value.Any().(slog.Level); level == slog.LevelWarn {
			a.Value = slog.StringValue("WARNING")
		}
	case slog.TimeKey:
		a.Key = "timestamp"
	case slog.MessageKey:
		a.Key = "message"
	}
	return a
}

Add instrumentation to the HTTP server

To add trace and metric instrumentation to the requests handled by the HTTP server, use OpenTelemetry. The following sample uses the otelhttp handler to propagate context, and for trace and metric instrumentation:

func runServer() error {
	handleHTTP("/single", handleSingle)
	handleHTTP("/multi", handleMulti)

	return http.ListenAndServe(":8080", nil)
}

// handleHTTP handles the http HandlerFunc on the specified route, and uses
// otelhttp for context propagation, trace instrumentation, and metric
// instrumentation.
func handleHTTP(route string, handleFn http.HandlerFunc) {
	instrumentedHandler := otelhttp.NewHandler(otelhttp.WithRouteTag(route, handleFn), route)

	http.Handle(route, instrumentedHandler)
}

In the previous code, the otelhttp handler uses the global TracerProvider, MeterProvider, and TextMapPropagator instances. The setupOpenTelemetry function configures these instances.

Link trace spans with logs and metrics

To link server and client spans, and to associate metrics and logs, pass the Go Context instance to the HTTP request and when you write logs. The following example illustrates a route handler that extracts the Go Context instance and the passes that instance to the logger and to the callSingle function, which makes an outgoing HTTP request:

func handleMulti(w http.ResponseWriter, r *http.Request) {
	subRequests := 3 + rand.Intn(4)
	// Write a structured log with the request context, which allows the log to
	// be linked with the trace for this request.
	slog.InfoContext(r.Context(), "handle /multi request", slog.Int("subRequests", subRequests))

	// Make 3-7 http requests to the /single endpoint.
	for i := 0; i < subRequests; i++ {
		if err := callSingle(r.Context()); err != nil {
			http.Error(w, err.Error(), http.StatusBadGateway)
			return
		}
	}

	fmt.Fprintln(w, "ok")
}

In the previous code, the function call r.Context() retrieves the Go Context from the HTTP request.

Add instrumentation to the HTTP client

To inject the trace context into outgoing HTTP requests and to add trace and metric instrumentation, call the otelhttp.Get function. In the following example, the callSingle function performs this action:

func callSingle(ctx context.Context) error {
	// otelhttp.Get makes an http GET request, just like net/http.Get.
	// In addition, it records a span, records metrics, and propagates context.
	res, err := otelhttp.Get(ctx, "http://localhost:8080/single")
	if err != nil {
		return err
	}

	return res.Body.Close()
}

In the previous code, the otelhttp handler uses the global TracerProvider, MeterProvider, and TextMapPropagator instances. The setupOpenTelemetry function configures these instances.

Write structured logs

To write structured logs that link to a trace, use Go's structured logging package, slog, and pass the Go Context instance to the logger. The Go Context instance is required when you want to link a log to a span. For example, the following statement shows how to call the InfoContext method for slog, and it illustrates how to add the field subRequests to the JSON instance:

slog.InfoContext(r.Context(), "handle /multi request", slog.Int("subRequests", subRequests))

Run a sample app configured to collect telemetry

The example app uses vendor-neutral formats, including JSON for logs and OTLP for metrics and traces. To route the telemetry to Google Cloud, this sample uses the OpenTelemetry Collector configured with Google exporters. The load generator in the app issues requests to the app's routes.

Download and deploy the app

Note: We recommend running the sample app by using Cloud Shell. However, if you want to run the sample app locally on Linux or Mac, then skip the first step in the following instructions.

To run the sample, do the following:

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

2. Clone the repository:

   git clone https://github.com/GoogleCloudPlatform/golang-samples
   

3. Go to the OpenTelemetry directory:

   cd golang-samples/opentelemetry/instrumentation
   

4. Build and run the sample:

   docker compose up --abort-on-container-exit
   

   If you aren't running on Cloud Shell, then run the application with the GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to a credentials file. Application Default Credentials provides a credentials file at $HOME/.config/gcloud/application_default_credentials.json.

   # Set environment variables
   export GOOGLE_CLOUD_PROJECT="PROJECT_ID"
   export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"
   export USERID="$(id -u)"
   
   # Run
   docker compose -f docker-compose.yaml -f docker-compose.creds.yaml up --abort-on-container-exit
   

View your metrics

The OpenTelemetry instrumentation in the sample app generates Prometheus metrics that you can view by using the Metrics Explorer:

• Prometheus/http_server_duration/histogram records the duration of server requests and stores the results in a histogram.

• Prometheus/http_server_request_content_length_total/counter records the request content length for the /multi and /single HTTP routes. The measurements for this metric are cumulative, which means that each value represents the total since collection of values began.

• Prometheus/http_server_response_content_length_total/counter records the response content length for the /multi and /single HTTP routes. The measurements for this metric are cumulative.

To view the metrics generated by the sample app, do the following:

1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Metrics explorer:

   Go to Metrics explorer

2. In the Metric element, expand the Select a metric menu, enter http_server in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In the Active resources menu, select Prometheus Target.
   2. In the Active metric categories menu, select Http.
   3. In the Active metrics menu, select a metric.
   4. Click Apply.
3. Configure how the data is viewed.
   

   When the measurements for a metric are cumulative, Metrics Explorer automatically normalizes the measured data by the alignment period, which which results in the chart displaying a rate. For more information, see Kinds, types, and conversions.

   When integer or double values are measured, such as with the two counter metrics, Metrics Explorer automatically sums all time series. To view the data for the /multi and /single HTTP routes, set the first menu of the Aggregation entry to None.

   For more information about configuring a chart, see Select metrics when using Metrics Explorer.

View your traces

To view your trace data, do the following:

1. In the navigation panel of the Google Cloud console, select Trace, and then select Trace explorer:

   Go to Trace explorer

2. In the scatter plot, select a trace with the URI of /multi.

3. In the Gantt chart on the Trace details panel, select the span labeled /multi.

   A panel opens that displays information about the HTTP request. These details include the method, status code, number of bytes, and the user agent of the caller.

4. To view the logs associated with this trace, select the Logs & Events tab.

   The tab shows individual logs. To view the details of the log entry, expand the log entry. You can also click View Logs and view the log by using the Logs Explorer.

For more information about using the Cloud Trace explorer, see Find and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can also view associated traces, when they exist.

1. In the navigation panel of the Google Cloud console, select Logging, and then select Logs Explorer:

   Go to Logs Explorer

2. Locate a log with the description of handle /multi request.

   To view the details of the log, expand the log entry. In the jsonPayload field, there is an entry labeled subRequests. This entry was added by a statement in the handleMulti function.

3. Click Traces on a log entry with the "handle /multi request" message, and then select View trace details.

   A Trace details panel opens and displays the selected trace.

For more information about using the Logs Explorer, see View logs by using the Logs Explorer.

What's next

• OpenTelemetry
• OTLP overview
• Structured logging
• Troubleshooting Managed Service for Prometheus
• Troubleshoot Cloud Trace