Home page
Back to articles
26 min readMehdi Hadeli

End-to-End Observability in .NET with Aspire, OpenTelemetry, and the LGTM Stack

dotnetaspireopentelemetryobservability

On this page

Table of contents

Introduction

Observability is not a feature you bolt on after launch — it is a first-class concern that shapes how you debug, monitor, and operate your system. In the .NET ecosystem, the combination of .NET Aspire, OpenTelemetry (OTel), and the OpenTelemetry Collector gives you a vendor-neutral pipeline that exports telemetry to any backend you choose.

In this article, we take the AspireShop sample — a microservices e-commerce app with a catalog service, basket service, and a Blazor frontend — and instrument it end-to-end. We stand up a full observability stack using Docker Compose, configure the OTel Collector to fan out traces, metrics, and logs to Prometheus, Tempo, Loki, and Grafana (the LGTM stack). As a bonus, we keep an ELK stack (Elasticsearch + Kibana) running alongside as an optional alternative, and we discuss when you would pick one over the other.

By the end you will understand:

  • How .NET Aspire projects emit OTel telemetry out of the box.
  • How to configure the OTel Collector to receive, process, and export signals.
  • How to provision Prometheus, Tempo, Loki, and Grafana with Docker Compose.
  • How to explore traces, metrics, and logs in Grafana.
  • Where the ELK stack fits and why you might choose it instead.

Architecture Overview

The telemetry pipeline follows a clean fan-out pattern: every .NET service pushes OTLP to the Collector, and the Collector routes each signal type to the appropriate backend.

Rendering diagram...

Every signal type is handled by the backend designed for it. Traces go to Tempo, logs to Loki, metrics to Prometheus — and all three are queried through a single pane of glass in Grafana. Tempo and Loki store their data in MinIO, an S3-compatible object store, so trace blocks and log chunks survive container restarts. The Elasticsearch exporter on the Collector also pushes traces and logs into Elasticsearch, where Kibana provides an alternative full-featured UI.

Prerequisites

Part 1: Instrumenting the .NET Application

The Service Defaults Library

Aspire projects share instrumentation through a Service Defaults project. In the AspireShop solution, AspireShop.ServiceDefaults defines the AddServiceDefaults() extension that every service calls in its Program.cs:

// AspireShop.CatalogService/Program.cs
var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();  // <-- wires up OTel, health checks, service discovery
builder.AddNpgsqlDbContext<CatalogDbContext>("catalogdb");
// ...

The AddServiceDefaults method does three things: it configures OpenTelemetry, adds health checks, and enables service discovery with resilience. The OpenTelemetry setup lives in ConfigureOpenTelemetry:

// AspireShop.ServiceDefaults/Extensions.cs (excerpt)
public static IHostApplicationBuilder ConfigureOpenTelemetry(
    this IHostApplicationBuilder builder)
{
    builder.Logging.AddOpenTelemetry(logging =>
    {
        logging.IncludeFormattedMessage = true;
        logging.IncludeScopes = true;
    });

    builder.Services.AddOpenTelemetry()
        .WithMetrics(metrics =>
        {
            metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation();
        })
        .WithTracing(tracing =>
        {
            tracing
                .AddSource(builder.Environment.ApplicationName)
                .AddAspNetCoreInstrumentation(tracing =>
                    tracing.Filter = httpContext =>
                        !(httpContext.Request.Path.StartsWithSegments("/health")
                          || httpContext.Request.Path.StartsWithSegments("/alive")))
                .AddGrpcClientInstrumentation()
                .AddHttpClientInstrumentation();
        });

    builder.AddOpenTelemetryExporters();
    return builder;
}

The instrumentation covers the key signals:

  • Metrics: ASP.NET Core request duration, HTTP client calls, and .NET runtime metrics (GC, heap, thread pool).
  • Tracing: Inbound HTTP requests, outbound HTTP calls, and gRPC client calls. Health-check endpoints are filtered out to reduce noise.
  • Logging: ILogger entries are bridged into the OTel pipeline, preserving formatted messages and scopes.

Exporting to the Collector

The OTLP exporter is wired in a separate private method. It checks for the OTEL_EXPORTER_OTLP_ENDPOINT environment variable — if set, it calls UseOtlpExporter():

// AspireShop.ServiceDefaults/Extensions.cs (excerpt)
private static IHostApplicationBuilder AddOpenTelemetryExporters(
    this IHostApplicationBuilder builder)
{
    var useOtlpExporter = !string.IsNullOrWhiteSpace(
        builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

    if (useOtlpExporter)
    {
        builder.Services.AddOpenTelemetry().UseOtlpExporter();
    }

    return builder;
}

The UseOtlpExporter() call reads the standard OTel environment variables — OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL — so you never hardcode a collector address.

Injecting the Endpoint from the AppHost

The .NET Aspire AppHost is the orchestration layer. It reads the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable (with a localhost fallback) and injects it into every service project:

// AspireShop.AppHost/AppHost.cs (excerpt)
var otlpEndpoint = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT") ?? "http://localhost:4317";
var otlpProtocol = Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_PROTOCOL") ?? "grpc";

var catalogService = builder
    .AddProject<Projects.AspireShop_CatalogService>("catalogservice")
    .WithEnvironment("OTEL_EXPORTER_OTLP_ENDPOINT", otlpEndpoint)
    .WithEnvironment("OTEL_EXPORTER_OTLP_PROTOCOL", otlpProtocol)
    .WithReference(catalogDb)
    .WaitFor(catalogDbManager);

// Same pattern for basketService and frontend...

No custom configuration keys — just the standard OTel environment variables. The AppHost reads them, and each service receives them through .WithEnvironment().

Fallback for Standalone Runs

When you run a service outside the AppHost (for example, during development with dotnet run), the OTLP variables come from launchSettings.json:

// AspireShop.CatalogService/Properties/launchSettings.json (excerpt)
"environmentVariables": {
  "ASPNETCORE_ENVIRONMENT": "Development",
  "ASPIRE_ALLOW_UNSECURED_TRANSPORT": "true",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4317"
}

The ASPIRE_ALLOW_UNSECURED_TRANSPORT flag is important: in local development the Collector runs without TLS, and Aspire needs explicit permission to skip it.

Running Services Standalone

There are two ways to start the services. When you run through the AppHost (dotnet run --project AspireShop.AppHost), the host injects OTEL_EXPORTER_OTLP_ENDPOINT into every project. But you can also run individual services directly — for example, when debugging the BasketService in isolation:

dotnet run --project AspireShop.BasketService --launch-profile https

This works because every service's launchSettings.json has the same OTEL_EXPORTER_OTLP_ENDPOINT variable. Even without the AppHost, the service resolves the endpoint from launchSettings.json and starts pushing telemetry to the Collector on localhost:4317. The same is true for CatalogService, CatalogDbManager, and Frontend — all four projects share the identical OTLP configuration in their launch profiles.

This dual-path design means you get consistent observability whether you run the full distributed app via the AppHost or spin up a single service for focused debugging.

The Aspire Dashboard: A Second Pane of Glass

While this article focuses on the external LGTM stack, the .NET Aspire Dashboard is also receiving telemetry. Look at the Collector's otlp/aspire exporter in the pipeline config:

otlp/aspire:
  endpoint: '${env:ASPIRE_OTLP_ENDPOINT}'
  headers:
    x-otlp-api-key: '${env:ASPIRE_API_KEY}'
  tls:
    insecure: ${env:ASPIRE_INSECURE}
    insecure_skip_verify: true

This exporter fans out all three signals (traces, metrics, and logs) to the Aspire Dashboard — a built-in developer UI that starts alongside the AppHost. The Collector acts as a telemetry fork: one copy goes to the LGTM stack (Prometheus/Tempo/Loki), another copy goes to the Aspire Dashboard for quick local inspection during development.

Rendering diagram...

The Aspire Dashboard gives you structured logs, trace waterfalls, and metrics charts without opening Grafana — useful during active development. The external Grafana stack is for deeper analysis, historical queries, alerting, and team-wide dashboards. Both consume the same telemetry; you choose which pane of glass fits the moment.

Two Paths to the Dashboard: Direct vs Collector-Routed

There are two ways to get telemetry into the Aspire Dashboard, and understanding the difference matters.

Path A: Direct AppHost → Dashboard (Default Aspire Pattern)

The AppHost's own launchSettings.json configures how the AppHost communicates with the dashboard it spawns. The official Aspire sample uses these variables:

// AspireShop.AppHost/Properties/launchSettings.json
"environmentVariables": {
  "ASPNETCORE_ENVIRONMENT": "Development",
  "DOTNET_ENVIRONMENT": "Development",
  "ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL": "https://localhost:16224",
  "ASPIRE_RESOURCE_SERVICE_ENDPOINT_URL": "https://localhost:22123"
}
VariableDefaultPurpose
ASPIRE_DASHBOARD_OTLP_ENDPOINT_URLhttp://localhost:18889OTLP/gRPC endpoint where the dashboard listens for telemetry
ASPIRE_RESOURCE_SERVICE_ENDPOINT_URLnullgRPC endpoint the dashboard connects to for the resource list and console logs

In the default Aspire pattern, the AppHost injects OTEL_EXPORTER_OTLP_ENDPOINT into child services pointing to this same dashboard OTLP endpoint. The result: every service sends telemetry directly to the Aspire Dashboard — no Collector in the path.

Rendering diagram...

This is perfect for local development when all you need is the built-in dashboard. Zero infrastructure beyond the AppHost itself.

Path B: Collector-Routed (Our Setup)

In this article we reroute telemetry through the OTel Collector. Our AppHost's launchSettings.json sets both variables side by side:

// AspireShop.AppHost/Properties/launchSettings.json
"environmentVariables": {
  "ASPIRE_ALLOW_UNSECURED_TRANSPORT": "true",
  "ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL": "https://localhost:16224",
  "ASPIRE_RESOURCE_SERVICE_ENDPOINT_URL": "https://localhost:22123",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4317"
}

ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL tells the AppHost where the dashboard listens. OTEL_EXPORTER_OTLP_ENDPOINT, set to localhost:4317, tells the AppHost (and every child service) to send telemetry to the Collector instead of directly to the dashboard. The Collector then fans out to both the dashboard and the LGTM stack:

Rendering diagram...

The AppHost's launchSettings.json still has ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL — that configures where the dashboard listens. But the AppHost code overrides OTEL_EXPORTER_OTLP_ENDPOINT to point to the Collector, not the dashboard.

When to Use Each

ScenarioUse
Local dev, only need the built-in dashboardPath A — default Aspire. No extra infrastructure.
Need external backends (Grafana, Prometheus, Tempo)Path B — route through Collector. One telemetry stream, multiple destinations.
Production or staging with centralized observabilityPath B — Collector gives you batching, filtering, and multi-backend export.
Debugging a single service in isolationEither — launchSettings fallback works for both paths.

The two paths are not mutually exclusive. The AppHost always sends its own infrastructure telemetry (resource lifecycle events, startup logs) to the dashboard via ASPIRE_DASHBOARD_OTLP_ENDPOINT_URL. Path B only changes where service-level telemetry goes first — it still ends up at the dashboard, just through the Collector.

No separate dashboard container. The Aspire Dashboard is not defined in docker-compose.infrastructure.yaml. The AppHost launches it as a child process automatically when you run dotnet run. The Collector (running in Docker) reaches it at host.docker.internal:16224 via the ASPIRE_OTLP_ENDPOINT env var. There is nothing extra to containerize — it just works.

At this point the .NET side is done. Every service emits OTLP over gRPC to localhost:4317, and the Collector fans it to both the Aspire Dashboard and the external observability stack. Next we build the infrastructure that receives it.

Part 2: The Docker Compose Infrastructure Stack

The full observability stack is defined in a single Docker Compose file: deployments/docker-compose/docker-compose.infrastructure.yaml. Here is the service roster:

ServiceImagePort(s)Role
postgrespostgres:16.4-alpine5432Catalog database
basketcacheredis/redis-stack:7.4.0-v06379, 8001Basket cache
miniominio/minio:RELEASE.2025-09-07T16-13-09Z9000, 9001S3 storage for Loki & Tempo
minio-bucket-creatorminio/mc:latest—Creates loki & tempo buckets
otel-collectorotel/opentelemetry-collector-contrib:0.110.04317, 4318, 8888Telemetry pipeline
prometheusprom/prometheus:v3.3.19090Metrics store
tempografana/tempo:2.6.03200, 4317*Trace store
lokigrafana/loki:3.2.03100Log store
grafanagrafana/grafana:11.2.33000Dashboards
node-exporterprom/node-exporter:v1.8.29100Host metrics
elasticsearchelasticsearch:8.15.29200, 9300Optional log/trace store
kibanakibana:8.15.25601Optional ELK UI

Tempo's internal OTLP port (4317) is used only by the Collector — it does not conflict with the Collector's own 4317 because they are on separate containers on the same Docker network.

OpenTelemetry Collector Configuration

The Collector is the heart of the pipeline. Its configuration (otel-collector-config.yaml) defines receivers, processors, and exporters:

# deployments/configs/otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  transform/environment:
    error_mode: ignore
    trace_statements:
      - context: resource
        statements:
          - set(attributes["environment"], "development")
    log_statements:
      - context: resource
        statements:
          - set(attributes["environment"], "development")
    metric_statements:
      - context: resource
        statements:
          - set(attributes["environment"], "development")

exporters:
  otlphttp/prometheus:
    endpoint: '${env:PROMETHEUS_ENDPOINT}'
    tls:
      insecure: true

  otlphttp/loki:
    endpoint: '${env:LOKI_ENDPOINT}'
    tls:
      insecure: true

  otlp/tempo:
    endpoint: '${env:TEMPO_URL}'
    tls:
      insecure: true

  otlp/aspire:
    endpoint: '${env:ASPIRE_OTLP_ENDPOINT}'
    headers:
      x-otlp-api-key: '${env:ASPIRE_API_KEY}'
    tls:
      insecure: ${env:ASPIRE_INSECURE}
      insecure_skip_verify: true

  elasticsearch:
    endpoints:
      - 'http://elasticsearch:9200'
    mapping:
      mode: otel
    sending_queue:
      enabled: true
    retry:
      enabled: true

  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, transform/environment]
      exporters: [otlp/tempo, otlp/aspire, elasticsearch, debug]
    metrics:
      receivers: [otlp]
      processors: [batch, transform/environment]
      exporters: [otlphttp/prometheus, otlp/aspire, debug]
    logs:
      receivers: [otlp]
      processors: [batch, transform/environment]
      exporters: [otlphttp/loki, otlp/aspire, elasticsearch, debug]

Key design decisions here:

  • otlphttp/prometheus pushes metrics from the Collector to Prometheus's OTLP HTTP endpoint. Prometheus v3.3.1 has a built-in OTLP receiver, enabled via --web.enable-otlp-receiver in the Compose command. Metrics arrive at http://prometheus:9090/api/v1/otlp as OTLP, and Prometheus stores them natively — no remote-write bridge needed.
  • otlphttp/loki sends logs as OTLP HTTP. Loki's allow_structured_metadata: true preserves OTel resource attributes.
  • otlp/tempo sends traces over gRPC, which Tempo receives on its own OTLP endpoint.
  • elasticsearch uses the native Elasticsearch exporter with mapping: mode: otel, which maps OTel semantic conventions to Elastic's schema.
  • The transform/environment processor tags all signals with environment=development, making it easy to filter by environment in dashboards.

How Metrics Flow: Collector → Prometheus via OTLP

This is the trickiest piece of the pipeline, so let's trace it step by step:

Rendering diagram...
  1. The .NET service emits metrics over OTLP gRPC to the Collector on port 4317.
  2. The Collector's otlphttp/prometheus exporter transforms those metrics back to OTLP (HTTP this time) and POSTs them to http://prometheus:9090/api/v1/otlp. The URL comes from the PROMETHEUS_ENDPOINT env var set in Docker Compose.
  3. Prometheus receives the OTLP payload through its native OTLP HTTP receiver — enabled by the --web.enable-otlp-receiver flag.
  4. The otlp: block in prometheus.yaml promotes resource attributes (service.name, service.instance.id, etc.) from OTLP metadata into Prometheus labels, so every metric is queryable by service.

Two sides must match:

  • Collector side: otlphttp/prometheus exporter with endpoint: "${env:PROMETHEUS_ENDPOINT}" → resolved to http://prometheus:9090/api/v1/otlp
  • Prometheus side: OTLP receiver enabled via CLI flag + otlp: config block for attribute promotion

The Collector's environment variables are injected by the Compose file:

# docker-compose.infrastructure.yaml (excerpt)
otel-collector:
  environment:
    - ASPIRE_OTLP_ENDPOINT=${ASPIRE_OTLP_ENDPOINT:-host.docker.internal:16224}
    - ASPIRE_API_KEY=${ASPIRE_API_KEY:-}
    - ASPIRE_INSECURE=true
    - PROMETHEUS_ENDPOINT=http://prometheus:9090/api/v1/otlp
    - LOKI_ENDPOINT=http://loki:3100/otlp
    - TEMPO_URL=tempo:4317

MinIO: S3 Object Storage for Loki & Tempo

By default, Loki and Tempo use local filesystem storage — simple, but not durable. Recreate the container and you lose your logs and traces. MinIO is an S3-compatible object store that gives you persistent, scalable storage without changing a single line of application code.

The Docker Compose file defines two cooperating services: the MinIO server and a bucket creator that runs once to initialize the loki and tempo buckets:

# docker-compose.infrastructure.yaml (minio services excerpt)
minio:
  image: minio/minio:RELEASE.2025-09-07T16-13-09Z
  restart: unless-stopped
  command:
    - server
    - /data
    - --console-address
    - :9001
  environment:
    MINIO_ROOT_USER: ${MINIO_USER:-minioadmin}
    MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD:-minioadmin}
    MINIO_PROMETHEUS_AUTH_TYPE: public
  ports:
    # MinIO S3 API
    - '9000:9000'
    # MinIO Console UI
    - '9001:9001'
  volumes:
    - minio_data:/data

minio-bucket-creator:
  image: minio/mc:latest
  restart: on-failure
  entrypoint:
    - /bin/sh
  command:
    - -c
    - |
      /usr/bin/mc alias set myminio http://minio:9000 $${MINIO_USER} $${MINIO_PASSWORD}
      /usr/bin/mc mb myminio/loki --ignore-existing
      /usr/bin/mc anonymous set public myminio/loki
      /usr/bin/mc mb myminio/tempo --ignore-existing
      /usr/bin/mc anonymous set public myminio/tempo
      exit 0
  environment:
    MINIO_USER: ${MINIO_USER:-minioadmin}
    MINIO_PASSWORD: ${MINIO_PASSWORD:-minioadmin}
  depends_on:
    minio:
      condition: service_started

The minio-bucket-creator runs mc (the MinIO CLI) to create two buckets — loki for log chunks and tempo for trace blocks — then exits immediately. The --ignore-existing flag makes the command idempotent: safe to restart.

Tempo and Loki are configured to wait for this setup before starting:

# docker-compose.infrastructure.yaml (tempo + loki excerpt)
tempo:
  image: grafana/tempo:2.6.0
  restart: unless-stopped
  command: ['-config.file=/etc/tempo.yaml', '-config.expand-env=true']
  environment:
    - MINIO_USER=${MINIO_USER:-minioadmin}
    - MINIO_PASSWORD=${MINIO_PASSWORD:-minioadmin}
  depends_on:
    minio-bucket-creator:
      condition: service_completed_successfully

loki:
  image: grafana/loki:3.2.0
  restart: unless-stopped
  command: -config.file=/etc/loki/local-config.yaml -config.expand-env=true
  environment:
    - MINIO_USER=${MINIO_USER:-minioadmin}
    - MINIO_PASSWORD=${MINIO_PASSWORD:-minioadmin}
  depends_on:
    minio-bucket-creator:
      condition: service_completed_successfully

The MINIO_USER and MINIO_PASSWORD env vars are passed through so Tempo and Loki can authenticate to the S3 API. The -config.expand-env=true flag is critical: without it, Tempo and Loki see the literal string ${MINIO_USER} instead of the actual value, and both crash with InvalidAccessKeyId. Loki supports this flag from v2.1 onward; Tempo follows the same convention.

Prometheus Configuration

The Prometheus OTLP receiver is enabled via the Docker Compose command, and the prometheus.yaml config file sets up resource attribute promotion alongside scrape targets for infrastructure self-monitoring:

# docker-compose.infrastructure.yaml (prometheus service excerpt)
prometheus:
  image: prom/prometheus:v3.3.1
  restart: unless-stopped
  ports:
    - '9090:9090'
  volumes:
    - ./../configs/prometheus.yaml:/etc/prometheus/prometheus.yml
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--web.enable-otlp-receiver' # ← enables OTLP HTTP ingest
    - '--enable-feature=native-histograms'
    - '--log.level=info'

The --web.enable-otlp-receiver flag is the critical piece — it turns on the /api/v1/otlp HTTP endpoint that the Collector's otlphttp/prometheus exporter targets. With this flag, Prometheus accepts OTLP metrics directly, no remote-write translation needed.

# deployments/configs/prometheus.yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    collector: otel-collector

# App metrics are pushed from the OpenTelemetry Collector to
# Prometheus through the OTLP HTTP receiver at /api/v1/otlp.
# The receiver is enabled via --web.enable-otlp-receiver.
otlp:
  promote_resource_attributes:
    - service.instance.id
    - service.name
    - service.namespace
    - service.version

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8888']
    metric_relabel_configs:
      - target_label: integration
        replacement: collector-self-metrics

  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']
    metric_relabel_configs:
      - target_label: integration
        replacement: prometheus-self-metrics

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    metric_relabel_configs:
      - target_label: integration
        replacement: node-host-metrics

  - job_name: 'tempo'
    static_configs:
      - targets: ['tempo:3200']
    metric_relabel_configs:
      - target_label: integration
        replacement: tempo-self-metrics

The otlp: block promotes key resource attributes from OTLP metadata into Prometheus labels. Every metric arriving via OTLP automatically gets service_name, service_instance_id, and service_version labels — no relabeling needed.

The scrape targets provide self-observability: you can monitor the Collector's own throughput, Prometheus internals, and the host machine's CPU/memory/disk through the node-exporter.

Tempo Configuration (S3-Backed)

Tempo stores trace blocks in MinIO while keeping the write-ahead log on local disk for performance. It receives OTLP traces from the Collector over gRPC and derives RED metrics from spans:

# deployments/configs/tempo.yaml
stream_over_http_enabled: true
server:
  http_listen_port: 3200
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: '0.0.0.0:4317'
        http:
          endpoint: '0.0.0.0:4318'

ingester:
  max_block_duration: 5m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

compactor:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

storage:
  trace:
    backend: s3
    s3:
      endpoint: minio:9000
      bucket: tempo
      access_key: ${MINIO_USER}
      secret_key: ${MINIO_PASSWORD}
      insecure: true
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics, local-blocks]
      generate_native_histograms: both

The storage.trace.backend: s3 switch is the only change from local storage. The WAL (/var/tempo/wal) and temporary blocks (/var/tempo/blocks) stay on local disk — Tempo writes completed trace blocks to the tempo bucket on MinIO. The insecure: true flag is needed because MinIO runs without TLS in the local Docker network.

The metrics generator derives RED metrics (Rate, Error, Duration) and service graphs from spans with zero code changes. These are remote-written back to Prometheus, so Grafana shows both application metrics and span-derived metrics in one place. The generate_native_histograms: both flag enables high-precision histogram buckets for latency distributions.

The metrics generator derives RED metrics (Rate, Error, Duration) and service graphs from spans with zero code changes. These are remote-written back to Prometheus, so Grafana shows both application metrics and span-derived metrics in one place. The generate_native_histograms: both flag enables high-precision histogram buckets for latency distributions.

Loki Configuration (S3-Backed)

Loki stores log chunks and the TSDB index in MinIO. The key settings are the storage_config.aws block pointing at MinIO and allow_structured_metadata: true for OTLP attribute preservation:

# deployments/configs/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

ingester:
  chunk_idle_period: 10s
  chunk_retain_period: 20s
  max_chunk_age: 30s
  wal:
    enabled: true
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

compactor:
  working_directory: /loki/compactor
  delete_request_store: s3
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

schema_config:
  configs:
    - from: 2025-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    endpoint: http://minio:9000
    access_key_id: ${MINIO_USER}
    secret_access_key: ${MINIO_PASSWORD}
    bucketnames: loki
    s3forcepathstyle: true
    insecure: true
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h

limits_config:
  allow_structured_metadata: true

The storage_config.aws block tells Loki to treat MinIO as an S3-compatible endpoint. s3forcepathstyle: true is required because MinIO uses path-style addressing rather than virtual-hosted-style. The ingester section controls how long Loki buffers logs in memory before flushing chunks to S3 — these short durations are fine for development; increase them in production to reduce S3 PUT operations.

The allow_structured_metadata: true flag is essential — without it, Loki strips OTel resource attributes from incoming OTLP log payloads. With it enabled, Grafana can filter logs by service_name, severity_text, and any other attribute pushed through the Collector's otlphttp/loki exporter.

Grafana Datasource Provisioning

Grafana datasources are provisioned declaratively so you never click through the UI:

# deployments/configs/grafana/provisioning/datasources/datasource.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    uid: prometheus-uid

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    editable: false
    uid: tempo-uid
    jsonData:
      httpMethod: GET
      tracesToLogsV2:
        datasourceUid: loki-uid
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'
        tags:
          - key: service_name
            value: service_name
      tracesToMetrics:
        datasourceUid: prometheus-uid
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'
      serviceMap:
        datasourceUid: prometheus-uid
      streamingEnabled:
        search: true

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false
    uid: loki-uid
    jsonData:
      derivedFields:
        - datasourceUid: tempo-uid
          matcherRegex: "^.*?traceI[d|D]=(\\w+).*$"
          name: traceId
          url: '$${__value.raw}'

  - name: Elasticsearch
    type: elasticsearch
    access: proxy
    url: http://elasticsearch:9200
    editable: true
    uid: elasticsearch-uid
    jsonData:
      index: 'logs-generic-default'
      esVersion: '8.15.2'
      timeField: '@timestamp'
      maxConcurrentShardRequests: 10
      logMessageField: 'body_text'
      logLevelField: 'severity_text'

The Tempo-to-Loki link (tracesToLogsV2) enables the classic "click a span → see related logs" workflow. The Loki-to-Tempo derived field extracts traceId from log lines and turns it into a clickable link that opens the full trace waterfall in Tempo.

The Elasticsearch datasource is an optional bridge: it brings logs and metrics from Elasticsearch into Grafana, so you can query both backends without switching tools. For example, you can run a Lucene full-text search against Elasticsearch logs in the same Explore view where you write PromQL and LogQL. Trace waterfalls and service graphs from Elasticsearch are not rendered in Grafana — for those you use Kibana, where Elastic APM provides native trace views, span waterfalls, and dependency maps.

Note on the ELK stack: Elasticsearch and Kibana run side-by-side with the LGTM stack in this article as an optional alternative. In production you would typically pick one observability path — either LGTM (Grafana-native: Prometheus + Tempo + Loki) or ELK (Elastic-native: Elasticsearch + Kibana). Kibana is the Grafana equivalent for the ELK world: it unifies logs (Discover), traces (APM), and metrics (Lens and Dashboards) under one UI — the same role Grafana fills for the LGTM stack. The choice comes down to operational preference: LGTM is lighter and more cloud-native; ELK gives you powerful full-text search, Elastic APM, and machine learning features.

loki tempo elasticsearch kibana

Part 3: Running the Stack

Step 1: Start the Infrastructure

cd deployments/docker-compose
docker compose -f docker-compose.infrastructure.yaml up -d

Wait for all containers to be healthy (docker ps should show all services running).

Step 2: Run the AppHost

cd samples/aspire-shop
dotnet run --project AspireShop.AppHost

The AppHost starts postgres, redis, catalog-db-manager, catalog-service, basket-service, and the frontend. Each service auto-discovers the OTLP endpoint and starts exporting.

Step 3: Generate Traffic

Open the frontend at https://localhost:7241 (or the HTTP endpoint shown in the Aspire dashboard). Browse products, add items to the basket. Every click generates spans, metrics, and logs.

Step 4: Explore in Grafana

Open http://localhost:3000 (login: admin / admin).

Metrics (Prometheus) — Use the Explore view with the Prometheus datasource. Try queries like:

rate(http_server_request_duration_seconds_count[1m])

Traces (Tempo) — Switch to the Tempo datasource. Use TraceQL:

{ .service.name = "catalogservice" }

Click any trace to see the waterfall, then click a span and choose "View related logs" to jump to Loki.

Logs (Loki) — Switch to the Loki datasource. Use LogQL:

{service_name="catalogservice"} |= ""

Log lines with trace IDs will have clickable links back to Tempo.

Step 5: Explore the Optional ELK Stack

Open http://localhost:5601 for Kibana. Think of Kibana as the Grafana equivalent for the ELK stack — one UI that covers all three signals:

SignalKibana ToolGrafana Equivalent
LogsDiscoverLoki Explore view
TracesAPMTempo trace waterfall
MetricsLens / DashboardsPrometheus + Grafana panels

Start with Analytics > Discover and select the logs-generic-default data view. You will see the same logs flowing from the Collector, with Lucene full-text search and field-level filtering. For traces, head to APM > Services — the Collector pushes trace data to Elasticsearch with mapping: mode: otel, and Elastic APM surfaces service maps, latency distributions, and span waterfalls. For metrics, Dashboards and Lens let you build visualizations from Elasticsearch aggregations without learning a separate query language.

Back in Grafana, the Elasticsearch datasource (elasticsearch-uid) gives you a bridge: query Elasticsearch logs and metrics without leaving Grafana's interface. Use it when you want Elastic's full-text search power inside your existing Grafana dashboards.

One stack or two? Running both LGTM and ELK side by side is fine for learning, but in production pick one. Both stacks give you logs, traces, and metrics. The difference is where the data lives (object storage vs. Elasticsearch) and which UI you prefer (Grafana vs. Kibana).

Part 4: Understanding the Signal Flow

Let's trace a single user request through the entire pipeline:

Rendering diagram...
  1. A request hits the Frontend, which calls CatalogService over gRPC.
  2. Both services emit spans (with trace context propagation), metrics (request duration, rate), and logs (ILogger entries).
  3. The Collector receives everything on :4317, batches it, tags it with environment=development, and fans it out:
    • Traces → Tempo (gRPC)
    • Metrics → Prometheus (OTLP HTTP)
    • Logs → Loki (OTLP HTTP) + Elasticsearch (native export)
  4. Grafana queries Tempo, Loki, Prometheus, and Elasticsearch through their respective datasources.

The trace context flows automatically because AddAspNetCoreInstrumentation and AddHttpClientInstrumentation handle W3C trace context propagation. gRPC calls propagate trace context via gRPC metadata.

Part 5: Choosing Between LGTM and ELK

Both stacks offer complete observability. Which one should you choose?

ConcernLGTM (Prometheus + Tempo + Loki)ELK (Elasticsearch + Kibana)
MetricsPrometheus — de-facto standard for cloud-native metrics, PromQL, AlertmanagerElasticsearch aggregations + Kibana Lens
TracesTempo — TraceQL, service graphs, no sampling required at ingestElastic APM — tight integration with Elasticsearch queries
LogsLoki — low-cost, label-based, integrates with Promtail/OTelElasticsearch — powerful full-text search, Lucene query syntax
StorageObject storage (S3/GCS/Azure) — cheap, scales horizontallyLocal or cloud disks — requires more tuning for scale
Learning curveModerate — three different query languages (PromQL, TraceQL, LogQL)Lower — one query language (Lucene/KQL), one store
Best forKubernetes-native, microservices, teams already using PrometheusFull-text log analysis, security/APM use cases, single-vendor preference

The AspireShop sample runs both side by side for demonstration. In a real project, pick one and keep your operational surface area smaller.

Conclusion

You have wired a .NET microservices application for full observability using open standards and open-source tooling. The recipe:

  1. .NET Aspire + Service Defaults gives you OTel instrumentation with zero-config defaults.
  2. AppHost environment injection keeps the OTLP endpoint configurable per environment.
  3. OTel Collector acts as a universal telemetry router — receive once, export everywhere.
  4. LGTM stack (Prometheus, Tempo, Loki, Grafana) provides a modern, scalable observability backend.
  5. ELK stack (Elasticsearch, Kibana) runs as an optional sidecar for teams that prefer Elastic-native tooling.
  6. Grafana ties everything together with cross-datasource linking (trace → logs, logs → trace).

The complete source code is available at github.com/mehdihadeli/blog-samples under samples/aspire-shop/. All Docker Compose files, Collector configs, and Grafana provisioning files are in the deployments/ folder — ready to run with a single docker compose up.