A simple demo project showcasing distributed tracing across microservices using Grafana Tempo, OpenTelemetry, and Go. This project implements a simple order processing system with two microservices that communicate with each other while maintaining full observability through distributed tracing also experimental for supporting context propagation with async processing with river queue.

This project consists of:

• Order Service (Port 8000): Handles order completion and triggers shipping
• Shipping Service (Port 9000): Processes shipping requests
• PostgreSQL Database: Stores data and job queue information
• Grafana OTEL-LGTM Stack: Provides observability (Logs, Grafana, Tempo, Metrics)

┌─────────────────┐    HTTP Request     ┌──────────────────┐
│   Order Service │ ───────────────────→│ Shipping Service │
|                 | Or Async River Queue|                  |
│    (Port 8000)  │ ───────────────────→|   (Port 9000)    │
└─────────────────┘                     └──────────────────┘
         │                                        │
         │               ┌─────────────────┐      │
         └──────────────→│   PostgreSQL    │ ←───-┘
                         │   Database      │
                         └─────────────────┘
                                 │
                     ┌─────────────────────┐
                     │ Grafana OTEL-LGTM   │
                     │ (Tracing & Metrics) │
                     └─────────────────────┘

• Distributed Tracing: Full request tracing across microservices
• Error Handling: Demonstrates error propagation and tracing
• Job Queue: Uses River for asynchronous job processing
• Observability: Complete monitoring with Grafana and Tempo
• Docker Orchestration: Easy setup with Docker Compose

• Go - Programming language
• OpenTelemetry - Distributed tracing instrumentation
• Grafana Tempo - Distributed tracing backend
• PostgreSQL - Database for persistence
• River - Job queue system
• Chi Router - HTTP routing
• Docker & Docker Compose - Container orchestration

grafana-tempo/
├── docker-compose.yml          # Docker orchestration
├── order-service/             # Order processing microservice
│   ├── main.go               # Service entry point
│   ├── handler/              # HTTP handlers
│   │   ├── order_handler.go  # Order completion endpoint
│   │   └── util.go           # Utility functions
│   ├── service/              # Business logic
│   │   └── order_service.go  # Order processing logic
│   ├── tracing/              # Tracing configuration
│   │   ├── otel.go           # OpenTelemetry setup
│   │   └── logger.go         # Logging setup
│   └── worker/               # Background job processing
│       └── shipping_worker.go
├── shipping-service/          # Shipping microservice
│   ├── main.go               # Service entry point
│   ├── handler/              # HTTP handlers
│   │   ├── shipping_handler.go # Shipping endpoint
│   │   └── util.go           # Utility functions
│   ├── service/              # Business logic
│   │   └── shipping_service.go # Shipping logic
│   ├── tracing/              # Tracing configuration
│   │   └── otel.go           # OpenTelemetry setup
│   └── worker/               # Background job processing
│       └── shipping_worker.go

Before running this project, make sure you have:

• Docker and Docker Compose installed
• Go 1.23+ installed (for development)
• Port availability: 3000, 4317, 4318, 5442, 8000, 9000

# Start PostgreSQL and Grafana OTEL-LGTM stack
docker-compose up -d

# Check if services are healthy
docker-compose ps

# Wait for PostgreSQL to be ready
docker-compose exec postgres pg_isready

Terminal 1 - Order Service:

cd order-service
go run main.go

Terminal 2 - Shipping Service:

cd shipping-service
go run main.go

You should see output like:

server started on :8000  # Order Service
server started on :9000  # Shipping Service

Once everything is running:

1. Grafana Dashboard: http://localhost:3000

   • Default credentials: admin/admin
   • Explore traces, metrics, and logs

2. Services:

   • Order Service: http://localhost:8000
   • Shipping Service: http://localhost:9000

# Complete an order successfully
curl -X POST "http://localhost:8000/complete-order?order_id=123"

Expected response:

{
  "code": "INT-OK-001",
  "message": "Order completed successfully",
  "trace_id": "abc123..."
}

# Trigger an error scenario
curl -X POST "http://localhost:8000/complete-order?order_id=error"

Expected response:

{
  "code": "INT-ERR-001",
  "message": "Internal Server Error",
  "trace_id": "def456..."
}

# Trigger shipping service error
curl -X POST "http://localhost:8000/complete-order?order_id=2"

1. Order Service receives the request
2. Trace starts with a unique trace ID
3. Order Service processes the order
4. HTTP call made to Shipping Service optional if you want to try river queue as async queue engine, just comment or uncomment the code i provide in order_service.go
5. Shipping Service processes the shipping
6. Background job queued for additional processing
7. Trace completes with full span hierarchy

• complete-order-handler - Order API endpoint
• complete-order-service - Order business logic
• shipping-order-handler - Shipping API endpoint
• shipping-order-service - Shipping business logic

The services use the following default configurations:

• Order Service Port: 8000
• Shipping Service Port: 9000
• PostgreSQL: localhost:5442
• OTEL Collector: localhost:4317

postgres://postgres@localhost:5442/tracing_db?sslmode=disable

1. Port conflicts: Ensure ports 3000, 4317, 4318, 5442, 8000, 9000 are available
2. Database connection: Wait for PostgreSQL to be fully ready
3. Service dependencies: Start infrastructure first, then services

docker-compose logs collector
docker-compose logs postgres

This project demonstrates:

1. Distributed Tracing Concepts:

   • How traces span across multiple services
   • Context propagation between services
   • Error handling and trace correlation

2. OpenTelemetry Integration:

   • Service instrumentation
   • Custom span creation
   • Trace context extraction

3. Microservices Communication:

   • HTTP service-to-service calls or
   • Using asynchronous job processing river queue
   • Database integration

4. Observability Best Practices:

   • Structured logging
   • Metric collection
   • Distributed tracing

• OpenTelemetry Go Documentation
• Grafana Tempo Documentation
• River Job Queue Documentation
• Distributed Tracing Best Practices