Centralized Logging for Docker: ELK, Loki and Fluentd Stacks

Running docker logs mycontainer works for a single container on a single host. It does not work when you have 50 containers across 5 servers, one of which logged an error 3 hours ago that caused a cascade of failures you are now debugging at 2 AM. Centralized logging collects logs from every container, enriches them with metadata (container name, host, timestamp), and provides a searchable interface for querying across your entire infrastructure.

This guide covers three centralized logging stacks for Docker: the ELK Stack (Elasticsearch, Logstash, Kibana) for teams that need full-text search and analytics, Grafana Loki with Promtail for teams that want lightweight log aggregation alongside Grafana metrics, and Fluentd/Fluent Bit for flexible log routing and transformation.

Docker Logging Drivers

Before setting up a centralized stack, understand how Docker handles logs. Every container has a logging driver that determines where stdout and stderr output goes:

# Configure logging driver per container in Compose
services:
  app:
    image: myapp:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"     # Rotate at 50MB
        max-file: "5"       # Keep 5 rotated files
        tag: "{{.Name}}"    # Add container name as tag

# Or set the default driver for all containers in /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

ELK Stack (Elasticsearch, Logstash, Kibana)

The ELK stack is the most established centralized logging solution. It provides full-text search, log analytics, and powerful visualization capabilities:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "127.0.0.1:9200:9200"
    deploy:
      resources:
        limits:
          memory: 2G
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5

  logstash:
    image: docker.elastic.co/logstash/logstash:8.12.0
    container_name: logstash
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "127.0.0.1:5044:5044"    # Beats input
      - "127.0.0.1:5000:5000"    # TCP input
      - "127.0.0.1:12201:12201/udp"  # GELF input
    environment:
      - "LS_JAVA_OPTS=-Xms512m -Xmx512m"
    depends_on:
      elasticsearch:
        condition: service_healthy

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    container_name: kibana
    ports:
      - "127.0.0.1:5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      elasticsearch:
        condition: service_healthy

volumes:
  es-data:

Logstash Pipeline Configuration

# logstash/pipeline/docker-logs.conf
input {
  # Receive logs from Docker GELF driver
  gelf {
    port => 12201
    type => "docker"
  }

  # Receive logs from Filebeat/Fluent Bit
  beats {
    port => 5044
  }

  # Receive logs via TCP (for fluentd driver)
  tcp {
    port => 5000
    codec => json
  }
}

filter {
  # Parse JSON log messages
  if [message] =~ /^\{/ {
    json {
      source => "message"
      target => "parsed"
    }
  }

  # Extract log level
  grok {
    match => {
      "message" => [
        "%{LOGLEVEL:log_level}",
        "level=%{WORD:log_level}"
      ]
    }
    tag_on_failure => []
  }

  # Add Docker metadata
  mutate {
    add_field => {
      "environment" => "${ENVIRONMENT:production}"
    }
  }

  # Parse timestamps
  date {
    match => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss"]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "docker-logs-%{+YYYY.MM.dd}"
  }
}

Configure containers to send logs to Logstash via GELF:

services:
  app:
    image: myapp:latest
    logging:
      driver: gelf
      options:
        gelf-address: "udp://logstash:12201"
        tag: "myapp"
        labels: "environment,version"

Grafana Loki + Promtail

Loki is a lightweight alternative to Elasticsearch that indexes only log metadata (labels), not the full text of every log line. This makes it dramatically cheaper to run while still providing useful log querying through Grafana:

services:
  loki:
    image: grafana/loki:2.9.4
    container_name: loki
    ports:
      - "127.0.0.1:3100:3100"
    volumes:
      - loki-data:/loki
      - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
    command: -config.file=/etc/loki/local-config.yaml
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3100/ready || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5

  promtail:
    image: grafana/promtail:2.9.4
    container_name: promtail
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./loki/promtail-config.yml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:10.3.1
    container_name: grafana
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro

volumes:
  loki-data:
  grafana-data:

Loki Configuration

# loki/loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 30d
  max_query_length: 721h

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h

Promtail Configuration

# loki/promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # Scrape Docker container logs
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      # Use container name as label
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      # Use image name as label
      - source_labels: ['__meta_docker_container_image']
        target_label: 'image'
      # Use compose service name
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
      # Use compose project name
      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
        target_label: 'project'
    pipeline_stages:
      # Try to parse JSON logs
      - docker: {}
      - json:
          expressions:
            level: level
            msg: msg
            timestamp: time
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: RFC3339Nano

Fluentd and Fluent Bit

Fluentd and Fluent Bit are log processors that collect, transform, and route logs. Fluent Bit is the lightweight version (~450KB binary vs ~40MB for Fluentd):

services:
  fluent-bit:
    image: fluent/fluent-bit:2.2
    container_name: fluent-bit
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    ports:
      - "127.0.0.1:24224:24224"  # Forward input
    depends_on:
      - loki  # Or elasticsearch

# fluent-bit/fluent-bit.conf
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf

# Collect Docker container logs
[INPUT]
    Name              tail
    Tag               docker.*
    Path              /var/lib/docker/containers/*/*.log
    Parser            docker
    DB                /fluent-bit/db/docker.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10

# Also accept logs via forward protocol (fluentd logging driver)
[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224

# Parse JSON log messages
[FILTER]
    Name         parser
    Match        docker.*
    Key_Name     log
    Parser       json
    Reserve_Data On

# Add hostname
[FILTER]
    Name   modify
    Match  *
    Add    hostname ${HOSTNAME}

# Output to Loki
[OUTPUT]
    Name       loki
    Match      *
    Host       loki
    Port       3100
    Labels     job=fluent-bit, container=$container_name
    Auto_Kubernetes_Labels Off

# Or output to Elasticsearch
# [OUTPUT]
#     Name          es
#     Match         *
#     Host          elasticsearch
#     Port          9200
#     Index         docker-logs
#     Type          _doc
#     Suppress_Type_Name On

Structured Logging

The most effective way to improve your logging is to output structured JSON from your applications. This eliminates the need for complex parsing in Logstash or Fluent Bit:

// Go: structured logging with zap (JSON output)
logger, _ := zap.NewProduction()
defer logger.Sync()

logger.Info("user authenticated",
    zap.String("user_id", "12345"),
    zap.String("method", "oauth2"),
    zap.Duration("latency", time.Since(start)),
    zap.Int("status", 200),
)

// Output (single JSON line, easy to parse):
// {"level":"info","ts":1684234567.890,"caller":"auth/handler.go:42",
//  "msg":"user authenticated","user_id":"12345","method":"oauth2",
//  "latency":0.023,"status":200}

Structured logs allow you to filter and aggregate without regex parsing:

# Loki LogQL query examples
{service="myapp"} | json | level="error"
{service="myapp"} | json | latency > 1000
{service="myapp"} | json | status >= 500 | line_format "{{.method}} {{.path}} {{.status}}"

# Count errors per minute
sum(rate({service="myapp"} | json | level="error" [1m])) by (service)

Log Retention and Storage

# Elasticsearch Index Lifecycle Management (ILM)
PUT _ilm/policy/docker-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": { "max_size": "50gb", "max_age": "1d" }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Alerting on Logs

# Grafana alerting rule on Loki logs
# Alert when error rate exceeds threshold
groups:
  - name: log-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({service="myapp"} | json | level="error" [5m])) > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate in {{ $labels.service }}"
          description: "Error rate is {{ $value }} errors/second"

      - alert: ContainerCrashLoop
        expr: |
          count_over_time({container=~".+"} |= "container died" [5m]) > 3
        for: 1m
        labels:
          severity: warning

Which Stack to Choose

Choose ELK when you need powerful full-text search across logs, complex aggregations for analytics, or compliance requirements that demand full log indexing. Be prepared for the resource cost: Elasticsearch needs at least 2GB RAM for a single node.

Choose Loki + Promtail when you already use Grafana for metrics (Prometheus), want minimal storage overhead, and your log querying is primarily label-based filtering. This is the best choice for most Docker deployments.

Choose Fluent Bit as a log collector that can send to either ELK or Loki (or both). It is especially useful when you need to route logs to multiple destinations or transform them before storage.

Container management platforms like usulnet provide built-in log viewing for individual containers, which complements centralized logging by giving operators quick access to recent logs without switching to Kibana or Grafana. For deeper analysis and historical queries, the centralized logging stack provides the full picture.