← Back to glossary

Log Aggregation (ELK/Loki)

observability Intermediate

TL;DR

Log aggregation collects logs from all services into a central searchable store — ELK (Elasticsearch+Logstash+Kibana) for full-text search, Loki (Prometheus-style) for cost-efficient label-based search.

Explanation

ELK stack: Logstash/Filebeat (collect) → Elasticsearch (store+index) → Kibana (search+dashboard). Full-text indexed — any field searchable, high storage cost. Loki: Grafana's log store — only indexes labels (not content), compressed content. Much cheaper than ELK. PromQL-like LogQL. Best with structured logs (JSON). Alternatives: Datadog Logs, Splunk (expensive but powerful), CloudWatch Logs. Key capabilities: full-text search, aggregation (error count by service), dashboards, alerts on log patterns. Ship logs: Filebeat/Fluentd agent → aggregator. In PHP: Monolog with socket/HTTP handler → Logstash/Loki.

Common Misconception

✗ More log storage is always better — logs stored but never searched are expensive waste. Store what you query; sample debug logs heavily.

Why It Matters

Centralised log aggregation transforms debugging from SSH-to-server-and-grep to sub-second search across all services — essential for microservices and autoscaling environments.

Common Mistakes

Not shipping logs to a central store — grep-across-servers debugging.
Storing all debug logs at full rate — expensive and noisy.
Not using structured logs — full-text search works, but JSON fields are essential for aggregation.

Code Examples

✗ Vulnerable

# No aggregation — SSH to each server:
ssh server1 grep 'ERROR' /var/log/app.log
ssh server2 grep 'ERROR' /var/log/app.log
# 20 servers = 20 SSH sessions

✓ Fixed

# Loki config:
- job_name: php_app
  static_configs:
    - targets: [localhost]
      labels:
        job: php-app
        env: production
  pipeline_stages:
    - json:
        expressions:
          level: level
          correlation_id: correlation_id
    - labels:
        level:
        correlation_id:

# LogQL query:
{job='php-app'} |= 'ERROR' | json | correlation_id='abc-123'

References

↗ https://grafana.com/docs/loki/latest/