第 20 章:日志聚合

学习目标

• 掌握 ELK / Loki 日志聚合方案
• 学会结构化日志输出
• 配置日志查询和告警
• 避免日志丢失、磁盘爆满的坑

一、为什么需要日志聚合?

单机时代
[App 日志] → /var/log/app.log → 人工 tail
3 个机器?轮流登录看 ❌

微服务时代
[Order 日志] → Filebeat → Elasticsearch → Kibana
[Payment 日志] →
[Inventory 日志] →
统一个控制台查询 ✅

二、ELK 体系

E - Elasticsearch(存储 + 搜索)
L - Logstash(采集 + 解析)
K - Kibana(可视化)

简化版:EFK
F - Filebeat(轻量采集)

# docker-compose.yml
version: '3'
services:
  elasticsearch:
    image: elasticsearch:8.10.0
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node

  kibana:
    image: kibana:8.10.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  filebeat:
    image: elastic/filebeat:8.10.0
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/log/app:/var/log/app:ro

三、Logback 配置

<!-- logback-spring.xml -->
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>userId</includeMdcKeyName>
            <customFields>{"service":"order-service","env":"prod"}</customFields>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/app/order.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>/var/log/app/order-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

四、结构化日志

// ❌ 不推荐:字符串拼接
log.info("用户 " + userId + " 下单 " + orderId + " 金额 " + amount);
// 解析困难,无法聚合

// ✅ 推荐:结构化字段
log.info("订单创建",
    kv("userId", userId),
    kv("orderId", orderId),
    kv("amount", amount),
    kv("channel", "APP")
);

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
</dependency>

// 使用 MDC 传递上下文
@Aspect
@Component
public class RequestLogAspect {
    @Around("@annotation(RequestLog)")
    public Object around(ProceedingJoinPoint pjp) throws Throwable {
        try {
            MDC.put("userId", SecurityContext.getUserId());
            MDC.put("traceId", MDC.get("traceId"));
            return pjp.proceed();
        } finally {
            MDC.remove("userId");
        }
    }
}

// ES 里一条日志
{
  "@timestamp": "2026-01-15T10:00:00.123Z",
  "level": "INFO",
  "service": "order-service",
  "env": "prod",
  "traceId": "abc123",
  "userId": "user-001",
  "message": "订单创建",
  "orderId": "ORD001",
  "amount": "99.00"
}

⚠️ 坑 1:日志里塞 password、secretKey、身份证号,ES 里被搜索引擎索引 = 重大事故。敏感字段绝对不写日志。

五、Filebeat 配置

# filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /var/log/app/*.log
    json.keys_under_root: true
    json.add_error_key: true
    fields:
      service: order-service

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.hostname"]

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "app-logs-%{+yyyy.MM.dd}"

logging.level: info

六、Kibana 查询

Kibana Discover
├── 过滤字段:service:order-service
├── 时间范围:last 1 hour
└── TraceId 查询:traceId:"abc123"

可视化
├── 错误日志分布图
├── 各服务日志量
└── 慢接口日志

常用查询语法
├── service:order-service AND level:ERROR
├── traceId:"abc123"
├── orderId:ORD001
└── @timestamp:[now-1h TO now]

七、Loki 轻量方案

# Grafana Loki + Promtail
# 资源占用比 ELK 小 10 倍

# promtail.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - labels:
          job: order-service
          __path__: /var/log/app/order.log

// Logback 写 Loki
<dependency>
    <groupId>com.github.loki4j</groupId>
    <artifactId>loki-logback-appender</artifactId>
</dependency>

<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
    <httpUrl>http://loki:3100/loki/api/v1/push</httpUrl>
    <labels>
        <label>service=order-service</label>
        <label>env=prod</label>
    </labels>
</appender>

八、日志成本控制

# 1. 日志分级
logging:
  level:
    root: WARN
    com.example.order: INFO
    org.springframework: WARN

# 2. 采样(INFO 100%,DEBUG 1%)
public class Sampler {
    public static boolean shouldLog() {
        return ThreadLocalRandom.current().nextDouble() < 0.01;
    }
}

// 3. 异步写日志,不阻塞业务
@Async
public void logBusinessEvent(String event, Object data) {
    log.info("{}: {}", event, data);
}

⚠️ 坑 2:@Async 线程池默认 SimpleAsyncTaskExecutor,每次创建新线程 = 内存泄漏。配置专用线程池。

# 4. 索引保留策略
# ES ILM(索引生命周期管理)
hot → warm → cold → delete

30 天后归档,90 天后删除

九、日志与告警

# ElastAlert
rules:
  - name: error_log_alert
    type: frequency
    index: app-logs-*
    filter:
      - term:
          level: ERROR
    query_key: service
    num_events: 10
    timeframe:
      minutes: 5
    alert:
      - slack
    slack:
      slack_webhook_url: "https://hooks.slack.com/..."

本章小结

方案	特点
ELK	成熟,功能强,资源重
Loki	轻量,标签查询,适合云原生
EFK	简化版,Filebeat 替代 Logstash

关键点	建议
结构化	字段而非字符串
敏感字段	必脱敏
保留	30 天,归档
告警	错误率阈值

动手练习

1. ELK 部署:用 Docker Compose 起 ES + Kibana + Filebeat
2. 结构化输出:用 LogstashEncoder 输出 JSON 日志
3. Kibana 查询:按 traceId 查一条请求的完整日志
4. 日志告警:配置错误率超阈值触发钉钉告警

---

下一章:第 21 章:性能调优 →