第 9 章:监控告警

学习目标

• 掌握 Prometheus 指标采集
• 配置 Grafana 可视化面板
• 接入告警(Alertmanager)
• 用 Spring Boot Actuator 暴露指标
• 避开指标爆炸和告警风暴坑

一、Prometheus 三大组件

Exporter(指标采集)  →  Prometheus(存储 + 查询)  →  Grafana(可视化)
                                              ↓
                                       Alertmanager(告警)

二、Spring Boot 暴露指标

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

# application.yml
management:
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint:
    health:
      show-details: always
  metrics:
    tags:
      application: myapp

验证:

curl http://localhost:8080/actuator/health         # 状态
curl http://localhost:8080/actuator/prometheus      # Prometheus 格式指标

⚠️ 坑 1:actuator 默认只暴露 health, /metrics 看不到。include: "*" 才能全开。生产环境别开 *,至少 include: health,prometheus,info。

三、Prometheus 配置

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'spring-app'
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ['app:8080']
        labels:
          env: prod

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'mysql'
    static_configs:
      - targets: ['mysqld-exporter:9104']

常用 exporter:

服务	Exporter
Linux 主机	node_exporter(9100)
Docker / K8s	cAdvisor(内置)
MySQL	mysqld_exporter(9104)
Redis	redis_exporter(9121)
Nginx	nginx_exporter(9113)
RabbitMQ	rabbitmq_exporter(9419)

⚠️ 坑 2:Prometheus 默认每 15 秒拉一次,小型集群够用。中大规模集群压得扛不住指标爆炸 → 调采样间隔到 30~60s,并丢掉不常用指标。

四、PromQL 常用查询

# CPU 使用率(节点)
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用率
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

# HTTP 请求率(每秒)
sum(rate(http_server_requests_seconds_count[5m]))

# 错误率(5xx)
sum(rate(http_server_requests_seconds_count{status="5xx"}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m])) * 100

# P99 延迟
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le))

# 容器内存
container_memory_usage_bytes{pod="myapp-xxx"}

五、Grafana 面板

Docker Compose 一键起

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - prometheus

  alertmanager:
    image: prom/alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

  node-exporter:
    image: prom/node-exporter
    ports:
      - "9100:9100"

volumes:
  grafana-data:

配置 Prometheus 数据源:URL http://prometheus:9090,Grafana 自动 Import 官方仪表盘(ID 1860 是 Node Exporter Full)。

六、告警规则(Alertmanager)

# prometheus 告警规则
groups:
- name: app
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_server_requests_seconds_count{status="5xx"}[5m]))
      / sum(rate(http_server_requests_seconds_count[5m])) > 0.05
    for: 5m                          # 持续 5 分钟才告警
    labels:
      severity: critical
    annotations:
      summary: "5xx 错误率 > 5%"
      description: "当前 {{ $value | humanizePercentage }}"

  - alert: HighCPU
    expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} CPU > 85%"

# alertmanager.yml
route:
  receiver: 'ops-team'
  group_wait: 30s                    # 告警分组等待
  group_interval: 5m                 # 同组告警发送间隔
  repeat_interval: 3h                # 相同告警重复间隔

receivers:
- name: 'ops-team'
  webhook_configs:
    - url: 'https://oapi.dingtalk.com/robot/send?access_token=xxx'
      send_resolved: true
  email_configs:
    - to: 'ops@example.com'

# 抑制(已有 Service Down 就不发 Pod 异常了)
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'instance']

⚠️ 坑 3:告警没加 for: 5m → 抖动瞬间触发,告警风暴。所有告警都要有 for,防止短暂尖刺刷屏。

七、接入钉钉 / 飞书 / Slack

receivers:
- name: 'dingtalk'
  webhook_configs:
    - url: 'https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN'
      send_resolved: true

测试模板(可在 Grafana 调试):

{
  "msgtype": "markdown",
  "markdown": {
    "title": "服务告警",
    "text": "**{{ .GroupLabels.alertname }}**\n 实例:{{ .CommonLabels.instance }}\n 状态:{{ .Status }}\n 时间:{{ .CommonAnnotations.description }}"
  }
}

八、日志聚合(Loki 替代方案)

ELK / Loki 是日志侧的补充,排查比指标更深:

方案	特点
ELK(Elasticsearch + Logstash + Kibana)	功能最全,运维重
EFK(Fluentd 替 Logstash)	容器友好
Loki + Grafana	标签存储,极轻量,和 Grafana 集成

Loki docker-compose 一段:

loki:
  image: grafana/loki
  ports:
    - "3100:3100"
  volumes:
    - ./loki-config.yaml:/etc/loki/config.yaml

promtail:
  image: grafana/promtail
  volumes:
    - /var/log:/var/log
    - ./promtail-config.yaml:/etc/promtail/config.yaml

Grafana 数据源加 Loki,日志查询 {app="myapp"} |= "error"。

⚠️ 坑 4:日志打全级别(INFO) + 不分索引 → ELK 撑爆。生产只收 INFO 关键业务 / ERROR 全收,DEBUG 不开。

九、告警值班轮换

最简值班:

P0(致命) → 立即电话/SMS
P1(严重) → 5 分钟内响应(钉钉 @值班人)
P2(警告) → 工作日处理

避免疲劳:

• 每类告警只发 1 条(Notification Grouping)
• 相同告警合并 group_by: [alertname, instance]
• 用抑制规则避免连锁

十、本章小结

层	工具
指标采集	Prometheus(主动 pull)
可视化	Grafana
告警	Alertmanager + 钉钉/Slack
日志	Loki / ELK(辅助)
探针	Spring Boot Actuator

动手练习

1. Spring Boot 项目加 actuator,/actuator/prometheus 返回 200,看到自己的自定义指标
2. 用 Docker Compose 起 Prometheus + Grafana,绑定数据源后看到 Spring Boot 指标入库
3. 配一条告警:HTTP 5xx 比例超过 5% 持续 5 分钟,推到钉钉机器人
4. Grafana Import Node Exporter Full(ID 1860)面板,看主机 CPU / 内存 / 磁盘图

---

文档结束:devops 9 章完结。下一阶段可看 架构篇 或 中间件篇 →