第 19 章:服务监控

学习目标

• 掌握 Prometheus + Grafana 监控体系
• 学会用 Micrometer 上报指标
• 配置告警和告警收敛
• 避免指标过多、告警轰炸的坑

一、监控的四类指标(USE / RED)

USE 方法(基础设施)
  U - Utilization(使用率):CPU、内存、磁盘
  S - Saturation(饱和度):队列长度
  E - Errors(错误数)

RED 方法(服务)
  R - Rate(请求量)
  E - Errors(错误数)
  D - Duration(延迟)

// 业务指标
private final Counter orderCount = Metrics.counter("order.create.count");
private final Timer orderTimer = Metrics.timer("order.create.duration");
private final Gauge activeUsers = Metrics.gauge("user.active", this);

public Order createOrder(OrderDto dto) {
    return orderTimer.record(() -> {
        Order order = orderService.create(dto);
        orderCount.increment();
        return order;
    });
}

二、Spring Boot Actuator

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  endpoint:
    health:
      show-details: always
  metrics:
    export:
      prometheus:
        enabled: true

# 访问指标
curl http://localhost:8080/actuator/health
curl http://localhost:8080/actuator/metrics
curl http://localhost:8080/actuator/prometheus

# 指标输出
jvm_memory_used_bytes{area="heap",id="G1 Old Gen"} 1.024e9
http_server_requests_seconds_count{status="200",uri="/api/order"} 1234

⚠️ 坑 1:management.endpoints.web.exposure.include=* 暴露所有端点,生产环境 /actuator/env 暴露配置 = 泄漏密钥。只暴露需要的。

三、Micrometer 自定义指标

@Service
public class OrderMetrics {

    private final MeterRegistry registry;

    public OrderMetrics(MeterRegistry registry) {
        this.registry = registry;
    }

    // 1. Counter:计数
    public void recordOrderCreated(String channel) {
        registry.counter("order.created", "channel", channel).increment();
    }

    // 2. Timer:耗时
    public void recordOrderProcess(Runnable task) {
        Timer.builder("order.process.duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry)
            .record(task);
    }

    // 3. Gauge:当前值
    @Bean
    public Gauge inventoryGauge(InventoryService service) {
        return Gauge.builder("inventory.total", service, InventoryService::totalStock)
            .register(registry);
    }

    // 4. DistributionSummary:分布
    public void recordOrderAmount(double amount) {
        DistributionSummary.builder("order.amount")
            .publishPercentiles(0.5, 0.95, 0.99)
            .register(registry)
            .record(amount);
    }
}

四、Prometheus 抓取

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'order-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['order-service:8080']
        labels:
          service: order

  - job_name: 'payment-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['payment-service:8081']

// 拉模式 vs 推模式
// Prometheus 拉模式(默认):服务只暴露端口
// Pushgateway:临时任务主动推送

五、Grafana 仪表盘

Grafana
├── JVM 仪表盘(JVM Memory / GC / Thread)
├── 应用仪表盘(QPS / 延迟 / 错误率)
├── 数据库仪表盘(连接数 / 慢查询)
└── 业务仪表盘(订单量 / GMV)

# 关键查询
# 1. QPS
rate(http_server_requests_seconds_count[5m])

# 2. P99 延迟
histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))

# 3. 错误率
rate(http_server_requests_seconds_count{status=~"5.."}[5m]) / rate(http_server_requests_seconds_count[5m])

# 4. JVM 堆内存使用
jvm_memory_used_bytes{area="heap"}

# 5. GC 次数
rate(jvm_gc_pause_seconds_count[5m])

六、告警规则

# alerting-rules.yml
groups:
  - name: service-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_server_requests_seconds_count{status=~"5.."}[5m]) /
          rate(http_server_requests_seconds_count[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "服务错误率超过 5%"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning

      - alert: JVMHeapHigh
        expr: jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"} > 0.85
        for: 10m
        labels:
          severity: warning

# Alertmanager 路由
route:
  receiver: 'team-dev'
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      repeat_interval: 1h

⚠️ 坑 2:告警阈值太敏感,半夜被短信炸醒。告警收敛 + 抑制,只对 critical 级别发钉钉/电话。

七、业务监控

// 业务异常指标
@Component
public class BusinessMetrics {

    @Autowired
    private MeterRegistry registry;

    // 业务关键指标
    public void onLoginSuccess(String channel) {
        registry.counter("business.login.success", "channel", channel).increment();
    }

    public void onLoginFailure(String reason) {
        registry.counter("business.login.failure", "reason", reason).increment();
    }

    public void onOrderPaid(BigDecimal amount) {
        registry.counter("business.order.paid").increment();
        registry.summary("business.order.amount").record(amount.doubleValue());
    }

    // 业务大盘指标
    public void onInventoryLow(Long skuId, int stock) {
        if (stock < 10) {
            registry.counter("business.inventory.low", "skuId", String.valueOf(skuId)).increment();
        }
    }
}

八、告警值班

告警分级
├── P0:核心业务宕机,立即响应
├── P1:部分功能不可用,30 分钟
├── P2:性能下降,2 小时
└── P3:潜在风险,24 小时

值班表(轮转)
├── 第一个月:张三
├── 第二个月:李四
└── ...

⚠️ 坑 3:一个人兜底所有告警,告警疲劳导致重要告警被忽略。轮值 + 抑制 + 分级。

本章小结

维度	工具
指标采集	Micrometer
存储	Prometheus
可视化	Grafana
告警	Alertmanager
通知	钉钉/飞书/PagerDuty

关键点	建议
指标	4 类黄金指标
采样	高频 1s,低频 30s
告警	收敛 + 分级
值班	7x24 轮值

动手练习

1. Actuator 暴露:开启 /actuator/prometheus,验证指标输出
2. 自定义指标:加一个 order.created 计数器,统计订单创建
3. Grafana 仪表盘:导入 JVM 仪表盘,观察 GC 曲线
4. 告警规则:配置错误率超 5% 触发告警,接入钉钉

---

下一章:第 20 章:日志聚合 →