Codebook

Appearance

第 215 / 250 章运维⏱ 12 分钟阅读

第 215 章:CI/CD 与监控 ​

学习目标

  • 掌握 GitHub Actions 工作流
  • 学会 Jenkins Pipeline
  • 部署 Prometheus + Grafana 监控
  • 搭建 ELK 日志聚合
  • 配置告警与值班

一、CI/CD 简介 ​

CI/CD 是 持续集成 / 持续交付 / 持续部署 的实践,自动化构建、测试、发布流程。

缩写含义
CIContinuous Integration(持续集成)
CDContinuous Delivery / Deployment(持续交付 / 部署)

二、GitHub Actions ​

2.1 基础概念 ​

概念说明
WorkflowYAML 配置文件(.github/workflows/*.yml)
Job一组步骤的集合,跑在 Runner 上
Step单个任务(shell 命令或 action)
Action可复用单元
Runner执行环境(ubuntu / windows)

2.2 Hello World ​

.github/workflows/ci.yml:

yaml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup JDK 17
        uses: actions/setup-java@v4
        with:
          java-version: '17'
          distribution: 'temurin'

      - name: Cache Maven
        uses: actions/cache@v3
        with:
          path: ~/.m2
          key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
          restore-keys: ${{ runner.os }}-m2-

      - name: Build
        run: mvn clean package -DskipTests

      - name: Test
        run: mvn test

2.3 构建 Docker 镜像并推送 ​

yaml
name: Build & Push

on:
  push:
    tags: ['v*']

jobs:
  docker:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Build & Push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            user/myapp:latest
            user/myapp:${{ github.ref_name }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

2.4 部署到 K8s ​

yaml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup kubectl
        uses: azure/setup-kubectl@v4

      - name: Set Kubeconfig
        run: |
          mkdir -p ~/.kube
          echo "${{ secrets.KUBECONFIG }}" > ~/.kube/config

      - name: Apply manifests
        run: |
          kubectl set image deployment/myapp \
            myapp=user/myapp:${{ github.sha }} \
            -n production

      - name: Wait for rollout
        run: kubectl rollout status deployment/myapp -n production

2.5 Matrix 矩阵 ​

yaml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        java: [11, 17, 21]
        node: [18, 20]

    steps:
      - uses: actions/setup-java@v4
        with:
          java-version: ${{ matrix.java }}

2.6 复用 Workflow ​

yaml
# .github/workflows/reusable.yml
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - run: echo "deploying to ${{ inputs.environment }}"

调用方:

yaml
jobs:
  call-workflow:
    uses: ./.github/workflows/reusable.yml
    with:
      environment: production

三、Jenkins ​

3.1 安装 ​

bash
# Docker
docker run -d -p 8080:8080 \
  -v jenkins_home:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  jenkins/jenkins:lts

# 首次密码
cat /var/jenkins_home/secrets/initialAdminPassword

3.2 Pipeline(Jenkinsfile) ​

groovy
pipeline {
    agent any

    tools {
        maven 'Maven-3.9'
        jdk 'JDK-17'
    }

    stages {
        stage('Checkout') {
            steps {
                git branch: 'main',
                    url: 'https://github.com/user/myapp.git',
                    credentialsId: 'github-token'
            }
        }

        stage('Build') {
            steps {
                sh 'mvn clean package -DskipTests'
            }
        }

        stage('Test') {
            steps {
                sh 'mvn test'
                junit '**/target/surefire-reports/*.xml'
            }
        }

        stage('Docker Build') {
            steps {
                script {
                    docker.build("user/myapp:${env.BUILD_ID}")
                }
            }
        }

        stage('Deploy') {
            when {
                branch 'main'
            }
            steps {
                sh 'kubectl set image deployment/myapp myapp=user/myapp:${BUILD_ID}'
            }
        }
    }

    post {
        success {
            echo '构建成功'
            mail to: 'team@example.com',
                 subject: "Build Success: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
        }
        failure {
            mail to: 'team@example.com',
                 subject: "Build Failed: ${env.JOB_NAME} ${env.BUILD_NUMBER}"
        }
    }
}

3.3 声明式 vs 脚本式 ​

groovy
// 脚本式 Pipeline
node {
    stage('Build') {
        sh 'mvn clean package'
    }
}

3.4 共享库 ​

vars/sayHello.groovy:

groovy
def call(String name) {
    echo "Hello ${name}"
}

Jenkinsfile:

groovy
@Library('my-shared-lib') _
sayHello('World')

四、Prometheus 监控 ​

4.1 简介 ​

4.2 安装 Prometheus ​

yaml
# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prom_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus

volumes:
  prom_data:
  grafana_data:

4.3 prometheus.yml ​

yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # Prometheus 自身
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # 应用 /metrics
  - job_name: 'myapp'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['host.docker.internal:8080']

  # node-exporter(主机监控)
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  # K8s 服务发现
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod

4.4 Spring Boot 集成 ​

pom.xml:

xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

application.yml:

yaml
management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
  metrics:
    tags:
      application: ${spring.application.name}

访问 http://localhost:8080/actuator/prometheus 即可看到指标。

4.5 自定义指标 ​

java
@RestController
public class OrderController {

    @Autowired
    private MeterRegistry registry;

    private final Counter orderCounter;
    private final Timer orderTimer;

    public OrderController(MeterRegistry registry) {
        this.registry = registry;
        this.orderCounter = Counter.builder("orders_total")
            .tag("type", "success")
            .register(registry);
        this.orderTimer = Timer.builder("order_duration")
            .register(registry);
    }

    @PostMapping("/order")
    public Result create() {
        Timer.Sample sample = Timer.start(registry);
        try {
            // 业务逻辑
            orderCounter.increment();
            return Result.ok();
        } finally {
            sample.stop(orderTimer);
        }
    }
}

4.6 PromQL 查询 ​

promql
# QPS(每分钟订单数)
rate(orders_total[1m])

# 99 分位响应时间
histogram_quantile(0.99, sum(rate(order_duration_seconds_bucket[5m])) by (le))

# CPU 使用率
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 内存使用
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

五、Grafana 可视化 ​

5.1 添加 Prometheus 数据源 ​

Configuration → Data Sources → Prometheus

URL: http://prometheus:9090

5.2 JVM 仪表板 ​

导入 4701(JVM Micrometer) 即可看到完整的 JVM 监控面板。

5.3 自定义面板 ​

json
{
  "panels": [
    {
      "title": "QPS",
      "type": "graph",
      "targets": [
        {
          "expr": "rate(orders_total[1m])"
        }
      ]
    },
    {
      "title": "P99 延迟",
      "type": "graph",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, rate(order_duration_seconds_bucket[5m]))"
        }
      ]
    }
  ]
}

六、AlertManager 告警 ​

6.1 alertmanager.yml ​

yaml
global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'critical'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://dingtalk-webhook/alert'

  - name: 'critical'
    webhook_configs:
      - url: 'http://dingtalk-webhook/critical'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster']

6.2 告警规则 ​

prometheus/rules/app.yml:

yaml
groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) /
          sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "错误率过高"
          description: "{{ $labels.instance }} 5xx 错误率 {{ $value }}"

      - alert: HighResponseTime
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 响应时间过高"
          description: "当前 {{ $value }}s"

      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning

七、ELK 日志聚合 ​

7.1 架构 ​

7.2 docker-compose ​

yaml
version: '3.8'

services:
  elasticsearch:
    image: elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

  kibana:
    image: kibana:8.11.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200

  filebeat:
    image: elastic/filebeat:8.11.0
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - elasticsearch

volumes:
  es_data:

7.3 filebeat.yml ​

yaml
filebeat.inputs:
  - type: container
    paths:
      - /var/lib/docker/containers/*/*.log
    processors:
      - add_docker_metadata: ~

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  indices:
    - index: "app-logs-%{+yyyy.MM.dd}"

setup.kibana:
  host: "kibana:5601"

logging.level: info

7.4 Spring Boot 日志输出 JSON ​

logback-spring.xml:

xml
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <fieldNames>
                <timestamp>@timestamp</timestamp>
                <message>message</message>
                <thread>thread</thread>
                <logger>logger</logger>
                <level>level</level>
            </fieldNames>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

依赖:

xml
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.4</version>
</dependency>

7.5 Kibana 查询 ​

status:500 AND app:myapp

# 错误日志 + 堆栈
level:ERROR AND stack_trace:*

# 慢请求
duration:>1000

# 按服务聚合
GET app-logs-*/_search
{
  "aggs": {
    "by_service": {
      "terms": { "field": "service.keyword" }
    }
  }
}

八、Loki(轻量级日志方案) ​

比 ELK 更轻量,适合 K8s 环境。

yaml
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:ro
      - ./promtail.yml:/etc/promtail/config.yml

promtail.yml:

yaml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          __path__: /var/log/*.log

九、K8s 监控方案 ​

9.1 kube-prometheus-stack ​

bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

包括:

  • Prometheus Operator
  • Prometheus + AlertManager
  • Grafana(预置仪表板)
  • node-exporter / kube-state-metrics
  • ServiceMonitor 自定义资源

9.2 ServiceMonitor ​

yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-monitor
  labels:
    release: kube-prometheus
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: http
      path: /actuator/prometheus
      interval: 15s

应用 Service:

yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  ports:
    - name: http
      port: 8080
  selector:
    app: myapp

十、APM(应用性能监控) ​

10.1 SkyWalking ​

yaml
# agent 下载
wget https://archive.apache.org/dist/skywalking/...

# 启动参数
java -javaagent:/path/skywalking-agent.jar \
     -Dskywalking.agent.service_name=myapp \
     -Dskywalking.collector.backend_service=127.0.0.1:11800 \
     -jar myapp.jar

可追踪:

  • 服务拓扑
  • 慢 SQL
  • 慢调用链
  • JVM 实时指标

10.2 替代方案 ​

  • Pinpoint:韩国出品,功能强大
  • Zipkin:Twitter 开源,轻量
  • Jaeger:Uber 开源,云原生

十一、SLO 与值班 ​

11.1 SLO 定义 ​

yaml
# Service Level Objective
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: myapp-slo
spec:
  service: myapp
  labels:
    owner: team-backend
  slos:
    - name: availability
      objective: 99.9
      description: "99.9% 可用性"
      events:
        error_query: sum(rate(http_requests_total{status=~"5.."}[5m]))
        total_query: sum(rate(http_requests_total[5m]))

11.2 错误预算 ​

SLO = 99.9%
30 天 = 30 × 24 × 60 = 43200 分钟
错误预算 = 0.1% × 43200 = 43.2 分钟

11.3 值班 OnCall ​

级别响应时间
P0(宕机)5 分钟
P1(严重)30 分钟
P2(一般)2 小时
P3(提示)下个工作日

工具: PagerDuty / OpsGenie / 自建

十二、本章小结 ​

工具用途
GitHub Actions轻量 CI/CD
Jenkins传统 CI/CD
Prometheus指标采集与存储
Grafana可视化仪表板
AlertManager告警路由
ELK / Loki日志聚合
SkyWalking链路追踪

动手练习 ​

  1. 编写 GitHub Actions 自动构建并推送镜像
  2. 用 docker-compose 部署 Prometheus + Grafana
  3. Spring Boot 暴露 /actuator/prometheus,在 Grafana 中配置仪表板
  4. 配置告警规则,触发后发送钉钉消息
  5. 用 Filebeat 收集容器日志,接入 ELK

推荐阅读 ​

下一章:第 216 章:DDD 与领域驱动设计

本站基于 VitePress 构建 · 由 Codebook 团队维护

Copyright © 2026-present Codebook