Prometheus 可观测性

第 13 章 · Prometheus 可观测性

本章目标：在 api-go-demo 暴露 /metrics；Gin 中间件采集 RED 指标（Rate、Errors、Duration）；增加缓存 hit/miss 业务指标；理解 Grafana 仪表盘与告警（选修）；满足 ch16 验收；对照 fastapi-web ch18 与 paas ServiceMonitor。

学时建议：3～4 小时（含 1.5 小时跟练）

前置：完成 gin-web ch12；ch08 access log 已有 request_id。

13.1 场景说明：可观测性三大支柱

         Metrics              Logs               Traces
    Prometheus/Grafana    zap JSON (ch08)     OpenTelemetry（选修）
    「系统是否异常？」     「发生了什么？」      「慢在哪里？」

最小运维端点：

GET /health   → {"status":"ok"}     # liveness
GET /metrics  → Prometheus text    # 抓取

严禁将 /metrics 暴露公网；虚构域名 api.example.com。

RED 方法（面向请求型 API）：

13.2 依赖安装

go get github.com/prometheus/client_golang@v1.20.5

目录：

internal/
└── middleware/
    └── metrics.go

13.3 HTTP RED 中间件（完整）

// internal/middleware/metrics.go
package middleware

import (
    "strconv"
    "time"

    "github.com/gin-gonic/gin"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    httpRequestsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests by method, path template, status",
        },
        []string{"method", "path", "status"},
    )
    httpRequestDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request latency in seconds",
            Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
        },
        []string{"method", "path"},
    )
    httpRequestsInFlight = promauto.NewGauge(
        prometheus.GaugeOpts{
            Name: "http_requests_in_flight",
            Help: "Current in-flight HTTP requests",
        },
    )
)

func Prometheus() gin.HandlerFunc {
    return func(c *gin.Context) {
        if c.Request.URL.Path == "/metrics" {
            c.Next()
            return
        }
        httpRequestsInFlight.Inc()
        start := time.Now()
        c.Next()
        httpRequestsInFlight.Dec()

        status := strconv.Itoa(cWriterStatus(c))
        path := c.FullPath()
        if path == "" {
            path = "unknown"
        }
        method := c.Request.Method
        httpRequestsTotal.WithLabelValues(method, path, status).Inc()
        httpRequestDuration.WithLabelValues(method, path).Observe(time.Since(start).Seconds())
    }
}

func cWriterStatus(c *gin.Context) int {
    if c.Writer.Status() == 0 {
        return 200
    }
    return c.Writer.Status()
}

为何用 FullPath：/api/v1/products/:slug 而非每个 slug 一个 label，避免高基数。

13.4 挂载 /metrics 与 /health

// cmd/server/main.go
import (
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func setupRouter(cfg *config.Config, deps *Deps) *gin.Engine {
    r := gin.New()
    r.Use(middleware.RequestID())
    r.Use(middleware.Prometheus())
    r.Use(middleware.AccessLog(deps.Log))

    r.GET("/health", func(c *gin.Context) {
        c.JSON(200, gin.H{"status": "ok"})
    })
    r.GET("/metrics", gin.WrapH(promhttp.Handler()))

    v1 := r.Group("/api/v1")
    // ... routes
    return r
}

验证：

curl -s http://127.0.0.1:8080/metrics | head -30
curl -s http://127.0.0.1:8080/metrics | grep http_requests_total

# 打流量
for i in $(seq 1 100); do curl -s -o /dev/null http://127.0.0.1:8080/api/v1/products/go-handbook; done
curl -s http://127.0.0.1:8080/metrics | grep 'path="/api/v1/products/:slug"'

13.5 业务指标：缓存与商品

衔接 ch09：

// internal/metrics/cache.go
package metrics

import "github.com/prometheus/client_golang/prometheus/promauto"

var (
    RedisCacheHits = promauto.NewCounter(prometheus.CounterOpts{
        Name: "redis_cache_hits_total",
        Help: "Redis cache hits for product detail",
    })
    RedisCacheMisses = promauto.NewCounter(prometheus.CounterOpts{
        Name: "redis_cache_misses_total",
        Help: "Redis cache misses for product detail",
    })
    ProductsPublished = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "products_published_total",
        Help: "Count of products with is_published=true",
    })
)

Service 埋点：

if hit {
    metrics.RedisCacheHits.Inc()
} else {
    metrics.RedisCacheMisses.Inc()
}

启动时刷新 Gauge：

func refreshProductGauge(ctx context.Context, db *gorm.DB) {
    var n int64
    db.WithContext(ctx).Model(&model.Product{}).Where("is_published = ?", true).Count(&n)
    metrics.ProductsPublished.Set(float64(n))
}

可选 Counter：orders_created_total（ch10 扩展）。