审计发现 5 处硬伤(/metrics 空壳、零备份、prod 双副本数据分裂、零审计、资源偏紧),
本轮全部落地。
=== P0-1 真实 /metrics(原来是空壳)===
原实现只返回一行注释 `# LokiStack Metrics`,注释却写着"可被 Prometheus 抓取",
平台挂了/DB 写不进去/告警引擎卡死都只能靠用户反馈。
现接入 prometheus/client_golang,四层指标:
HTTP : lokistack_http_requests_total / _duration_seconds / _in_flight_requests
存储 : lokistack_db_size_bytes / _db_save_duration_seconds / _db_save_total / _db_records
业务 : lokistack_deployments{status,mode} / lokistack_collectors{status,type,run_mode}
后台任务: lokistack_alert_evaluations_total / _duration_seconds / lokistack_loki_queries_total
+ go_* / process_*(GoCollector、ProcessCollector)
关键点:path 标签用 c.FullPath() 路由模板,不用原始 URL —— 否则 117 个 Deployment
会把 Prometheus 基数炸掉。指标定义下沉 pkg/metrics 叶子包,避开
services ↔ repository 循环依赖。
=== P0-2 备份与恢复(原来 0 处代码)===
平台全部状态在 /app/data/opsxloki.json(51KB)+ kubeconfigs/ + known_hosts/,
DB 丢了 = 失去"这些资源是谁建的"的映射,从此无法管理/卸载/升级在跑的 Loki。
CLI : lokistack backup --output <path> / restore <pkg> / self-check / version
API : GET /system/backup(下载)/ POST /system/restore(校验暂存)/
POST /system/restore/apply(确认应用)/ GET /system/backups
K8s : kubernetes/pdb-backup.yaml —— 每日 03:30 CronJob,保留最近 7 份
安全设计:
- manifest.json 记录 ENCRYPTION_KEY 的 sha256 前 16 位**指纹**,用于恢复时
校验"这把钥匙能不能解开这份备份",不泄露密钥本身
- 恢复分两步(校验暂存 → 确认应用),应用前自动把当前 DB 备份为
opsxloki.json.before-restore.<时间戳>
- 包内不含明文密钥(kubeconfig 本就是加密态)
=== P0-3 prod compose 双副本 → 单副本 ===
lokistack-1 / lokistack-2 各挂独立数据卷 + nginx 轮询 = 数据分裂:
页面刷新两次可能看到不同结果,写操作只落一个副本。不是高可用。
已改单实例(nginx 仅做反代),文件顶部与 nginx.conf 都写明原因。
=== P0-4 操作审计日志(借鉴 /usr/local/src/aiops 的字段口径)===
aiops 是每个 controller 各写 recordXxxAudit();本平台有 60+ 写接口,
逐个埋点必然漏,漏掉的那个恰好最危险(删集群、改存储凭证)。
改为 **gin 中间件统一记录** + handler 通过 SetAudit 补充业务语义:
models/audit.go : 字段对齐 aiops,另加 Method/Path/Before/After
services/audit.go : 写入 + 90 天/2 万条自动裁剪 + 脱敏 + diff 生成
middleware/audit.go : 自动识别 module/action/resource,跳过只读接口
controllers/audit.go: 列表/详情/统计(仅 admin)
敏感字段(password/secretKey/privateKey/token)写入前统一替换为 ***。
=== P1 稳定性 ===
- 平台资源 1C/1Gi → 4C/4Gi 限(500m/512Mi 请求):JSON DB 全量读写 +
每分钟告警评估 + 批量发布并发,1Gi 会 OOM
- terminationGracePeriodSeconds: 30 —— 收到 SIGTERM 要等 DB 完整落盘,
否则 kubelet SIGKILL 会把 JSON 写一半(全库丢失)
- liveness timeoutSeconds 10s(默认 1s 写盘时会被误杀)
- PDB minAvailable: 0 —— 让 kubectl drain 正确等待重建,不卡维护
- DB schemaVersion + migrateSchema() 版本递进迁移(原先加字段全靠零值兜底)
- scripts/release.sh —— 不可变 tag <版本>-<日期>-<commit> + IfNotPresent,
根治 latest 并发构建互相覆盖导致的静默回滚(本项目已实际踩过)
- PVC 保持 30Gi(按长期增长预留,云盘单价低,不做在线扩容)
=== P2 易用性 ===
- 一键自检 GET /system/self-check + 仪表盘自检卡:真实探活 Loki /ready、
集群凭据解密、磁盘空间、主机状态、近 24h 失败操作
- 审计日志页 /audit:统计卡 + 用户名/模块/结果筛选 + 变更前后 diff 弹窗
- 菜单收敛 13 → 7(页内 tab 切换,URL 全部不变):
资源接入 = 集群 + 主机 + 存储
日志服务 = Loki 实例 + 采集器
日志 = 查询 + 分析
系统管理 = 用户 + 审计 + 设置
新增 GroupTabs 组件 + meta.group/groupKey,侧栏按分组去重并正确高亮
测试:go vet 通过、go test 全绿、tsc 通过
207 lines
5.7 KiB
Go
207 lines
5.7 KiB
Go
package middleware
|
||
|
||
import (
|
||
"net/http"
|
||
"strings"
|
||
"time"
|
||
|
||
"github.com/gin-gonic/gin"
|
||
|
||
"lokistack/models"
|
||
"lokistack/services"
|
||
)
|
||
|
||
// AuditInfo 由 controller 主动补充的业务语义。
|
||
//
|
||
// 中间件已经能自动记录 method/path/status/耗时/IP/用户,
|
||
// 但「改了哪个部署」「保留天数 7 → 3」这类信息只有 handler 自己知道,
|
||
// 所以提供 SetAudit 让 handler 补一刀,不补也能记全(只是 Detail 为空)。
|
||
type AuditInfo struct {
|
||
Module string
|
||
Action string
|
||
Resource string
|
||
ResourceID string
|
||
Detail string
|
||
Before string
|
||
After string
|
||
// Skip 置 true 表示这条请求不记(例如纯查询用了 POST 的场景)
|
||
Skip bool
|
||
}
|
||
|
||
const auditInfoKey = "audit.info"
|
||
|
||
// SetAudit 由 controller 调用,补充审计语义。
|
||
func SetAudit(c *gin.Context, info AuditInfo) {
|
||
c.Set(auditInfoKey, info)
|
||
}
|
||
|
||
// auditSkipPaths 这些路径不记审计:
|
||
// - 健康检查 / metrics:高频噪音
|
||
// - 日志查询:GET 本来就不记,但 /logs/query 之类用 POST 的也不记(只读)
|
||
var auditSkipPrefixes = []string{
|
||
"/api/v1/health",
|
||
"/api/v1/metrics",
|
||
"/api/v1/logs/query",
|
||
"/api/v1/logs/stats",
|
||
"/api/v1/logs/histogram",
|
||
"/api/v1/logs/labels",
|
||
"/api/v1/logs/label",
|
||
"/api/v1/logs/context",
|
||
"/api/v1/logs/volume",
|
||
"/api/v1/logs/my-scope",
|
||
"/api/v1/deployments/", // 例外:其中 /deploy /stop /restart 要记,见下方白名单
|
||
}
|
||
|
||
// auditAlwaysRecord 匹配到的路径即使在上面的 skip 前缀里也要记。
|
||
var auditAlwaysRecordSuffixes = []string{
|
||
"/deploy", "/stop", "/restart", "/install", "/uninstall", "/test", "/validate",
|
||
"/preview", "/rollback", "/reset-password", "/backup", "/restore",
|
||
}
|
||
|
||
// AuditLog 审计中间件:自动记录所有写操作(POST / PUT / PATCH / DELETE)。
|
||
//
|
||
// 为什么用中间件而不是每个 controller 各写一段:
|
||
// - LokiStack 有 60+ 个写接口,逐个埋点必然漏,漏掉的那个恰好最危险(删集群、改存储)
|
||
// - 中间件能拿到统一的 status / 耗时 / IP / UA
|
||
// - handler 需要更细的语义时再 SetAudit 补,不补也有基础记录
|
||
//
|
||
// 不含请求体(读 body 会影响后续 binding),敏感字段由 handler 通过 SetAudit
|
||
// 传入已脱敏的 Before/After。
|
||
func AuditLog() gin.HandlerFunc {
|
||
return func(c *gin.Context) {
|
||
method := c.Request.Method
|
||
// 只读请求不记(GET/HEAD/OPTIONS)
|
||
if method == http.MethodGet || method == http.MethodHead || method == http.MethodOptions {
|
||
c.Next()
|
||
return
|
||
}
|
||
|
||
path := c.Request.URL.Path
|
||
if !shouldAudit(path) {
|
||
c.Next()
|
||
return
|
||
}
|
||
|
||
start := time.Now()
|
||
c.Next()
|
||
elapsed := time.Since(start).Milliseconds()
|
||
|
||
info, _ := c.Get(auditInfoKey)
|
||
ai, _ := info.(AuditInfo)
|
||
if ai.Skip {
|
||
return
|
||
}
|
||
|
||
status := c.Writer.Status()
|
||
userID, _ := c.Get("userId")
|
||
username, _ := c.Get("username")
|
||
role, _ := c.Get("role")
|
||
|
||
entry := models.AuditLog{
|
||
IP: c.ClientIP(),
|
||
UserAgent: c.Request.UserAgent(),
|
||
Method: method,
|
||
Path: path,
|
||
Status: status,
|
||
DurationMs: elapsed,
|
||
Detail: ai.Detail,
|
||
Before: ai.Before,
|
||
After: ai.After,
|
||
CreatedAt: time.Now(),
|
||
}
|
||
if v, ok := userID.(uint); ok {
|
||
entry.UserID = v
|
||
}
|
||
if v, ok := username.(string); ok {
|
||
entry.Username = v
|
||
}
|
||
if v, ok := role.(string); ok {
|
||
entry.Role = v
|
||
}
|
||
|
||
// 失败时把错误信息带上(gin 的 Error 里有 handler 写入的错误)
|
||
if status >= 400 {
|
||
if len(c.Errors) > 0 {
|
||
entry.Error = c.Errors.String()
|
||
}
|
||
}
|
||
|
||
// handler 没补语义时按路径推断,保证「谁删了什么」至少能看出模块
|
||
module, action, resource, resourceID := inferAuditFields(method, path)
|
||
if ai.Module != "" {
|
||
module = ai.Module
|
||
}
|
||
if ai.Action != "" {
|
||
action = ai.Action
|
||
}
|
||
if ai.Resource != "" {
|
||
resource = ai.Resource
|
||
}
|
||
if ai.ResourceID != "" {
|
||
resourceID = ai.ResourceID
|
||
}
|
||
entry.Module, entry.Action, entry.Resource, entry.ResourceID = module, action, resource, resourceID
|
||
|
||
services.RecordAudit(entry)
|
||
}
|
||
}
|
||
|
||
func shouldAudit(path string) bool {
|
||
for _, suffix := range auditAlwaysRecordSuffixes {
|
||
if strings.HasSuffix(path, suffix) {
|
||
return true
|
||
}
|
||
}
|
||
for _, p := range auditSkipPrefixes {
|
||
if strings.HasPrefix(path, p) {
|
||
return false
|
||
}
|
||
}
|
||
return true
|
||
}
|
||
|
||
// inferAuditFields 从 method + path 推断模块与动作(handler 未补充时的兜底)。
|
||
func inferAuditFields(method, path string) (module, action, resource, resourceID string) {
|
||
action = map[string]string{
|
||
http.MethodPost: "create",
|
||
http.MethodPut: "update",
|
||
http.MethodPatch: "update",
|
||
http.MethodDelete: "delete",
|
||
}[method]
|
||
|
||
segments := strings.Split(strings.Trim(path, "/"), "/")
|
||
// /api/v1/<module>/<resource>/<id>
|
||
if len(segments) >= 3 {
|
||
module = segments[2]
|
||
}
|
||
if len(segments) >= 4 {
|
||
resource = segments[3]
|
||
resourceID = segments[4]
|
||
}
|
||
|
||
// 语义化覆盖
|
||
switch {
|
||
case strings.HasSuffix(path, "/deploy"):
|
||
action, resource = "deploy", "部署"
|
||
case strings.HasSuffix(path, "/stop"):
|
||
action, resource = "stop", "部署"
|
||
case strings.HasSuffix(path, "/restart"):
|
||
action, resource = "restart", "部署"
|
||
case strings.HasSuffix(path, "/install"):
|
||
action, resource = "install", "采集器"
|
||
case strings.HasSuffix(path, "/uninstall"):
|
||
action, resource = "uninstall", "采集器"
|
||
case strings.HasSuffix(path, "/test"):
|
||
action = "test"
|
||
case strings.HasSuffix(path, "/validate"):
|
||
action = "validate"
|
||
case strings.HasSuffix(path, "/reset-password"):
|
||
action, resource = "reset-password", "用户"
|
||
case strings.HasSuffix(path, "/login"):
|
||
module, action, resource = "auth", "login", "登录"
|
||
case strings.HasSuffix(path, "/password"):
|
||
module, action, resource = "auth", "change-password", "登录"
|
||
}
|
||
return module, action, resource, resourceID
|
||
}
|