feat(alerts,logs): A 方案—告警窗口 15m→5m 消重复扫描;数据源可编辑/删除且删后不被自动重建
=== 1. 告警窗口 15m → 5m(阈值同比例 1/3)===
背景:检查间隔已改 5 分钟,但规则窗口仍是 15 分钟 ——
同一个 15 分钟窗口每 5 分钟被完整重扫一遍,等于 3 倍重复扫描。
在 370GB/天、全命名空间 selector 下,这是 Loki CPU 的最大稳态来源。
FATAL 10 → 4 窗口 5m
Go panic 10 → 4 窗口 5m
JVM OOM 5 → 2 窗口 5m
StackOverflow 5 → 2 窗口 5m
应用错误激增 100000 → 35000 窗口 5m(改名 error > 3.5万)
ingress 5xx 2000 → 700 窗口 5m
ingress 4xx 20000 → 7000 窗口 5m
DB 连接失败 1000 → 350 窗口 5m
阈值按 1/3 缩放是必须的:窗口缩短 3 倍后,同样的流量在 5 分钟内只有
1/3 的条数,阈值不动就永远打不到了。
迁移 v10→v11 只改"窗口仍是 15 且名字匹配内置规则"的项,用户手调的不动。
=== 2. 数据源支持编辑/删除(含托管源)===
原来前后端都把托管源锁死:
- 前端 `!managed &&` 直接不渲染编辑/删除按钮
- 后端 DeleteSource 返回「平台部署的数据源随部署管理,请先删除对应部署」
用户的诉求是"数据源必须能编辑和删除",且确实有场景(不想让某个源出现在
查询下拉里、想改认证头)。
=== 3. 修「删了又回来」的真 bug ===
EnsureLogSourcesForRunningDeployments() 在 **每次打开数据源列表和启动时**
都会给 running 部署补齐数据源 —— 所以删掉托管源后,刷新页面它又回来了,
用户看到的现象就是"删不掉"。
新增 LokiDeployment.AutoSourceDisabled:
- 删除托管源时置位(DisableAutoSourceFor)
- EnsureLogSourceForDeployment 见到该开关直接 return
- 部署被编辑/重新部署时重置(EnableAutoSourceFor),即"你要用它了就恢复"
前端配套文案:
- 编辑托管源弹窗提示"重新部署会覆盖集群/Service/端口,认证与请求头保留"
- 删除确认明确"删除后不会再自动重建,重新部署该 Loki 才恢复"
=== 实测(20300eef92d2)===
v11 迁移后 8 条规则全部 窗口=5m,阈值 4/4/2/2/35000/700/7000/350
go test ./... 全绿
This commit is contained in:
1 parent
c9e604b35f
commit
ad0d35f72a
9 files changed
+156
-41
No files matched your search
@@ -72,10 +72,10 @@ var builtinRules = []builtinRule{
|
||||
// —— 致命类:这类日志本该极少出现,但也不能一条就报 ——
|
||||
// 全命名空间类查询一律排除 app=loki:Loki 会把执行过的查询语句记成自身 info 日志,
|
||||
// 查询文本包含规则关键字时告警会"命中自己",造成持续误报。
|
||||
{name: "致命日志 FATAL", query: `{namespace=~".+", app!="loki"} |= "FATAL"`, windowMin: 15, threshold: 10, cooldown: 120},
|
||||
{name: "Go panic", query: `{namespace=~".+", app!="loki"} |= "panic:"`, windowMin: 15, threshold: 10, cooldown: 120},
|
||||
{name: "JVM OutOfMemoryError", query: `{namespace=~".+", app!="loki"} |~ "(^|[^a-zA-Z])OutOfMemoryError"`, windowMin: 15, threshold: 5, cooldown: 120},
|
||||
{name: "StackOverflowError", query: `{namespace=~".+", app!="loki"} |= "StackOverflowError"`, windowMin: 15, threshold: 5, cooldown: 120},
|
||||
{name: "致命日志 FATAL", query: `{namespace=~".+", app!="loki"} |= "FATAL"`, windowMin: 5, threshold: 4, cooldown: 120},
|
||||
{name: "Go panic", query: `{namespace=~".+", app!="loki"} |= "panic:"`, windowMin: 5, threshold: 4, cooldown: 120},
|
||||
{name: "JVM OutOfMemoryError", query: `{namespace=~".+", app!="loki"} |~ "(^|[^a-zA-Z])OutOfMemoryError"`, windowMin: 5, threshold: 2, cooldown: 120},
|
||||
{name: "StackOverflowError", query: `{namespace=~".+", app!="loki"} |= "StackOverflowError"`, windowMin: 5, threshold: 2, cooldown: 120},
|
||||
// 注意:这里**只放日志类告警**。
|
||||
// 「节点 NotReady」「容器 OOMKilled」属于基础设施指标监控(kube-state-metrics /
|
||||
// Prometheus)的职责,用日志去匹配这些字符串既不可靠(日志没开或轮转就漏),
|
||||
@@ -84,12 +84,12 @@ var builtinRules = []builtinRule{
|
||||
// 第二轮上调(500~5000 这一档实测仍然太敏感):
|
||||
// 这几个量级在业务高峰期本来就是常态,阈值贴着基线就等于"每天准点报警"。
|
||||
// 原则:宁可只报"明显异常",也不要让人对告警脱敏。
|
||||
{name: "应用错误激增(error > 10万)", query: `{namespace=~".+", app!="loki"} |~ "(?i)\berror\b"`, windowMin: 15, threshold: 100000, cooldown: 240},
|
||||
{name: "ingress-nginx 5xx 错误", query: `{app=~"ingress-nginx|nginx-ingress-controller"} |~ " (50[0-4]|5[2-9][0-9]) "`, windowMin: 15, threshold: 2000, cooldown: 240},
|
||||
{name: "ingress-nginx 4xx 激增", query: `{app=~"ingress-nginx|nginx-ingress-controller"} |~ " 4[0-9][0-9] "`, windowMin: 15, threshold: 20000, cooldown: 240},
|
||||
{name: "应用错误激增(error > 3.5万)", query: `{namespace=~".+", app!="loki"} |~ "(?i)\berror\b"`, windowMin: 5, threshold: 35000, cooldown: 240},
|
||||
{name: "ingress-nginx 5xx 错误", query: `{app=~"ingress-nginx|nginx-ingress-controller"} |~ " (50[0-4]|5[2-9][0-9]) "`, windowMin: 5, threshold: 700, cooldown: 240},
|
||||
{name: "ingress-nginx 4xx 激增", query: `{app=~"ingress-nginx|nginx-ingress-controller"} |~ " 4[0-9][0-9] "`, windowMin: 5, threshold: 7000, cooldown: 240},
|
||||
// 数据库连接类错误:覆盖 MySQL / PostgreSQL / Redis 的主流报错文案
|
||||
// (详见函数上方注释,不要加裸 "EOF" / "timeout",那会匹配到几乎所有网络日志)
|
||||
{name: "数据库连接失败", query: `{namespace=~".+", app!="loki"} |~ "(?i)(connection refused|connection reset by peer|broken pipe|no route to host|cannot assign requested address|connection timed out|i/o timeout|too many connections|too many clients already|max number of clients reached|connection pool|connection is not available|could not create connection|communications link failure|server closed the connection unexpectedly|unexpected EOF|driver: bad connection|FATAL:\\s*(password authentication failed|sorry, too many clients|terminating connection)|NOAUTH|WRONGPASS|READONLY you can.t write|CLUSTERDOWN)"`, windowMin: 15, threshold: 1000, cooldown: 240},
|
||||
{name: "数据库连接失败", query: `{namespace=~".+", app!="loki"} |~ "(?i)(connection refused|connection reset by peer|broken pipe|no route to host|cannot assign requested address|connection timed out|i/o timeout|too many connections|too many clients already|max number of clients reached|connection pool|connection is not available|could not create connection|communications link failure|server closed the connection unexpectedly|unexpected EOF|driver: bad connection|FATAL:\\s*(password authentication failed|sorry, too many clients|terminating connection)|NOAUTH|WRONGPASS|READONLY you can.t write|CLUSTERDOWN)"`, windowMin: 5, threshold: 350, cooldown: 240},
|
||||
}
|
||||
|
||||
// seedBuiltinAlertRules 为每个平台管理的 LogSource(managed=true)补齐内置规则:
|
||||
|
||||
Reference in new issue
Block a user