feat(alerts,logs): A 方案—告警窗口 15m→5m 消重复扫描;数据源可编辑/删除且删后不被自动重建
=== 1. 告警窗口 15m → 5m(阈值同比例 1/3)===
背景:检查间隔已改 5 分钟,但规则窗口仍是 15 分钟 ——
同一个 15 分钟窗口每 5 分钟被完整重扫一遍,等于 3 倍重复扫描。
在 370GB/天、全命名空间 selector 下,这是 Loki CPU 的最大稳态来源。
FATAL 10 → 4 窗口 5m
Go panic 10 → 4 窗口 5m
JVM OOM 5 → 2 窗口 5m
StackOverflow 5 → 2 窗口 5m
应用错误激增 100000 → 35000 窗口 5m(改名 error > 3.5万)
ingress 5xx 2000 → 700 窗口 5m
ingress 4xx 20000 → 7000 窗口 5m
DB 连接失败 1000 → 350 窗口 5m
阈值按 1/3 缩放是必须的:窗口缩短 3 倍后,同样的流量在 5 分钟内只有
1/3 的条数,阈值不动就永远打不到了。
迁移 v10→v11 只改"窗口仍是 15 且名字匹配内置规则"的项,用户手调的不动。
=== 2. 数据源支持编辑/删除(含托管源)===
原来前后端都把托管源锁死:
- 前端 `!managed &&` 直接不渲染编辑/删除按钮
- 后端 DeleteSource 返回「平台部署的数据源随部署管理,请先删除对应部署」
用户的诉求是"数据源必须能编辑和删除",且确实有场景(不想让某个源出现在
查询下拉里、想改认证头)。
=== 3. 修「删了又回来」的真 bug ===
EnsureLogSourcesForRunningDeployments() 在 **每次打开数据源列表和启动时**
都会给 running 部署补齐数据源 —— 所以删掉托管源后,刷新页面它又回来了,
用户看到的现象就是"删不掉"。
新增 LokiDeployment.AutoSourceDisabled:
- 删除托管源时置位(DisableAutoSourceFor)
- EnsureLogSourceForDeployment 见到该开关直接 return
- 部署被编辑/重新部署时重置(EnableAutoSourceFor),即"你要用它了就恢复"
前端配套文案:
- 编辑托管源弹窗提示"重新部署会覆盖集群/Service/端口,认证与请求头保留"
- 删除确认明确"删除后不会再自动重建,重新部署该 Loki 才恢复"
=== 实测(20300eef92d2)===
v11 迁移后 8 条规则全部 窗口=5m,阈值 4/4/2/2/35000/700/7000/350
go test ./... 全绿
This commit is contained in:
1 parent
c9e604b35f
commit
ad0d35f72a
9 files changed
+156
-41
No files matched your search
@@ -44,6 +44,8 @@ export default {
|
||||
testing: 'Testing…',
|
||||
deleteTitle: 'Delete data source',
|
||||
deleteDesc: 'Deleting "{{name}}" breaks queries and access policies based on it. Delete it?',
|
||||
deleteDescManaged: '"{{name}}" was created automatically by a deployment. The platform will NOT recreate it (access policies are removed too) until you redeploy or edit that Loki deployment.',
|
||||
managedEditHint: 'This source is created by Component Deploy: redeploying overwrites the cluster/service/port connection details; auth and header changes you make here are kept.',
|
||||
deleteAria: 'Delete {{name}}',
|
||||
defaultBadge: 'Default',
|
||||
emptyTitle: 'No data sources yet',
|
||||
|
||||
@@ -43,6 +43,8 @@ export default {
|
||||
testing: '测试中…',
|
||||
deleteTitle: '删除数据源',
|
||||
deleteDesc: '删除「{{name}}」后,基于它的查询与授权策略将失效,确定删除吗?',
|
||||
deleteDescManaged: '「{{name}}」由部署自动创建。删除后平台不会再自动重建它(授权策略一并失效);重新部署或编辑该 Loki 部署才会恢复。',
|
||||
managedEditHint: '该数据源由「组件部署」自动创建:重新部署会覆盖集群/Service/端口等连接信息,你在此修改的认证方式与请求头会保留。',
|
||||
deleteAria: '删除 {{name}}',
|
||||
defaultBadge: '默认',
|
||||
emptyTitle: '还没有数据源',
|
||||
|
||||
@@ -127,23 +127,22 @@ export const SourcesManager = memo(function SourcesManager({ sources, onSaved }:
|
||||
{testing ? <Loader2 className="mr-1 size-3.5 animate-spin" /> : <Zap className="mr-1 size-3.5" />}
|
||||
{testing ? t('source.testing') : t('source.test')}
|
||||
</Button>
|
||||
{!managed && (
|
||||
<Button variant="outline" size="sm" onClick={() => setForm({ ...source, headers: Object.fromEntries((source.headerNames ?? []).map(name => [name, ''])) })}>
|
||||
<Pencil className="mr-1 size-3.5" />
|
||||
{t('source.edit')}
|
||||
</Button>
|
||||
)}
|
||||
{!managed && (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="text-destructive hover:bg-destructive/10"
|
||||
aria-label={t('source.deleteAria', { name: source.name })}
|
||||
onClick={() => setDeleteTarget(source)}
|
||||
>
|
||||
<Trash2 className="size-3.5" />
|
||||
</Button>
|
||||
)}
|
||||
{/* 托管源同样允许编辑/删除:
|
||||
托管只代表"由部署自动创建",不代表用户不能改连接参数或不想让它出现在下拉里。
|
||||
删除托管源时后端会把该部署标记为 AutoSourceDisabled,不会立刻重建。 */}
|
||||
<Button variant="outline" size="sm" onClick={() => setForm({ ...source, headers: Object.fromEntries((source.headerNames ?? []).map(name => [name, ''])) })}>
|
||||
<Pencil className="mr-1 size-3.5" />
|
||||
{t('source.edit')}
|
||||
</Button>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="text-destructive hover:bg-destructive/10"
|
||||
aria-label={t('source.deleteAria', { name: source.name })}
|
||||
onClick={() => setDeleteTarget(source)}
|
||||
>
|
||||
<Trash2 className="size-3.5" />
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
@@ -159,6 +158,11 @@ export const SourcesManager = memo(function SourcesManager({ sources, onSaved }:
|
||||
<DialogHeader>
|
||||
<DialogTitle>{form?.id ? `${t('source.editTitle')} · ${form.name}` : t('source.createTitle')}</DialogTitle>
|
||||
</DialogHeader>
|
||||
{form?.id && form.managed && (
|
||||
<p className="rounded-md border border-warning/40 bg-warning/10 px-3 py-2 text-xs text-warning">
|
||||
{t('source.managedEditHint')}
|
||||
</p>
|
||||
)}
|
||||
{form && (
|
||||
<div className="grid gap-4 sm:grid-cols-2">
|
||||
<label className={labelClass}>
|
||||
@@ -293,7 +297,9 @@ export const SourcesManager = memo(function SourcesManager({ sources, onSaved }:
|
||||
onOpenChange={open => { if (!open) setDeleteTarget(null) }}
|
||||
name={deleteTarget?.name}
|
||||
title={t('source.deleteTitle')}
|
||||
description={t('source.deleteDesc', { name: deleteTarget?.name ?? '' })}
|
||||
description={deleteTarget?.managed
|
||||
? t('source.deleteDescManaged', { name: deleteTarget.name })
|
||||
: t('source.deleteDesc', { name: deleteTarget?.name ?? '' })}
|
||||
loading={saving}
|
||||
onConfirm={() => void remove()}
|
||||
/>
|
||||
|
||||
Reference in new issue
Block a user