feat(collectors): 可编辑采集配置(B 档结构化编辑器)+ 修 host 标签 / syslog 时间戳

=== 一、两个线上 bug(172.19.16.8 真机取证)===

1) host 标签是字面量 ${HOSTNAME}
   Promtail 不做环境变量展开(除非 -config.expand-env + systemd Environment),
   实测 21,825 条日志的 host 全是 "${HOSTNAME}",无法按主机过滤。
   → 安装时 `hostname -s` 探测真实主机名并渲染进配置,结果存 Collector.ResolvedHostname,
     让「预览配置」与线上一致。

2) 系统日志时间戳解析失效
   TencentOS 4 的 /var/log/messages 是 ISO8601(2026-07-17T15:38:24.923784+08:00),
   旧正则按传统 syslog(Oct 28 18:11:06)写死 → timestamp 阶段匹配不上,用的是采集时间,
   回填历史日志时间全错。
   → 安装时探测目标机格式:iso8601 → RFC3339Nano;rfc3164 → Jan _2 15:04:05 + Local;
     系统日志路径同时探测(Ubuntu 没有 messages,写死会一条都采不到)。

   实测:写一条 10 分钟前的 ISO8601 探针行,Loki 里 loki_ts 与该行自身时间偏差 **0 秒**;
   另写一条 2 小时前的被 Loki 以 entry too far behind 拒绝 —— 恰好反证时间戳是按事件时间
   解析后推送的(旧实现会用采集时间,反而被接收)。

=== 二、B 档:结构化多 Job 编辑器 ===

后端
- models/scrape_job.go:ScrapeJob + ValidateScrapeJobs
  校验 job 名格式、绝对路径、**敏感路径拦截**(/etc/shadow、/etc/sudoers、/root/.ssh、
  id_rsa、*.pem…)、正则可编译、job/host/__path__ 保留标签名、数量上限。
- services/promtail_host_config.go:多 job 渲染;多路径展开成多个 static_configs;
  pipeline 顺序硬约束 cri → multiline → json → timestamp → drop → limit;
  encoding 写在 scrape_config 级(放进 static_configs 里 promtail 直接起不来)。
- services/host_profile.go:探测 distro / init / hostname / 时区 / 架构 / journald
  与真实存在的系统日志路径。
- CollectorService:renderConfig 实时渲染、PreviewConfig(只渲染不落库)、
  Update 支持 scrapeJobs 并校验、安装结果(hostname/syslog 格式/配置快照)落库。
- 新增 POST /api/v1/collectors/:id/preview → {config, warnings, hostname, syslogFormat}。
- installHost:配置先写 .new + `promtail -check-syntax` 通过才替换,重启后轮询
  systemctl is-active,失败把 journalctl 尾部带回前端。

前端
- 采集项结构化编辑器:名称/路径/标签常驻,多行合并 / JSON / 自定义时间戳 / 命中丢弃 /
  限流折叠在「高级」里;「预览 YAML」弹窗在保存前看到将要下发的配置。
- 本地即时校验 + 旧数据(只有 logPath)自动迁移成 app + system 两个采集项。

=== 三、验证(host-verify-promtail @172.19.16.8)===
- 预览:host=pre-cex-ops-jenkins(无 ${HOSTNAME}),2 job / 3 条系统日志路径 /
  ISO8601 正则 / 多行 + drop + 限流 全部按填写渲染。
- 部署:promtail active、NRestarts=0、-check-syntax 通过,主机侧配置与预览逐行一致。
- Loki:新数据 host 全为 pre-cex-ops-jenkins;job=system 415 条/5min、job=app 40 条/5min。
- 负向 7/7 返回指名到字段的 400:/etc/shadow、/root/.ssh/id_rsa、相对路径、坏正则、
  job 名带空格、保留标签名、空路径。

镜像:lghost/lokistack:latest + v0.1 = sha256:26172f51187d
This commit is contained in:
opsx committed 2026-10-28 18:41:39 +00:00
1 parent 55d6a74eb3
commit c983d0e3c1
18 files changed
+1678 -254

No files matched your search

+19 -1
View File
@@ -1,6 +1,15 @@
// 采集器域 API
import { http, buildQuery, type ListParams, type RequestOptions } from '../http';
import type { PageData, Collector, CreateCollectorRequest } from '@/types';
import type { PageData, Collector, CreateCollectorRequest, ScrapeJob } from '@/types';
export interface CollectorPreview {
/** 渲染出来的 promtail.yaml */
config: string;
/** 平台给出的提示(未安装过 → 主机名/syslog 格式待探测等) */
warnings?: string[];
hostname?: string;
syslogFormat?: string;
}
export const collectorApi = {
list: (params: ListParams = {}, options?: RequestOptions) =>
@@ -16,9 +25,18 @@ export const collectorApi = {
targetNamespaces?: string[];
appLabelSource?: string;
appInclude?: string[];
/** 主机模式的结构化采集项(B 档) */
scrapeJobs?: ScrapeJob[];
}) =>
http.put<{ message: string }>(`/collectors/${id}`, data),
/** 只预览渲染结果,不落库(结构化编辑器的「预览 YAML」用) */
preview: (id: number, data: {
logPath?: string;
labels?: string;
scrapeJobs?: ScrapeJob[];
}) => http.post<CollectorPreview>(`/collectors/${id}/preview`, data),
remove: (id: number) => http.delete<{ message: string }>(`/collectors/${id}`),
install: (id: number) => http.post<{ message: string }>(`/collectors/${id}/install`),
@@ -120,6 +120,41 @@ const collectors = {
appIncludeHint:
'Keeps only logs whose app matches these values (<code>action: keep</code>) — the most direct way to cap log volume.',
logPathHint: 'Sidecar needs a shared log volume or the dedicated logs directory; click Deploy after saving.',
// ── host mode: structured scrape jobs ──
jobs: {
title: 'Log scrape jobs',
hint: 'One entry = one promtail job; add more paths for more directories. Click Deploy after saving.',
add: 'Add job',
remove: 'Remove job',
namePlaceholder: 'job name, e.g. app-api',
paths: 'Log paths (comma separated, glob supported)',
systemPathsPlaceholder: 'Empty = use the system logs that actually exist on this host',
systemBadge: 'system logs · auto timestamp',
labels: 'Extra labels (key=value, comma separated)',
advanced: 'Advanced: multiline / JSON / timestamp / drop / rate limit',
system: 'System logs (auto ISO8601 / classic)',
json: 'Parse JSON',
multiline: 'Multiline (Java stack traces)',
timestamp: 'Custom timestamp',
maxLinesPlaceholder: 'max lines 128',
tsSourcePlaceholder: 'field name, e.g. time',
tsLocationPlaceholder: 'location, e.g. Asia/Shanghai',
dropIfMatch: 'Drop when matching (regex, comma separated)',
dropIfMatchPlaceholder: 'Empty = keep all; e.g. "status":200',
ratePlaceholder: 'rate limit (default 2000)',
burstPlaceholder: 'burst limit (default 4000)',
hostInfo: 'Detected on last install: hostname {{host}}, syslog format {{format}}',
},
preview: {
button: 'Preview YAML',
loading: 'Rendering…',
failed: 'Preview failed',
title: 'promtail.yaml that will be written',
description: 'Preview only — nothing is saved or deployed. Then click Save → Deploy.',
copy: 'Copy',
copied: 'Copied',
close: 'Close',
},
save: 'Save',
saving: 'Saving…',
validation: {
@@ -124,6 +124,41 @@ const collectors = {
appIncludeHint:
'只保留 app 命中这些值的日志(<code>action: keep</code>),是控制日志量最直接的手段。',
logPathHint: 'Sidecar 需共享日志卷或专用 logs 目录;保存后点「部署」生效。',
// ── 主机模式:结构化采集项(B 档) ──
jobs: {
title: '日志采集项',
hint: '一个采集项 = promtail 的一个 job;多个目录就多填几个路径。保存后点「部署」下发。',
add: '添加采集项',
remove: '删除采集项',
namePlaceholder: 'job 名称,如 app-api',
paths: '日志路径(逗号分隔,支持 glob)',
systemPathsPlaceholder: '留空 = 自动使用该主机实际存在的系统日志',
systemBadge: '系统日志 · 自动时间戳',
labels: '附加标签(key=value,逗号分隔)',
advanced: '高级:多行合并 / JSON / 时间戳 / 丢弃 / 限流',
system: '系统日志(自动选 ISO8601/传统格式)',
json: 'JSON 解析',
multiline: '多行合并(Java 堆栈)',
timestamp: '自定义时间戳',
maxLinesPlaceholder: '最大行数 128',
tsSourcePlaceholder: '字段名,如 time',
tsLocationPlaceholder: '时区,如 Asia/Shanghai',
dropIfMatch: '命中即丢弃(正则,逗号分隔)',
dropIfMatchPlaceholder: '留空不丢;如 "status":200',
ratePlaceholder: '限流 rate(默认 2000)',
burstPlaceholder: '限流 burst(默认 4000)',
hostInfo: '上次安装探测:主机名 {{host}},系统日志格式 {{format}}',
},
preview: {
button: '预览 YAML',
loading: '渲染中…',
failed: '预览失败',
title: '将要下发的 promtail.yaml',
description: '只预览,不保存也不下发;确认无误再点「保存」→「部署」。',
copy: '复制',
copied: '已复制',
close: '关闭',
},
save: '保存',
saving: '保存中…',
validation: {
@@ -8,12 +8,15 @@ import type { TFunction } from 'i18next'
import '@/i18n/locales/zh/collectors'
import '@/i18n/locales/en/collectors'
import { toast } from 'sonner'
import { clusterApi, collectorApi } from '@/api'
import { Eye } from 'lucide-react'
import { clusterApi, collectorApi, type CollectorPreview } from '@/api'
import { Button } from '@/components/ui/button'
import { Dialog, DialogContent, DialogHeader, DialogTitle } from '@/components/ui/dialog'
import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle } from '@/components/ui/dialog'
import { Form, FormControl, FormField, FormItem, FormLabel, FormMessage } from '@/components/ui/form'
import { Input } from '@/components/ui/input'
import { Checkbox } from '@/components/ui/checkbox'
import { ScrapeJobsEditor } from './ScrapeJobsEditor'
import { draftIssues, draftToJob, draftsFromCollector, type JobDraft } from './scrape-jobs'
import type { Collector } from '@/types'
// 表单 schema:错误消息经 t 注入(随语言重建,RHF 每次渲染刷新 resolver)
@@ -53,6 +56,10 @@ export const CollectorEditDialog = memo(function CollectorEditDialog({ collector
const [nsLoading, setNsLoading] = useState(false)
const [nsError, setNsError] = useState('')
const [nsFilter, setNsFilter] = useState('')
/** 主机模式的采集项草稿(结构化编辑器,B 档) */
const [jobs, setJobs] = useState<JobDraft[]>([])
const [preview, setPreview] = useState<CollectorPreview | null>(null)
const [previewing, setPreviewing] = useState(false)
// schema 随语言重建(t 变化即语言切换),校验消息即时跟随当前语言
const schema = useMemo(() => buildSchema(t), [t])
@@ -62,7 +69,8 @@ export const CollectorEditDialog = memo(function CollectorEditDialog({ collector
})
const isDaemonSet = collector?.type === 'k8s-daemonset'
const supportsPath = !isDaemonSet
const isHost = collector?.type === 'host-promtail'
const supportsPath = !isDaemonSet && !isHost
const selectedNs = form.watch('targetNamespaces')
useEffect(() => {
@@ -75,6 +83,8 @@ export const CollectorEditDialog = memo(function CollectorEditDialog({ collector
appIncludeRaw: (collector.appInclude ?? []).join(', '),
})
setNsFilter('')
setPreview(null)
setJobs(collector.type === 'host-promtail' ? draftsFromCollector(collector.scrapeJobs, collector.logPath) : [])
}, [collector, form])
// 拉集群命名空间列表(供多选)
@@ -108,8 +118,40 @@ export const CollectorEditDialog = memo(function CollectorEditDialog({ collector
form.setValue('targetNamespaces', [], { shouldDirty: true })
}
function jobPayload() {
return jobs.map(draftToJob).filter((j) => j.name || j.paths.length > 0)
}
const runPreview = async () => {
if (!collector) return
const issues = draftIssues(jobs)
if (issues.length > 0) {
toast.error(issues[0])
return
}
setPreviewing(true)
try {
const data = await collectorApi.preview(collector.id, {
logPath: form.getValues('logPath'),
scrapeJobs: jobPayload(),
})
setPreview(data)
} catch (err) {
toast.error(err instanceof Error ? err.message : t('edit.preview.failed'))
} finally {
setPreviewing(false)
}
}
const submit = form.handleSubmit(async (values) => {
if (!collector) return
if (isHost) {
const issues = draftIssues(jobs)
if (issues.length > 0) {
toast.error(issues[0])
return
}
}
setSaving(true)
try {
await collectorApi.update(collector.id, {
@@ -121,6 +163,7 @@ export const CollectorEditDialog = memo(function CollectorEditDialog({ collector
.split(/[,\s]+/)
.map((s) => s.trim())
.filter(Boolean),
...(isHost ? { scrapeJobs: jobPayload() } : {}),
})
onOpenChange(false)
onSaved()
@@ -132,125 +175,184 @@ export const CollectorEditDialog = memo(function CollectorEditDialog({ collector
})
return (
<Dialog open={collector !== null} onOpenChange={onOpenChange}>
<DialogContent className="sm:max-w-2xl max-h-[90vh] overflow-y-auto">
<DialogHeader><DialogTitle>{t('edit.title')}</DialogTitle></DialogHeader>
<Form {...form}>
<form onSubmit={submit} className="space-y-4">
<FormField control={form.control} name="name" render={({ field }) => (
<FormItem><FormLabel>{t('form.name')}</FormLabel><FormControl><Input {...field} /></FormControl><FormMessage /></FormItem>
)} />
<>
<Dialog open={collector !== null} onOpenChange={onOpenChange}>
<DialogContent className="sm:max-w-2xl max-h-[90vh] overflow-y-auto">
<DialogHeader><DialogTitle>{t('edit.title')}</DialogTitle></DialogHeader>
<Form {...form}>
<form onSubmit={submit} className="space-y-4">
<FormField control={form.control} name="name" render={({ field }) => (
<FormItem><FormLabel>{t('form.name')}</FormLabel><FormControl><Input {...field} /></FormControl><FormMessage /></FormItem>
)} />
{isDaemonSet && (
<>
{/* ── 采集命名空间:多选 ── */}
{/* ── 主机模式:结构化采集项(多 job / 多路径 / 多行 / 丢弃 / 限流) ── */}
{isHost && (
<div className="space-y-2">
<div className="flex items-center justify-between">
<FormLabel>{t('edit.namespaces')}</FormLabel>
<span className="text-xs text-muted-foreground">
{allSelected ? t('edit.allNamespaces') : t('edit.selectedCount', { count: selectedNs.length })}
</span>
<FormLabel>{t('edit.jobs.title')}</FormLabel>
<Button type="button" size="sm" variant="outline" className="h-7" onClick={runPreview} disabled={previewing}>
<Eye className="mr-1 h-3.5 w-3.5" />
{previewing ? t('edit.preview.loading') : t('edit.preview.button')}
</Button>
</div>
{nsError && <p className="text-xs text-destructive">{nsError}</p>}
<div className="rounded-md border">
<label className="flex items-center gap-2 border-b px-3 py-2 text-sm font-medium cursor-pointer hover:bg-muted/50">
<Checkbox checked={allSelected} onCheckedChange={() => setAll()} />
{t('edit.allCheckbox')}
</label>
<div className="p-2">
<Input
className="mb-2 h-8"
placeholder={t('edit.nsSearch')}
value={nsFilter}
onChange={(e) => setNsFilter(e.target.value)}
/>
<div className="max-h-44 overflow-auto grid gap-1 sm:grid-cols-2 lg:grid-cols-3">
{nsLoading && <p className="p-2 text-xs text-muted-foreground">{t('edit.loading')}</p>}
{!nsLoading && filteredNs.length === 0 && (
<p className="p-2 text-xs text-muted-foreground">
{namespaces.length === 0 ? t('edit.nsListEmpty') : t('edit.nsNoMatch')}
</p>
)}
{filteredNs.map((ns) => (
<label key={ns} className="flex items-center gap-2 rounded px-2 py-1 text-sm hover:bg-muted/50 cursor-pointer">
<input
type="checkbox"
disabled={allSelected}
checked={selectedNs.includes(ns)}
onChange={() => toggleNs(ns)}
/>
<span className="truncate">{ns}</span>
</label>
))}
<ScrapeJobsEditor drafts={jobs} onChange={setJobs} />
<p className="text-xs text-muted-foreground">{t('edit.jobs.hint')}</p>
{collector?.resolvedHostname && (
<p className="text-xs text-muted-foreground">
{t('edit.jobs.hostInfo', {
host: collector.resolvedHostname,
format: collector.syslogFormat || 'rfc3164',
})}
</p>
)}
</div>
)}
{isDaemonSet && (
<>
{/* ── 采集命名空间:多选 ── */}
<div className="space-y-2">
<div className="flex items-center justify-between">
<FormLabel>{t('edit.namespaces')}</FormLabel>
<span className="text-xs text-muted-foreground">
{allSelected ? t('edit.allNamespaces') : t('edit.selectedCount', { count: selectedNs.length })}
</span>
</div>
{nsError && <p className="text-xs text-destructive">{nsError}</p>}
<div className="rounded-md border">
<label className="flex items-center gap-2 border-b px-3 py-2 text-sm font-medium cursor-pointer hover:bg-muted/50">
<Checkbox checked={allSelected} onCheckedChange={() => setAll()} />
{t('edit.allCheckbox')}
</label>
<div className="p-2">
<Input
className="mb-2 h-8"
placeholder={t('edit.nsSearch')}
value={nsFilter}
onChange={(e) => setNsFilter(e.target.value)}
/>
<div className="max-h-44 overflow-auto grid gap-1 sm:grid-cols-2 lg:grid-cols-3">
{nsLoading && <p className="p-2 text-xs text-muted-foreground">{t('edit.loading')}</p>}
{!nsLoading && filteredNs.length === 0 && (
<p className="p-2 text-xs text-muted-foreground">
{namespaces.length === 0 ? t('edit.nsListEmpty') : t('edit.nsNoMatch')}
</p>
)}
{filteredNs.map((ns) => (
<label key={ns} className="flex items-center gap-2 rounded px-2 py-1 text-sm hover:bg-muted/50 cursor-pointer">
<input
type="checkbox"
disabled={allSelected}
checked={selectedNs.includes(ns)}
onChange={() => toggleNs(ns)}
/>
<span className="truncate">{ns}</span>
</label>
))}
</div>
</div>
</div>
<p className="text-xs text-muted-foreground">
<Trans i18nKey="edit.nsHint" t={t} components={{ code: <code className="font-mono" /> }}>
勾「所有命名空间」= 采集全集群;否则只采集勾选的命名空间。
生成 Promtail 的 <code className="font-mono">kubernetes_sd_configs.namespaces</code> 白名单。
</Trans>
</p>
</div>
<p className="text-xs text-muted-foreground">
<Trans i18nKey="edit.nsHint" t={t} components={{ code: <code className="font-mono" /> }}>
勾「所有命名空间」= 采集全集群;否则只采集勾选的命名空间。
生成 Promtail 的 <code className="font-mono">kubernetes_sd_configs.namespaces</code> 白名单。
</Trans>
</p>
</div>
{/* ── app 标签来源 ── */}
<FormField control={form.control} name="appLabelSource" render={({ field }) => (
<FormItem>
<FormLabel>{t('edit.appLabelSource')}</FormLabel>
<FormControl>
<Input list="app-source-presets" placeholder={t('edit.appSourcePlaceholder')} {...field} />
</FormControl>
<datalist id="app-source-presets">
{APP_SOURCE_PRESETS.filter((p) => p.value).map((p) => (
<option key={p.value} value={p.value}>{t(p.labelKey)}</option>
))}
</datalist>
<FormMessage />
<p className="text-xs text-muted-foreground">
<Trans i18nKey="edit.appSourceHint" t={t} components={{ code: <code className="font-mono" /> }}>
默认按 <code className="font-mono">app.kubernetes.io/name → app → controller 名 → pod 名</code> 四级回退。
若集群有自己的私有规范标签(如 <code className="font-mono">k8s.kuboard.cn/name</code>),
填在这里会作为最高优先级,其余仍兜底。
</Trans>
</p>
</FormItem>
)} />
{/* ── app 标签来源 ── */}
<FormField control={form.control} name="appLabelSource" render={({ field }) => (
<FormItem>
<FormLabel>{t('edit.appLabelSource')}</FormLabel>
<FormControl>
<Input list="app-source-presets" placeholder={t('edit.appSourcePlaceholder')} {...field} />
</FormControl>
<datalist id="app-source-presets">
{APP_SOURCE_PRESETS.filter((p) => p.value).map((p) => (
<option key={p.value} value={p.value}>{t(p.labelKey)}</option>
))}
</datalist>
<FormMessage />
<p className="text-xs text-muted-foreground">
<Trans i18nKey="edit.appSourceHint" t={t} components={{ code: <code className="font-mono" /> }}>
默认按 <code className="font-mono">app.kubernetes.io/name → app → controller 名 → pod 名</code> 四级回退。
若集群有自己的私有规范标签(如 <code className="font-mono">k8s.kuboard.cn/name</code>),
填在这里会作为最高优先级,其余仍兜底。
</Trans>
</p>
</FormItem>
)} />
{/* ── app 白名单 ── */}
<FormField control={form.control} name="appIncludeRaw" render={({ field }) => (
<FormItem>
<FormLabel>{t('edit.appInclude')}</FormLabel>
<FormControl>
<Input placeholder={t('edit.appIncludePlaceholder')} {...field} />
</FormControl>
<FormMessage />
<p className="text-xs text-muted-foreground">
<Trans i18nKey="edit.appIncludeHint" t={t} components={{ code: <code className="font-mono" /> }}>
填了之后 Promtail 只保留 app 命中这些值的日志(<code className="font-mono">action: keep</code>),
是控制日志量最直接的手段。值需与上面「app 标签取值来源」解析出的 app 一致。
</Trans>
</p>
</FormItem>
)} />
</>
)}
{/* ── app 白名单 ── */}
<FormField control={form.control} name="appIncludeRaw" render={({ field }) => (
<FormItem>
<FormLabel>{t('edit.appInclude')}</FormLabel>
<FormControl>
<Input placeholder={t('edit.appIncludePlaceholder')} {...field} />
</FormControl>
<FormMessage />
<p className="text-xs text-muted-foreground">
<Trans i18nKey="edit.appIncludeHint" t={t} components={{ code: <code className="font-mono" /> }}>
填了之后 Promtail 只保留 app 命中这些值的日志(<code className="font-mono">action: keep</code>),
是控制日志量最直接的手段。值需与上面「app 标签取值来源」解析出的 app 一致。
</Trans>
</p>
</FormItem>
)} />
</>
)}
{supportsPath && <FormField control={form.control} name="logPath" render={({ field }) => (
<FormItem>
<FormLabel>{t('form.logPath')}</FormLabel>
<FormControl><Input placeholder={collector?.type === 'host-promtail' ? '/var/log/*.log' : '/var/log/app/*.log'} {...field} /></FormControl>
<FormMessage />
<p className="text-xs text-muted-foreground">{t('edit.logPathHint')}</p>
</FormItem>
)} />}
{supportsPath && <FormField control={form.control} name="logPath" render={({ field }) => (
<FormItem>
<FormLabel>{t('form.logPath')}</FormLabel>
<FormControl><Input placeholder="/var/log/app/*.log" {...field} /></FormControl>
<FormMessage />
<p className="text-xs text-muted-foreground">{t('edit.logPathHint')}</p>
</FormItem>
)} />}
<div className="flex justify-end gap-2 pt-2">
<Button type="button" variant="outline" onClick={() => onOpenChange(false)}>{t('common:dialog.cancel')}</Button>
<Button type="submit" disabled={saving}>{saving ? t('edit.saving') : t('edit.save')}</Button>
</div>
</form>
</Form>
</DialogContent>
</Dialog>
<div className="flex justify-end gap-2 pt-2">
<Button type="button" variant="outline" onClick={() => onOpenChange(false)}>{t('common:dialog.cancel')}</Button>
<Button type="submit" disabled={saving}>{saving ? t('edit.saving') : t('edit.save')}</Button>
</div>
</form>
</Form>
</DialogContent>
</Dialog>
{/* 预览:将要写到目标机的 promtail.yaml(不落库、不下发) */}
<Dialog open={preview !== null} onOpenChange={(open) => { if (!open) setPreview(null) }}>
<DialogContent className="sm:max-w-3xl max-h-[90vh] overflow-y-auto">
<DialogHeader>
<DialogTitle>{t('edit.preview.title')}</DialogTitle>
<DialogDescription>{t('edit.preview.description')}</DialogDescription>
</DialogHeader>
{(preview?.warnings ?? []).length > 0 && (
<ul className="space-y-1 rounded-md border border-amber-300/60 bg-amber-50 p-2 text-xs text-amber-900 dark:bg-amber-950/30 dark:text-amber-200">
{(preview?.warnings ?? []).map((w) => <li key={w}>· {w}</li>)}
</ul>
)}
<pre className="max-h-[55vh] overflow-auto rounded-md bg-muted p-3 text-[11px] leading-relaxed">
{preview?.config}
</pre>
<div className="flex items-center justify-between">
<Button
type="button"
size="sm"
variant="outline"
onClick={() => {
if (preview?.config) {
void navigator.clipboard?.writeText(preview.config)
toast.success(t('edit.preview.copied'))
}
}}
>
{t('edit.preview.copy')}
</Button>
<Button type="button" size="sm" onClick={() => setPreview(null)}>{t('edit.preview.close')}</Button>
</div>
</DialogContent>
</Dialog>
</>
)
})
@@ -0,0 +1,193 @@
import { memo } from 'react'
import { useTranslation } from 'react-i18next'
import '@/i18n/locales/zh/collectors'
import '@/i18n/locales/en/collectors'
import { Plus, Trash2 } from 'lucide-react'
import { Button } from '@/components/ui/button'
import { Input } from '@/components/ui/input'
import { Label } from '@/components/ui/label'
import { Switch } from '@/components/ui/switch'
import { emptyJobDraft, type JobDraft } from './scrape-jobs'
interface Props {
drafts: JobDraft[]
onChange: (next: JobDraft[]) => void
}
/**
* 采集项结构化编辑器(B 档)。
*
* 只暴露能覆盖常见场景的几个开关,不逼用户写 YAML / Promtail 配置:
* 名称、路径、标签常驻;多行合并 / JSON / 自定义时间戳 / 丢弃 / 限流收进「高级」,
* 免得默认视图就一屏表单(用户明确要求紧凑、高频项置顶、低频项折叠)。
*/
export const ScrapeJobsEditor = memo(function ScrapeJobsEditor({ drafts, onChange }: Props) {
const { t } = useTranslation('collectors')
const patch = (index: number, part: Partial<JobDraft>) => {
onChange(drafts.map((d, i) => (i === index ? { ...d, ...part } : d)))
}
const add = () => {
const n = drafts.length + 1
onChange([...drafts, emptyJobDraft(n <= 2 ? `app-${n}` : `job-${n}`)])
}
const remove = (index: number) => onChange(drafts.filter((_, i) => i !== index))
return (
<div className="space-y-2">
{drafts.map((d, i) => (
<div key={i} className="space-y-2 rounded-md border p-3">
<div className="flex items-center gap-2">
<Input
className="h-8 w-44 font-mono text-xs"
value={d.name}
placeholder={t('edit.jobs.namePlaceholder')}
onChange={(e) => patch(i, { name: e.target.value })}
/>
{d.system && (
<span className="rounded bg-muted px-1.5 py-0.5 text-[11px] text-muted-foreground">
{t('edit.jobs.systemBadge')}
</span>
)}
<Button
type="button"
size="sm"
variant="ghost"
className="ml-auto h-7 px-2 text-muted-foreground"
onClick={() => remove(i)}
aria-label={t('edit.jobs.remove')}
>
<Trash2 className="h-3.5 w-3.5" />
</Button>
</div>
<div className="grid gap-2">
<div className="space-y-1">
<Label className="text-xs text-muted-foreground">{t('edit.jobs.paths')}</Label>
<Input
className="h-8 font-mono text-xs"
value={d.paths}
placeholder={d.system ? t('edit.jobs.systemPathsPlaceholder') : '/data/app/logs/*.log, /data/app/logs/error/*.log'}
onChange={(e) => patch(i, { paths: e.target.value })}
/>
</div>
<div className="space-y-1">
<Label className="text-xs text-muted-foreground">{t('edit.jobs.labels')}</Label>
<Input
className="h-8 font-mono text-xs"
value={d.labels}
placeholder="app=cex-api, env=pre"
onChange={(e) => patch(i, { labels: e.target.value })}
/>
</div>
</div>
<details className="group">
<summary className="cursor-pointer text-xs text-muted-foreground hover:text-foreground">
{t('edit.jobs.advanced')}
</summary>
<div className="mt-2 space-y-3 rounded-md bg-muted/40 p-2">
<div className="flex flex-wrap items-center gap-4">
<label className="flex items-center gap-2 text-xs">
<Switch checked={d.system} onCheckedChange={(v) => patch(i, { system: v })} />
{t('edit.jobs.system')}
</label>
<label className="flex items-center gap-2 text-xs">
<Switch checked={d.json} onCheckedChange={(v) => patch(i, { json: v })} />
{t('edit.jobs.json')}
</label>
<label className="flex items-center gap-2 text-xs">
<Switch
checked={d.multilineEnabled}
onCheckedChange={(v) => patch(i, { multilineEnabled: v })}
/>
{t('edit.jobs.multiline')}
</label>
<label className="flex items-center gap-2 text-xs">
<Switch
checked={d.timestampEnabled}
onCheckedChange={(v) => patch(i, { timestampEnabled: v })}
disabled={d.system}
/>
{t('edit.jobs.timestamp')}
</label>
</div>
{d.multilineEnabled && (
<div className="grid gap-2 sm:grid-cols-[1fr_120px]">
<Input
className="h-8 font-mono text-xs"
value={d.firstline}
placeholder="^\d{4}-\d{2}-\d{2}"
onChange={(e) => patch(i, { firstline: e.target.value })}
/>
<Input
className="h-8 text-xs"
value={d.maxLines}
placeholder={t('edit.jobs.maxLinesPlaceholder')}
onChange={(e) => patch(i, { maxLines: e.target.value })}
/>
</div>
)}
{d.timestampEnabled && !d.system && (
<div className="grid gap-2 sm:grid-cols-3">
<Input
className="h-8 text-xs"
value={d.tsSource}
placeholder={t('edit.jobs.tsSourcePlaceholder')}
onChange={(e) => patch(i, { tsSource: e.target.value })}
/>
<Input
className="h-8 font-mono text-xs"
value={d.tsFormat}
placeholder="RFC3339"
onChange={(e) => patch(i, { tsFormat: e.target.value })}
/>
<Input
className="h-8 font-mono text-xs"
value={d.tsLocation}
placeholder={t('edit.jobs.tsLocationPlaceholder')}
onChange={(e) => patch(i, { tsLocation: e.target.value })}
/>
</div>
)}
<div className="space-y-1">
<Label className="text-xs text-muted-foreground">{t('edit.jobs.dropIfMatch')}</Label>
<Input
className="h-8 font-mono text-xs"
value={d.dropIfMatch}
placeholder={t('edit.jobs.dropIfMatchPlaceholder')}
onChange={(e) => patch(i, { dropIfMatch: e.target.value })}
/>
</div>
<div className="grid gap-2 sm:grid-cols-2">
<Input
className="h-8 text-xs"
value={d.rate}
placeholder={t('edit.jobs.ratePlaceholder')}
onChange={(e) => patch(i, { rate: e.target.value })}
/>
<Input
className="h-8 text-xs"
value={d.burst}
placeholder={t('edit.jobs.burstPlaceholder')}
onChange={(e) => patch(i, { burst: e.target.value })}
/>
</div>
</div>
</details>
</div>
))}
<Button type="button" size="sm" variant="outline" onClick={add}>
<Plus className="mr-1 h-3.5 w-3.5" />
{t('edit.jobs.add')}
</Button>
</div>
)
})
@@ -0,0 +1,166 @@
// 采集项草稿模型:编辑器里全部用字符串保存(输入框友好),
// 提交/预览时再转成后端要的 ScrapeJob 结构。
import type { ScrapeJob } from '@/types'
export interface JobDraft {
name: string
paths: string
labels: string
system: boolean
json: boolean
multilineEnabled: boolean
firstline: string
maxLines: string
timestampEnabled: boolean
tsSource: string
tsFormat: string
tsLocation: string
dropIfMatch: string
rate: string
burst: string
}
export const emptyJobDraft = (name = ''): JobDraft => ({
name,
paths: '',
labels: '',
system: false,
json: false,
multilineEnabled: false,
firstline: '',
maxLines: '',
timestampEnabled: false,
tsSource: '',
tsFormat: 'RFC3339',
tsLocation: '',
dropIfMatch: '',
rate: '',
burst: '',
})
export function jobToDraft(j: ScrapeJob): JobDraft {
return {
name: j.name ?? '',
paths: (j.paths ?? []).join(', '),
labels: Object.entries(j.labels ?? {})
.map(([k, v]) => `${k}=${v}`)
.join(', '),
system: Boolean(j.system),
json: Boolean(j.json),
multilineEnabled: Boolean(j.multiline?.firstline),
firstline: j.multiline?.firstline ?? '',
maxLines: j.multiline?.maxLines ? String(j.multiline.maxLines) : '',
timestampEnabled: Boolean(j.timestamp?.source),
tsSource: j.timestamp?.source ?? '',
tsFormat: j.timestamp?.format ?? 'RFC3339',
tsLocation: j.timestamp?.location ?? '',
dropIfMatch: (j.dropIfMatch ?? []).join(', '),
rate: j.rateLimit?.rate ? String(j.rateLimit.rate) : '',
burst: j.rateLimit?.burst ? String(j.rateLimit.burst) : '',
}
}
function splitList(raw: string): string[] {
return raw
.split(/[,\n]/)
.map((s) => s.trim())
.filter(Boolean)
}
/** "app=cex-api, env=pre" → {app:'cex-api', env:'pre'};不合法键由后端校验给具体报错 */
export function parseLabels(raw: string): Record<string, string> {
const out: Record<string, string> = {}
for (const item of splitList(raw)) {
const i = item.indexOf('=')
if (i <= 0) continue
const k = item.slice(0, i).trim()
const v = item.slice(i + 1).trim()
if (k) out[k] = v
}
return out
}
const toInt = (raw: string) => {
const n = Number.parseInt(raw.trim(), 10)
return Number.isFinite(n) && n > 0 ? n : 0
}
export function draftToJob(d: JobDraft): ScrapeJob {
const job: ScrapeJob = {
name: d.name.trim(),
paths: splitList(d.paths),
}
if (d.system) job.system = true
const labels = parseLabels(d.labels)
if (Object.keys(labels).length > 0) job.labels = labels
if (d.multilineEnabled && d.firstline.trim()) {
job.multiline = { firstline: d.firstline.trim() }
if (toInt(d.maxLines) > 0) job.multiline.maxLines = toInt(d.maxLines)
}
if (d.json) job.json = true
if (d.timestampEnabled && d.tsSource.trim()) {
job.timestamp = {
source: d.tsSource.trim(),
format: d.tsFormat.trim() || 'RFC3339',
}
if (d.tsLocation.trim()) job.timestamp.location = d.tsLocation.trim()
}
const drops = splitList(d.dropIfMatch)
if (drops.length > 0) job.dropIfMatch = drops
const rate = toInt(d.rate)
const burst = toInt(d.burst)
if (rate > 0 || burst > 0) job.rateLimit = { rate, burst }
return job
}
/** 本地即时校验(后端仍会再校验一次,这边只做「不用等请求就看出填错」) */
export function draftIssues(drafts: JobDraft[]): string[] {
const issues: string[] = []
const seen = new Set<string>()
drafts.forEach((d, i) => {
const label = d.name.trim() || `第 ${i + 1} 项`
if (!d.name.trim()) {
issues.push(`第 ${i + 1} 个采集项缺少名称`)
} else if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(d.name.trim())) {
issues.push(`采集项「${label}」名称只能用字母、数字、点、下划线、中划线`)
} else if (seen.has(d.name.trim())) {
issues.push(`采集项名称「${label}」重复`)
} else {
seen.add(d.name.trim())
}
const paths = splitList(d.paths)
if (!d.system && paths.length === 0) {
issues.push(`采集项「${label}」至少填一个日志路径`)
}
for (const p of paths) {
if (!p.startsWith('/')) {
issues.push(`采集项「${label}」的路径「${p}」必须是绝对路径`)
}
if (/^\/(etc\/shadow|etc\/gshadow|etc\/sudoers|root\/\.ssh)|id_rsa|id_ed25519|\.pem$/i.test(p)) {
issues.push(`采集项「${label}」的路径「${p}」是敏感凭据文件,禁止采集`)
}
}
if (d.multilineEnabled && !d.firstline.trim()) {
issues.push(`采集项「${label}」开了多行合并但没填首行正则`)
}
if (d.timestampEnabled && !d.tsSource.trim()) {
issues.push(`采集项「${label}」开了自定义时间戳但没填字段名`)
}
for (const item of splitList(d.labels)) {
if (!item.includes('=')) {
issues.push(`采集项「${label}」的标签「${item}」要写成 key=value`)
}
}
})
return issues
}
/** 没有任何采集项时,从旧字段 logPath 迁移出默认两项(应用日志 + 系统日志) */
export function draftsFromCollector(jobs: ScrapeJob[] | undefined, logPath: string): JobDraft[] {
if (jobs && jobs.length > 0) return jobs.map(jobToDraft)
const app = emptyJobDraft('app')
app.paths = (logPath || '/var/log/*.log').trim()
const system = emptyJobDraft('system')
system.system = true
return [app, system]
}
+21
View File
@@ -182,6 +182,21 @@ export interface PageData<T> {
pageSize: number;
}
// ScrapeJob 主机模式的一个采集项(对应 promtail 的 scrape_configs 一项)
export interface ScrapeJob {
name: string;
paths: string[];
labels?: Record<string, string>;
encoding?: string;
/** 内置系统日志采集项:自动 syslog 正则 + 时间戳解析,paths 留空用探测到的系统日志 */
system?: boolean;
multiline?: { firstline: string; maxLines?: number };
json?: boolean;
timestamp?: { source: string; format: string; location?: string };
dropIfMatch?: string[];
rateLimit?: { rate: number; burst: number };
}
// Collector(日志采集器 / 部署客户端)
export interface Collector {
id: number;
@@ -204,6 +219,12 @@ export interface Collector {
appInclude?: string[];
logPath: string;
labels: string;
/** 主机模式的结构化采集项(B 档);空 = 按 logPath 生成 app + system 默认项 */
scrapeJobs?: ScrapeJob[];
/** 上次安装探测到的真实主机名(host 标签用它,不是 ${HOSTNAME}) */
resolvedHostname?: string;
/** 上次安装探测到的 syslog 时间戳格式:iso8601 | rfc3164 */
syslogFormat?: string;
status: string;
statusMessage: string;
createdAt: string;