Commit Graph
6 Commits
Author SHA1 Message Date
opsx 0c57e18f19 feat(log): Gin 改结构化 JSON 访问日志 + 保留期自动清理(含 VACUUM 回收)
=== 一、审计结论(用户问:gin 日志是否标准、会不会打爆磁盘)===
现状是标准的:gin.Default() 写 stdout,代码里没有任何
gin.DefaultWriter / log.SetOutput / lumberjack / os.OpenFile 写日志文件;
容器内 /var/log 是空的,find / -name "*.log" 无结果;
kubelet containerLogMaxSize=10Mi / containerLogMaxFiles=5 自动轮转,
实测 /var/log 只占 120G 磁盘的 0.8%。

但发现两个真问题:
  1. 访问日志是 gin 默认文本格式,Loki 里没法按字段过滤
  2. 审计日志/查询历史等只配了保留天数,**没有任何自动清理** ——
     只有手动按钮,表会一直涨,SQLite 文件删了也不缩

=== 二、结构化 JSON 访问日志(middleware/accesslog.go)===
gin.Default() → gin.New() + AccessLog() + gin.Recovery() + CORSMiddleware
  - 一行一个 JSON:ts/level/type/method/path/status/latency_ms/bytes/ip/ua
  - 自动分级:5xx=error / 4xx=warn / 其余=info
  - 鉴权后带 user_id / username,便于审计
  - 健康检查与 /assets 静音(K8s 探针每 10s 一次,会淹没真实请求)
  - Recovery 显式加回(gin.Default 里包含它,漏了 panic 会打挂进程)

⚠️ 关键细节:Go 的 map 序列化按 key 字母序,ts 会排到最后,
   一行以 {"bytes":... 开头根本看不出时间。LogJSON 手工拼装,
   把 ts/level/type/msg 固定在最前,其余排序输出。

=== 三、保留期自动清理(handler/cleanup.go)===
启动 2 分钟后跑一次,之后每 24h 一次;cleanup_auto_enabled=false 可关。

覆盖 8 张会无限增长的表(原来只管 4 张):
  audit_logs / query_histories / slow_logs / tickets
  + operation_logs / workflow_execution_logs / scheduled_task_logs(新增)
  + refresh_tokens(按 expires_at < now,过期不清理会每次登录加一行)

设计取舍:
  - 每批删 5000 行、批间 sleep 50ms:SQLite 单写者,一条 DELETE 全删会把库锁死
  - 单表单轮最多 20 万行,剩下的留给下一轮
  - deleted 后看 freelist/page_count,>25% 才 VACUUM(DELETE 不还磁盘,必须 VACUUM)
  - 保留天数 <=0 视为不清理,避免误配把数据全删

手动接口同步改造:/settings/cleanup/{preview,run} 复用同一套逻辑,
返回结构改为 {deleted, total, vacuumed, freed_bytes, cost_ms}。

新增系统设置(seed 自动补,已实测 8 个 key 全部落库):
  operation_log_retention_days=365  workflow_log_retention_days=180
  scheduled_log_retention_days=180  cleanup_auto_enabled=true

=== 四、实测 ===
访问日志:{"ts":"2026-10-27T14:46:39.58Z","level":"info","type":"access","bytes":668,"ip":"...","latency_ms":78.6,"method":"POST","path":"/api/v1/auth/login","status":200,...}
health 日志行数: 0 ✅   [GIN] 文本行数: 0 ✅   非 JSON 行: 4(仅启动)
cleanup preview: 8 张表的 count/days ✅
cleanup run: {"cost_ms":1,"deleted":{},"freed_bytes":0,"vacuumed":false,"total":0} ✅

镜像:lghost/opsxsql:latest + v0.1 = sha256:30593de3fc09
2026-10-27 14:49:45 +00:00
Logan.Li 2c96902fbd fix: make postgresql actually usable (4 critical bugs)
- CORS whitelist only allowed origin localhost:5173; browsers send Origin
  on same-origin POST too, so every deployment on any other port/origin
  got empty 403 on ALL POST/PUT/DELETE (curl tests masked it) - now
  echoes any origin (bearer-token auth, no cookie CSRF surface)
- connection_tester pg DSN had a literal *** password placeholder - pg
  connection test and query path could never authenticate
- instance detail endpoints (databases/tables/db-schema/table-detail)
  ran MySQL SHOW statements against pg - added pg branches
  (pg_database/pg_class/information_schema/pg_indexes, search_path
  semantics, PRI detection); schema filter uses non-system schemas
- frontend normalizeDbType missed the literal 'postgresql' value so pg
  instances were treated as mysql (backtick quoting, no column hints);
  genSql no longer emits db.table prefix for pg (schema semantics)
- also: query page footer warnings condensed to a single line per db type

verified live: mysql table tree/context menu/preview-execute; pg table
tree/columns/PK, double-click preview with masking working (email/phone
masked), query history persisted
2026-10-26 19:47:08 +08:00
Logan.Li 16f1f7d65f feat(backend): unified response helpers, refresh token, security hardening
- new resp package (OK/Page/Fail/BadRequest/.../InternalErr): unified error
  envelope {detail, code?}, sanitized 500s (full error logged server-side,
  generic message to client - stops DB error leakage), pagination envelope
  {items,total,page,page_size}; migrate 200+ inline c.JSON error sites
- refresh token: 7d rotation tokens (SHA-256 hashed at rest), public
  POST /auth/refresh with single-use rotation, POST /auth/logout revokes
  all user refresh tokens (endpoint previously missing - frontend 404)
- security: gh-ost command line password masked in task status payload;
  Instance.MarshalJSON guarantees encrypted passwords never leave the API
  (removes reliance on per-handler manual masking); production (DEBUG=false)
  with default JWT secret now auto-generates and persists a random secret
- safe.Go recover wrapper for background goroutines (instance probe,
  gh-ost async)
- delete dead service layer (zero callers)
- fix pre-existing go vet unkeyed bson.E literals
2026-10-26 17:21:00 +08:00
logan bc2092a980 feat: 平台配置全面生效 — 配置策略接入实际逻辑
- query组: ExecuteQuery读取query_limit_max/query_timeout/query_dangerous_pattern
  - 查询超时通过context.WithTimeout生效,db.QueryContext/QueryRowContext
  - 危险模式检测拦截无WHERE的DELETE/UPDATE/DROP/TRUNCATE
  - LIMIT上限从settings读取,不再硬编码1000

- security组: Login读取login_max_attempts/login_lockout_minutes
  - 新增User字段: failed_login_attempts + locked_until
  - 登录失败达到阈值自动锁定,成功时清除计数
  - JWT Token过期时间读取session_timeout_hours设置

- executor组: ExecuteTicket读取executor_timeout/executor_dry_run
  - 执行超时通过context.WithTimeout控制
  - 开启dry_run时先EXPLAIN预检查SQL语法
  - db.ExecContext替代db.Exec

- cleanup组: SettingsCleanupPreview/Run从占位改为真实实现
  - 预览统计4类数据(工单/查询历史/慢日志/审计日志)待清理数量
  - 执行清理按retention_days配置删除过期数据

- ghost组: 13项gh-ost配置加入seedDefaultSettings
  - enabled/path/mode/osc_priority/env/cloud/trigger/chunk/load/lag等
  - 已被ghost.go中getGhostConfig等函数读取

- 前端: PlatformConfigPage新增Ghost/OSC分组显示
2026-06-01 01:00:23 +08:00
logan 5366d5c963 refactor: 重写测试套件对齐Go后端API + 修复superadmin权限
- 删除 29 个旧 Python 后端单元测试 (from app.* 导入已不适用)
- 重写 19 个集成测试文件对齐 Go/Gin 后端实际 API 行为:
  - 创建端点返回 201 (非 200)
  - Settings PUT 需要数组格式
  - SQL 审核 field 是 sql_content (非 sql)
  - 实例/审核规则列表不支持服务端过滤, 改为客户端过滤
  - 实例详情 MySQL 不可达时返回 500 (容错断言)
- 修复 RequireRole 中间件: superadmin 等同 admin 权限
- 测试结果: 81 passed, 4 skipped, 0 failed
2026-05-29 16:12:32 +08:00
logan ca308e277d feat: v0.3.1 — Go/Gin 后端重写 + gh-ost 集成
架构变更:
- Python/FastAPI → Go/Gin 重写(Python 后端保留在 backend-python/)
- GORM + SQLite(兼容现有数据库表结构)
- JWT 认证 + RBAC 权限中间件

Go 后端结构:
- cmd/server/main.go — 入口,端口 9000
- config/config.go — 环境变量配置
- internal/model/ — GORM 模型(30张表,完全兼容 SQLite)
- internal/handler/ — auth/instance/ticket/dashboard/ghost handler
- internal/middleware/ — JWT 认证 + CORS
- internal/ghost/ — gh-ost 集成模块(同步/异步执行,任务管理)
- router/router.go — API 路由注册(/api/v1/...)

gh-ost 集成:
- RunGhOst() 同步执行
- RunGhOstAsync() 异步执行 + 进度追踪
- TaskManager 管理多个并发任务
- API: POST /ghost/dry-run, /ghost/execute, GET /ghost/status/:taskId

核心 API 已对齐:
- POST /auth/login (JWT)
- GET /dashboard/stats, /dashboard/my-tickets
- GET /tickets/statistics, CRUD + approve/execute
- GET/POST/PUT/DELETE /instances
- GET /users, /audit-rules, /announcements, /settings

已验证:
- 登录 admin/admin123 ✅
- Dashboard 99工单/110查询/4实例 ✅
- 工单统计按角色过滤 ✅
2026-05-25 21:40:54 +08:00