Files
opsx 7dfbc0b680 feat(redis): 大 key 分页 + 流式导出(含票据鉴权)+ 查询行数上限默认 5000
=== 一、问题:大 key 会把内存打爆,且查询结果没有导出 ===
RedisGetKeyDetail 原来对每种类型都是全量读取:
  string → GET        list → LRANGE 0 -1     set → SMEMBERS
  zset   → ZRANGE 0 -1                        hash → HGETALL
然后整包序列化返回。像 PERP:market:* 这类盘口/kline key,
一个 key 几十万条就会:Go 侧整值入内存 → JSON 序列化翻倍 → 浏览器渲染卡死。

=== 二、后端:键详情分页(统一令牌,兼容四种类型)===
GET /redis/:id/keys/:key?db=&count=&token=
  count  默认 200,硬上限 2000
  token  语义按类型区分,前端无感知:
           list / zset  → 十进制 offset(LRANGE / ZRANGE 区间)
           hash / set   → Redis SCAN 游标(HSCAN / SSCAN)
           string       → 忽略
  返回   total / count / nextToken / truncated / previewCap

string 特殊处理:不能分页,改为预览上限 256KB
  - 用 STRLEN 拿真实大小(O(1))
  - 超过上限时用 GETRANGE 只取前 256KB,truncated=true
  - 完整内容只能靠导出

新增 wrapper 方法:HScan / ZScan / HLen / ZCard / StrLen / GetRange
(HGETALL、SMEMBERS、ZRANGE 0 -1 一律不再用于详情读取)

=== 三、后端:流式导出(两段式票据,避免令牌进 URL 日志)===
① POST /redis/:id/keys/:key/export-ticket   (走 Bearer 鉴权)
     → HMAC-SHA256(JWTSecret, instance|db|key|exp) 生成票据,2 分钟过期,
       绑定具体 instance/db/key,不能跨 key 复用
② GET  /redis/:id/keys/:key/export?db=&format=csv|jsonl&ticket=
     → 注册在鉴权组【之外】,靠票据校验
     为什么这么做:浏览器原生下载(<a download>/window.open)带不了
     Authorization 头;若前端用 axios 取 blob,整份导出会进浏览器内存,
     与「大 key 不爆内存」的目标相悖。

导出实现全程流式(服务端内存 = 单批 + 64KB 缓冲):
  string → GETRANGE 256KB 分块
  list   → LRANGE 每批 1000
  zset   → ZRANGE WITHSCORES 每批 1000
  hash   → HSCAN 游标每批 1000
  set    → SSCAN 游标每批 1000
单次上限 200 万行;CSV 带 UTF-8 BOM(Excel 直接打开不乱码);
列顺:hash(field,value) / list(index,value) / set(member) / zset(rank,member,score)

=== 四、前端:详情面板加分页条 + 4 个导出按钮 ===
RedisKeyDetailPanel:
  - 「共 N 条 · 第 X 页 · 每页 [100/200/500/1000/2000]」+ 上一页/下一页
  - SCAN 游标不可逆,上一页用令牌栈回放
  - string 超预览上限时显示黄色提示「仅预览前 256KB,共 N 字节」
  - 导出四档:当前页 CSV / 当前页 JSON(纯前端,瞬时)
             导出全部 CSV / 导出全部 JSONL(服务端流式,边读边写)
  顺带修掉头注释里「已知限制:大值不分页」的过时说明。

=== 五、查询行数上限:默认 1000 → 5000,并修掉硬编码 ===
用户要求:「默认设置里面有查询最大上限吧 比如 mysql pg 查询最大Limit限制 5000」

发现两个问题:
1. seed 默认值只有 1000(main.go query_limit_max)
2. ExecuteQuery 里还有一段硬编码:
     if input.Limit <= 0 { input.Limit = maxLimit; if input.Limit > 1000 { input.Limit = 1000 } }
   → 管理员把设置改成 5000 也白搭,前端不传 limit 时永远被压回 1000

修复:
  - seed 默认值 1000 → 5000(Value + DefaultValue)
  - getSettingInt("query_limit_max", 1000) → 5000
  - 删掉硬编码 1000 封顶,唯一口径就是设置值
  - 线上已有记录的 value 已通过 PUT /settings 改成 5000(seed 只在缺失时插入,不会自动更新旧值)

对 MySQL / PostgreSQL / Redis / MongoDB 统一生效(ExecuteQuery 按 db_type 分发,
limit 在入口处统一 clamp)。

=== 六、实测结果(线上)===
分页:
  zset PERP:market:kline:symbol_kline_12h_AAVEUSDT_PERP_MARK
        total=83,count=5 → 第1页 5 条 nextToken=5 → 第2页 5 条 nextToken=10,数据不同 ✅
  hash ST:market:ticker:symbol_ticker
        total=27,count=3 → 第1页 3 字段 nextToken=24 → 第2页 3 字段(完全不同)✅
  string PERP:market:depth:symbol_depth_BTCUSDT_PERP
        STRLEN=14192 → 导出 14192 字节,完全一致 ✅(实时 key 两次采样有差异属正常)
导出:
  zset CSV  200 / 30970 字节 / 84 行(header+83)✅
  zset JSONL 200 / 84 行 ✅
  string CSV 200 / 分块 GETRANGE 拼接完整 ✅
  无票据 → 403 ✅   伪造票据 → 403 ✅
查询上限:
  limit=99999 → 5000 行 ✅    limit=0(默认)→ 5000 行 ✅
SQL 导出 CSV(原已存在 ExportQueryCSV,支持 mysql/pg/redis/mongo)200 / 101 行 ✅

镜像:lghost/opsxsql:latest + v0.1 = sha256:68fc996681c6
2026-10-27 09:14:02 +00:00

232 lines
9.9 KiB
Go

package router
import (
"sql-platform/internal/handler"
"sql-platform/internal/middleware"
"github.com/gin-gonic/gin"
)
func Setup(r *gin.Engine) {
api := r.Group("/api/v1")
// Public routes
api.POST("/auth/login", handler.Login)
api.POST("/auth/refresh", handler.RefreshToken)
api.GET("/public/login-style", handler.GetPublicLoginStyle)
// Redis 大 Key 导出下载(浏览器原生下载带不了 Authorization 头,
// 改由一次性短时效票据鉴权,票据绑定 instance/db/key 且 2 分钟过期)。
// 必须注册在鉴权组之外,否则 <a download> 会被 401。
api.GET("/redis/:id/keys/:key/export", handler.RedisExportKeyStream)
// Authenticated routes
auth := api.Group("")
auth.Use(middleware.AuthMiddleware())
{
// Auth
auth.GET("/auth/me", handler.GetMe)
auth.POST("/auth/logout", handler.Logout)
// Dashboard
auth.GET("/dashboard/stats", handler.DashboardStats)
auth.GET("/dashboard/my-tickets", handler.MyTickets)
// Tickets
auth.GET("/tickets/statistics", handler.TicketStatistics)
auth.GET("/tickets", handler.ListTickets)
auth.GET("/tickets/scheduled/list", handler.ListScheduledTasks)
auth.GET("/tickets/:id", handler.GetTicket)
auth.POST("/tickets", handler.CreateTicket)
auth.POST("/tickets/:id/approve", handler.ApproveTicket)
auth.POST("/tickets/:id/review", handler.ApproveTicket)
auth.POST("/tickets/:id/execute", handler.ExecuteTicket)
auth.POST("/tickets/:id/cancel-schedule", handler.CancelScheduledTask)
auth.POST("/tickets/:id/cancel", handler.CancelTicket)
auth.GET("/tickets/:id/timeline", handler.GetTicketTimeline)
auth.GET("/tickets/:id/approvals", handler.GetTicketApprovals)
auth.POST("/tickets/:id/rollback", handler.RollbackTicket)
// Instances
auth.GET("/instances", handler.ListInstances)
auth.GET("/instances/accessible", handler.AccessibleInstances)
auth.GET("/instances/stats", handler.InstanceStats)
auth.GET("/instances/:id", handler.GetInstance)
auth.GET("/instances/:id/databases", handler.ListDatabases)
auth.GET("/instances/:id/databases/:db/tables", handler.ListTables)
auth.POST("/instances/:id/test-connection", handler.TestConnection)
auth.POST("/instances/test-connection", handler.TestNewConnection)
auth.GET("/instances/:id/metrics", handler.InstanceMetrics)
// Instance Detail (databases/tables/schema/processlist/status/users)
auth.GET("/instances/:id/detail/databases", handler.DetailListDatabases)
auth.GET("/instances/:id/detail/tables", handler.DetailListTables)
auth.GET("/instances/:id/detail/db-schema", handler.DetailDbSchema)
auth.GET("/instances/:id/detail/table-detail", handler.DetailTableDetail)
auth.GET("/instances/:id/detail/processlist", handler.DetailProcesslist)
auth.GET("/instances/:id/detail/status", handler.DetailStatus)
auth.POST("/instances/:id/detail/processlist/:pid/kill", handler.DetailKillProcess)
auth.GET("/instances/:id/detail/users", handler.DetailListUsers)
// Instance management (admin/dba only)
admin := auth.Group("")
admin.Use(middleware.RequireRole("admin", "dba"))
{
admin.POST("/instances", handler.CreateInstance)
admin.PUT("/instances/:id", handler.UpdateInstance)
admin.DELETE("/instances/:id", handler.DeleteInstance)
}
// Instance Groups
auth.GET("/instance-groups", handler.ListInstanceGroups)
auth.GET("/instance-groups/:id", handler.GetInstanceGroup)
auth.GET("/instance-groups/:id/users", handler.ListGroupUsers)
auth.POST("/instance-groups/:id/instances", handler.AssignGroupInstances)
auth.DELETE("/instance-groups/:id/instances/:instanceId", handler.RemoveGroupInstance)
auth.POST("/instance-groups/:id/users", handler.AssignGroupUsers)
auth.PUT("/instance-groups/users/:userId/groups", handler.UpdateUserGroups)
// Instance Group management (admin/dba only)
igAdmin := auth.Group("")
igAdmin.Use(middleware.RequireRole("admin", "dba"))
{
igAdmin.POST("/instance-groups", handler.CreateInstanceGroup)
igAdmin.PUT("/instance-groups/:id", handler.UpdateInstanceGroup)
igAdmin.DELETE("/instance-groups/:id", handler.DeleteInstanceGroup)
}
// Redis
auth.GET("/redis/:id/info", handler.RedisInfo)
auth.GET("/redis/:id/dbsize", handler.RedisDbsize)
auth.GET("/redis/:id/dbs", handler.RedisDbs)
auth.GET("/redis/:id/keys", handler.RedisScanKeysPaged)
auth.POST("/redis/:id/keys/:key/field", handler.RedisUpdateKeyField)
auth.GET("/redis/:id/keys/:key", handler.RedisGetKeyDetail)
// 大 key 导出:先取一次性票据,再由浏览器原生下载 /export?ticket=...
auth.POST("/redis/:id/keys/:key/export-ticket", handler.RedisCreateExportTicket)
auth.GET("/redis/:id/slowlog", handler.RedisSlowlog)
auth.GET("/redis/:id/clients", handler.RedisClients)
auth.POST("/redis/:id/command", handler.RedisExecuteCommand)
auth.POST("/redis/:id/keys", handler.RedisCreateKey)
auth.PUT("/redis/:id/keys/:key", handler.RedisUpdateKey)
auth.DELETE("/redis/:id/keys/:key", handler.RedisDeleteKey)
// Users (admin only)
userAdmin := auth.Group("")
userAdmin.Use(middleware.RequireRole("admin"))
{
userAdmin.GET("/users", handler.ListUsers)
userAdmin.POST("/users", handler.CreateUser)
userAdmin.PUT("/users/:id", handler.UpdateUser)
userAdmin.DELETE("/users/:id", handler.DeleteUser)
userAdmin.POST("/users/:id/reset-password", handler.ResetUserPassword)
userAdmin.POST("/users/batch-import", handler.BatchImportUsers)
userAdmin.POST("/users/batch-import-ldap", handler.BatchImportLdapUsers)
userAdmin.GET("/users/import-template", handler.ImportTemplate)
}
// LDAP config (admin only)
ldapAdmin := auth.Group("")
ldapAdmin.Use(middleware.RequireRole("admin"))
{
ldapAdmin.GET("/ldap-config", handler.GetLDAPConfig)
ldapAdmin.PUT("/ldap-config", handler.UpdateLDAPConfig)
ldapAdmin.POST("/ldap-config/test", handler.TestLDAPConnection)
ldapAdmin.POST("/ldap-config/ou-list", handler.ListLDAPOUs)
ldapAdmin.POST("/ldap-config/user-list", handler.ListLDAPUsers)
ldapAdmin.POST("/ldap-config/create-ou", handler.CreateLDAPOU)
ldapAdmin.POST("/ldap-config/create-user", handler.CreateLDAPUser)
ldapAdmin.POST("/ldap-config/batch-import-ldap", handler.BatchImportLDAP)
ldapAdmin.POST("/ldap-config/delete-user", handler.DeleteLDAPUser)
}
// User Preferences (per-user, auth required)
auth.GET("/user/preferences", handler.GetUserPreferences)
auth.PUT("/user/preferences", handler.UpdateUserPreferences)
// Settings
auth.GET("/settings", handler.ListSettings)
auth.PUT("/settings", handler.UpdateSettings)
auth.GET("/settings/cleanup/preview", handler.SettingsCleanupPreview)
auth.POST("/settings/cleanup/run", handler.SettingsCleanupRun)
// gh-ost (admin/dba only)
ghostGroup := auth.Group("")
ghostGroup.Use(middleware.RequireRole("admin", "dba"))
{
ghostGroup.GET("/ghost/config", handler.GhOstGetConfig)
ghostGroup.POST("/ghost/analyze", handler.GhOstAnalyzeDDL)
ghostGroup.POST("/ghost/dry-run", handler.GhOstDryRun)
ghostGroup.POST("/ghost/execute", handler.GhOstExecute)
ghostGroup.GET("/ghost/status/:taskId", handler.GhOstStatus)
ghostGroup.GET("/ghost/tasks", handler.GhOstListTasks)
}
// Audit Logs
auth.GET("/audit-logs", handler.ListAuditLogs)
// Query execution (auditor excluded)
queryGroup := auth.Group("")
queryGroup.Use(middleware.RequireRole("admin", "dba", "developer"))
{
queryGroup.POST("/query/execute", handler.ExecuteQuery)
queryGroup.POST("/query", handler.ExecuteQuery)
queryGroup.POST("/query/export", handler.ExportQueryCSV)
}
// SQL Assistant
auth.POST("/sql-assistant/format", handler.SQLFormat)
auth.POST("/sql-assistant/analyze", handler.SQLAnalyze)
// Ticket Audit
auth.POST("/tickets/audit", handler.AuditTicketSQL)
// Dictionary (stub)
auth.GET("/dictionary/instances/:instanceId/databases", handler.DictionaryDatabases)
auth.GET("/dictionary/instances/:instanceId/tables", handler.DictionaryTables)
auth.GET("/dictionary/instances/:instanceId/tables/:tableName", handler.DictionaryTableDetail)
auth.GET("/dictionary/instances/:instanceId/export", handler.DictionaryExport)
// MongoDB (stub)
auth.GET("/mongo/:id/databases", handler.MongoDatabases)
auth.GET("/mongo/:id/databases/:database/collections", handler.MongoCollections)
auth.GET("/mongo/:id/collections/:collection/stats", handler.MongoCollectionStats)
auth.POST("/mongo/:id/query", handler.MongoQuery)
auth.POST("/mongo/:id/execute", handler.MongoExecute)
auth.POST("/mongo/:id/test-connection", handler.MongoTestConnection)
// Audit rules — read for all, write for admin/dba
auth.GET("/audit-rules", handler.ListAuditRules)
ruleAdmin := auth.Group("")
ruleAdmin.Use(middleware.RequireRole("admin", "dba"))
{
ruleAdmin.POST("/audit-rules/seed/:dbType", handler.SeedAuditRules)
ruleAdmin.POST("/audit-rules/:id/toggle", handler.ToggleAuditRule)
ruleAdmin.PUT("/audit-rules/:id", handler.UpdateAuditRule)
}
auth.GET("/announcements", handler.ListAnnouncements)
auth.GET("/query-histories", handler.ListQueryHistories)
auth.PUT("/query-histories/:id/favorite", handler.ToggleQueryHistoryFavorite)
// Notifications CRUD (replaces old channels/templates endpoints)
auth.GET("/notifications", handler.ListNotifications)
auth.POST("/notifications", handler.CreateNotification)
auth.PUT("/notifications/:id", handler.UpdateNotification)
auth.DELETE("/notifications/:id", handler.DeleteNotification)
auth.POST("/notifications/:id/test", handler.TestNotificationSend)
auth.GET("/workflows", handler.ListWorkflows)
auth.GET("/workflows/:id", handler.GetWorkflow)
auth.POST("/workflows/:id/clone", handler.CloneWorkflow)
// Workflow management (admin/dba only)
wfAdmin := auth.Group("")
wfAdmin.Use(middleware.RequireRole("admin", "dba"))
{
wfAdmin.POST("/workflows", handler.CreateWorkflow)
wfAdmin.PUT("/workflows/:id", handler.UpdateWorkflow)
wfAdmin.DELETE("/workflows/:id", handler.DeleteWorkflow)
wfAdmin.POST("/workflows/:id/set-default", handler.SetDefaultWorkflow)
}
}
}