feat: opsx-loki 全面升级 — 前端重构/用户与LDAP/部署向导/日志查询
基于 WEB-UI-Development-Guide 标准的整体升级,并更名 LokiDeploy → opsx-loki。 前端(Gin+React+shadcn/ui): - 按 02/07 章重构:页面域化拆分(无超 400 行文件)、PageContainer/ListQuery、 反馈分级(成功静默/失败 toast/危险确认)、401 单飞刷新+队列重放 - 暗色模式(next-themes)+ 语义 Token(--primary 品牌色/success/warning/info) - 部署向导(状态推导四步:环境→存储→Loki→采集,内嵌表单闭环) - 日志查询页:LogQL 标签构建器/拖拽直方图/日志上下文/Live Tail/查询历史 - 按用户+标签策略的查询授权(服务端强制合并策略) 后端(Go+Gin+JSON DB): - 简单用户管理(仅 admin,无 RBAC)+ TokenVersion 会话吊销 + 登录 IP 限流 - LDAP 接入:配置(AES 加密存储/hasPassword 脱敏)/测试连接/批量导入/登录 - 存储配置测试连接(SigV4 零依赖)+ 桶不存在自动创建 - Loki 三模式部署修复:SSH 密钥认证、3.x 配置模板、ruler 对象存储嵌套块、 凭证解密、孤儿 deploying 状态恢复、Grafana 部署(可选默认关)+面板导入 - DaemonSet 采集纯 YAML(去 helm 依赖)+ DNS 自愈回退 ClusterIP + 集群名标签 - 日志查询代理:cluster-svc 经 kube-apiserver proxy/url 直连 + cursor 翻页 - 统一 log 命名空间;部署就绪探活;Grafana 访问地址自动记录 修复:run.sh 增量编译检测、Sanitized 脱敏补全、LokiDeployment Update 白名单、 健康检查 CreatedAt 零值等 10+ 存量 bug。 数据兼容:DB 沿用旧 lokideploy.json(新名优先);localStorage 一次性迁移; HKDF 加密前缀保留(已加密凭证不受影响)。
This commit is contained in:
1 parent
327e282356
commit
f9c6c1d826
204 files changed
+14107
-9103
No files matched your search
+186
-11
@@ -1,11 +1,12 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"lokideploy/models"
|
||||
"lokideploy/pkg/crypto"
|
||||
"lokideploy/pkg/jwt"
|
||||
"lokideploy/repository"
|
||||
"lokideploy/utils"
|
||||
"fmt"
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
"opsxloki/pkg/jwt"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
)
|
||||
|
||||
type AuthService struct{}
|
||||
@@ -17,23 +18,89 @@ func NewAuthService() *AuthService {
|
||||
func (s *AuthService) Login(username, password string) (*models.User, *jwt.TokenPair, error) {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
var local *models.User
|
||||
for i := range db.Users {
|
||||
if db.Users[i].Username == username {
|
||||
if !crypto.VerifyPassword(password, db.Users[i].Password) {
|
||||
return nil, nil, utils.ErrInvalidCredentials
|
||||
}
|
||||
tokenPair, err := jwt.GenerateTokenPair(db.Users[i].ID, db.Users[i].Username, db.Users[i].Role)
|
||||
u := db.Users[i]
|
||||
local = &u
|
||||
break
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
|
||||
// 本地用户密码校验(普通本地用户唯一路径;LDAP 用户不使用本地密码)
|
||||
if local != nil && local.AuthSource != "ldap" {
|
||||
if !crypto.VerifyPassword(password, local.Password) {
|
||||
return nil, nil, utils.ErrInvalidCredentials
|
||||
}
|
||||
return s.issueTokenPair(local)
|
||||
}
|
||||
|
||||
// LDAP 用户(本地已导入或首次登录自动创建);enabled=false 或验证失败返回统一错误
|
||||
if ldapCfg := GetLDAPConfig(); ldapCfg.Enabled && ldapCfg.Host != "" {
|
||||
if entry, err := verifyLDAPPassword(ldapCfg, username, password); err == nil {
|
||||
user, err := s.syncLDAPUser(entry)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return &db.Users[i], tokenPair, nil
|
||||
return s.issueTokenPair(user)
|
||||
}
|
||||
}
|
||||
return nil, nil, utils.ErrInvalidCredentials
|
||||
}
|
||||
|
||||
func (s *AuthService) issueTokenPair(u *models.User) (*models.User, *jwt.TokenPair, error) {
|
||||
pair, err := jwt.GenerateTokenPair(u.ID, u.Username, u.Role, u.TokenVersion)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return u, pair, nil
|
||||
}
|
||||
|
||||
// syncLDAPUser LDAP 验证通过后同步本地记录:不存在则创建(user 角色,密码占位不可本地登录),
|
||||
// 存在则同步昵称/邮箱
|
||||
func (s *AuthService) syncLDAPUser(entry *ldapUserEntry) (*models.User, error) {
|
||||
db := repository.GetDB()
|
||||
// 已存在 → 同步昵称并标记 LDAP 来源
|
||||
db.Lock()
|
||||
for i := range db.Users {
|
||||
if db.Users[i].Username == entry.Username {
|
||||
db.Users[i].Nickname = entry.DisplayName
|
||||
db.Users[i].AuthSource = "ldap"
|
||||
db.RequestSave()
|
||||
u := db.Users[i]
|
||||
db.Unlock()
|
||||
return &u, nil
|
||||
}
|
||||
}
|
||||
db.Unlock()
|
||||
|
||||
// 不存在 → 创建(CreateUser 自管锁与 ID 分配)
|
||||
hash, _ := crypto.HashPassword("ldap-user-" + entry.Username + "-no-local-login")
|
||||
user := &models.User{
|
||||
Username: entry.Username,
|
||||
Password: hash,
|
||||
Nickname: entry.DisplayName,
|
||||
Role: "user",
|
||||
AuthSource: "ldap",
|
||||
}
|
||||
if err := db.CreateUser(user); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return user, nil
|
||||
}
|
||||
|
||||
func (s *AuthService) Register(username, password, nickname, role string) (*models.User, error) {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
for i := range db.Users {
|
||||
if db.Users[i].Username == username {
|
||||
db.RUnlock()
|
||||
return nil, utils.ErrUsernameExists
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
|
||||
hash, err := crypto.HashPassword(password)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -70,6 +137,9 @@ func (s *AuthService) ChangePassword(userID uint, oldPassword, newPassword strin
|
||||
defer db.Unlock()
|
||||
for i := range db.Users {
|
||||
if db.Users[i].ID == userID {
|
||||
if db.Users[i].AuthSource == "ldap" {
|
||||
return fmt.Errorf("LDAP 用户请在 LDAP 目录中修改密码")
|
||||
}
|
||||
if !crypto.VerifyPassword(oldPassword, db.Users[i].Password) {
|
||||
return utils.ErrInvalidCredentials
|
||||
}
|
||||
@@ -78,6 +148,111 @@ func (s *AuthService) ChangePassword(userID uint, oldPassword, newPassword strin
|
||||
return err
|
||||
}
|
||||
db.Users[i].Password = hash
|
||||
db.Users[i].TokenVersion++
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return utils.ErrNotFound
|
||||
}
|
||||
|
||||
// ListUsers 分页返回用户列表(按 ID 升序),pageSize 上限 100
|
||||
func (s *AuthService) ListUsers(page, pageSize int) ([]models.User, int64) {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
|
||||
total := int64(len(db.Users))
|
||||
if page < 1 {
|
||||
page = 1
|
||||
}
|
||||
if pageSize < 1 || pageSize > 100 {
|
||||
pageSize = 20
|
||||
}
|
||||
start := (page - 1) * pageSize
|
||||
if start >= int(total) {
|
||||
return []models.User{}, total
|
||||
}
|
||||
end := start + pageSize
|
||||
if end > int(total) {
|
||||
end = int(total)
|
||||
}
|
||||
users := make([]models.User, end-start)
|
||||
copy(users, db.Users[start:end])
|
||||
return users, total
|
||||
}
|
||||
|
||||
// DeleteUser 删除用户;不能删除自己,不能删除最后一个管理员
|
||||
func (s *AuthService) DeleteUser(id, requesterID uint) error {
|
||||
db := repository.GetDB()
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
|
||||
idx := -1
|
||||
adminCount := 0
|
||||
for i := range db.Users {
|
||||
if db.Users[i].ID == id {
|
||||
idx = i
|
||||
}
|
||||
if db.Users[i].Role == "admin" {
|
||||
adminCount++
|
||||
}
|
||||
}
|
||||
if idx == -1 {
|
||||
return utils.ErrNotFound
|
||||
}
|
||||
if id == requesterID {
|
||||
return utils.ErrCannotDeleteSelf
|
||||
}
|
||||
if db.Users[idx].Role == "admin" && adminCount <= 1 {
|
||||
return utils.ErrLastAdmin
|
||||
}
|
||||
|
||||
db.Users = append(db.Users[:idx], db.Users[idx+1:]...)
|
||||
for i := range db.LogSources {
|
||||
kept := db.LogSources[i].Access[:0]
|
||||
for _, rule := range db.LogSources[i].Access {
|
||||
if rule.UserID != id {
|
||||
kept = append(kept, rule)
|
||||
}
|
||||
}
|
||||
db.LogSources[i].Access = kept
|
||||
}
|
||||
groups := db.LogPolicyGroups[:0]
|
||||
for _, group := range db.LogPolicyGroups {
|
||||
members := group.UserIDs[:0]
|
||||
for _, member := range group.UserIDs {
|
||||
if member != id {
|
||||
members = append(members, member)
|
||||
}
|
||||
}
|
||||
group.UserIDs = members
|
||||
if len(members) > 0 {
|
||||
groups = append(groups, group)
|
||||
}
|
||||
}
|
||||
db.LogPolicyGroups = groups
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
|
||||
// ResetPassword 管理员重置用户密码,并递增 TokenVersion 吊销其全部旧会话
|
||||
func (s *AuthService) ResetPassword(userID uint, newPassword string) error {
|
||||
db := repository.GetDB()
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
|
||||
for i := range db.Users {
|
||||
if db.Users[i].ID == userID {
|
||||
if db.Users[i].AuthSource == "ldap" {
|
||||
return fmt.Errorf("LDAP 用户密码由目录服务管理,不能在此重置")
|
||||
}
|
||||
hash, err := crypto.HashPassword(newPassword)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
db.Users[i].Password = hash
|
||||
db.Users[i].TokenVersion++
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -7,10 +7,10 @@ import (
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"lokideploy/config"
|
||||
"lokideploy/models"
|
||||
"lokideploy/repository"
|
||||
"lokideploy/utils"
|
||||
"opsxloki/config"
|
||||
"opsxloki/models"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
)
|
||||
|
||||
type ClusterService struct{}
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
"os/exec"
|
||||
"strings"
|
||||
|
||||
"lokideploy/models"
|
||||
"opsxloki/models"
|
||||
)
|
||||
|
||||
type ClusterInfo struct {
|
||||
|
||||
@@ -5,9 +5,9 @@ import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"lokideploy/models"
|
||||
"lokideploy/repository"
|
||||
"lokideploy/utils"
|
||||
"opsxloki/models"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
)
|
||||
|
||||
type CollectorService struct{}
|
||||
@@ -66,11 +66,15 @@ func (s *CollectorService) generateConfigSnapshot(c *models.Collector) string {
|
||||
if c.LokiDeployment != nil {
|
||||
endpoint = c.LokiDeployment.Endpoint
|
||||
}
|
||||
clusterName := "host"
|
||||
if c.Cluster != nil && c.Cluster.Name != "" {
|
||||
clusterName = c.Cluster.Name
|
||||
}
|
||||
switch c.Type {
|
||||
case "host-promtail":
|
||||
return PromtailConfig(endpoint, c.LogPath)
|
||||
return PromtailConfig(endpoint, c.LogPath, "host", c.Labels)
|
||||
case "k8s-daemonset":
|
||||
return PromtailHelmValues(endpoint)
|
||||
return PromtailHelmValues(endpoint, clusterName, c.Labels)
|
||||
case "k8s-fluentbit-sidecar":
|
||||
host, port := parseLokiURL(endpoint)
|
||||
app := c.TargetDeployment
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"os/exec"
|
||||
"strings"
|
||||
|
||||
"lokideploy/models"
|
||||
"opsxloki/models"
|
||||
)
|
||||
|
||||
// CollectorDeployer 采集器部署器,按 type 分发到三种安装/卸载流程
|
||||
@@ -97,11 +97,7 @@ func (d *CollectorDeployer) installHost(ctx context.Context, endpoint string) er
|
||||
if host == nil {
|
||||
return fmt.Errorf("主机采集器缺少目标主机")
|
||||
}
|
||||
password, err := DecryptCred(host.Password)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
client, err := NewSSHClient(host.HostIP, host.Port, host.User, password)
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -145,7 +141,7 @@ func (d *CollectorDeployer) installHost(ctx context.Context, endpoint string) er
|
||||
if logPath == "" {
|
||||
logPath = "/var/log/*.log"
|
||||
}
|
||||
conf := PromtailConfig(endpoint, logPath)
|
||||
conf := PromtailConfig(endpoint, logPath, "host", d.collector.Labels)
|
||||
d.log("info", "上传 Promtail 配置", "")
|
||||
if _, _, err := client.Execute(ctx, fmt.Sprintf("cat > /etc/promtail/promtail.yaml << 'EOF'\n%s\nEOF", conf)); err != nil {
|
||||
return fmt.Errorf("上传配置失败: %w", err)
|
||||
@@ -191,8 +187,7 @@ func (d *CollectorDeployer) uninstallHost(ctx context.Context) error {
|
||||
if host == nil {
|
||||
return nil
|
||||
}
|
||||
password, _ := DecryptCred(host.Password)
|
||||
client, err := NewSSHClient(host.HostIP, host.Port, host.User, password)
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -209,7 +204,7 @@ func (d *CollectorDeployer) uninstallHost(ctx context.Context) error {
|
||||
func (d *CollectorDeployer) installDaemonset(ctx context.Context, endpoint string) error {
|
||||
cluster := d.collector.Cluster
|
||||
if cluster == nil {
|
||||
return fmt.Errorf("DaemonSet 采集器缺少目标集群")
|
||||
return fmt.Errorf("集群采集器缺少目标集群")
|
||||
}
|
||||
env, cleanup, err := kubeconfigEnv(cluster)
|
||||
if err != nil {
|
||||
@@ -217,37 +212,106 @@ func (d *CollectorDeployer) installDaemonset(ctx context.Context, endpoint strin
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
// 默认部署到关联 Loki 的命名空间(统一 log 命名空间策略)
|
||||
ns := d.collector.Namespace
|
||||
if ns == "" && d.loki != nil {
|
||||
ns = d.loki.Namespace
|
||||
}
|
||||
if ns == "" {
|
||||
ns = "logging"
|
||||
ns = "log"
|
||||
}
|
||||
|
||||
values := PromtailHelmValues(endpoint)
|
||||
valuesFile, valuesCleanup, err := writeTempFile("promtail-values-*.yaml", values, 0600)
|
||||
clusterName := "unknown"
|
||||
if cluster.Name != "" {
|
||||
clusterName = cluster.Name
|
||||
}
|
||||
|
||||
// 幂等:先清掉同名残留(如历史失败安装留下的无注解资源)
|
||||
del := exec.CommandContext(ctx, "kubectl", "delete",
|
||||
"daemonset/promtail", "configmap/promtail", "serviceaccount/promtail",
|
||||
"-n", ns, "--ignore-not-found=true", "--wait=false")
|
||||
del.Env = env
|
||||
del.CombinedOutput()
|
||||
|
||||
// 纯 YAML 安装(SA/RBAC/ConfigMap/DaemonSet),不依赖后端安装 helm
|
||||
manifest := PromtailDaemonsetYAML(ns, endpoint, clusterName, d.collector.Labels)
|
||||
if err := d.applyYAML(ctx, env, manifest, "promtail"); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
d.log("info", fmt.Sprintf("等待 DaemonSet promtail 就绪(命名空间 %s)...", ns), "")
|
||||
waitCmd := exec.CommandContext(ctx, "kubectl", "rollout", "status", "daemonset/promtail",
|
||||
"-n", ns, "--timeout=5m")
|
||||
waitCmd.Env = env
|
||||
waitOut, _ := waitCmd.CombinedOutput()
|
||||
d.log("info", string(waitOut), "")
|
||||
|
||||
// DNS 自愈:CoreDNS 异常的集群上服务名不可解析,回退 Loki ClusterIP 重建配置
|
||||
svcURL := endpoint
|
||||
if strings.Contains(endpoint, ".svc") {
|
||||
probe := exec.CommandContext(ctx, "kubectl", "exec", "-n", ns,
|
||||
"daemonset/promtail", "--", "wget", "-q", "-O", "/dev/null", "-T", "5", endpoint+"/ready")
|
||||
probe.Env = env
|
||||
if pout, perr := probe.CombinedOutput(); perr != nil {
|
||||
d.log("warn", fmt.Sprintf("promtail 无法解析 Loki 服务名(%s),回退 ClusterIP...", strings.TrimSpace(string(pout))), "")
|
||||
ip := d.lokiClusterIP(ctx, env, endpoint)
|
||||
if ip != "" {
|
||||
// 只替换 host 部分,保留 scheme/port/path: http://loki.ns.svc:3100/... → http://IP:3100/...
|
||||
afterScheme := strings.SplitN(endpoint, "//", 2)
|
||||
if len(afterScheme) == 2 {
|
||||
rest := afterScheme[1]
|
||||
host := rest
|
||||
if i := strings.IndexAny(rest, ":/"); i >= 0 {
|
||||
host = rest[:i]
|
||||
}
|
||||
fallback := afterScheme[0] + "//" + strings.Replace(rest, host, ip, 1)
|
||||
if err := d.applyYAML(ctx, env, PromtailDaemonsetYAML(ns, fallback, clusterName, d.collector.Labels), "promtail"); err == nil {
|
||||
rs := exec.CommandContext(ctx, "kubectl", "rollout", "restart", "daemonset/promtail", "-n", ns)
|
||||
rs.Env = env
|
||||
rs.CombinedOutput()
|
||||
d.log("info", fmt.Sprintf("promtail 推送地址已回退为 ClusterIP: %s", fallback), "")
|
||||
svcURL = fallback
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
d.collector.Config = svcURL
|
||||
d.log("success", fmt.Sprintf("Promtail DaemonSet 已部署到 %s(推送至 %s)", ns, svcURL), "")
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyYAML kubectl apply 一个 manifest(临时文件)
|
||||
func (d *CollectorDeployer) applyYAML(ctx context.Context, env []string, manifest, label string) error {
|
||||
f, cleanup, err := writeTempFile("collector-*.yaml", manifest, 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer valuesCleanup()
|
||||
|
||||
d.log("info", fmt.Sprintf("helm install promtail(命名空间 %s)", ns), "")
|
||||
// 添加 grafana repo(幂等)
|
||||
repo := exec.CommandContext(ctx, "helm", "repo", "add", "grafana", "https://grafana.github.io/helm-charts")
|
||||
repo.Env = env
|
||||
repo.CombinedOutput()
|
||||
upd := exec.CommandContext(ctx, "helm", "repo", "update")
|
||||
upd.Env = env
|
||||
upd.CombinedOutput()
|
||||
|
||||
cmd := exec.CommandContext(ctx, "helm", "upgrade", "--install", "promtail", "grafana/promtail",
|
||||
"-f", valuesFile, "--namespace", ns, "--create-namespace", "--wait", "--timeout", "5m")
|
||||
defer cleanup()
|
||||
cmd := exec.CommandContext(ctx, "kubectl", "apply", "-f", f)
|
||||
cmd.Env = env
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("helm 安装 Promtail 失败: %w\n%s", err, string(out))
|
||||
d.log("error", fmt.Sprintf("kubectl apply %s failed: %v", label, err), string(out))
|
||||
return fmt.Errorf("kubectl apply %s failed: %w\n%s", label, err, string(out))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// lokiClusterIP 从 endpoint 的命名空间取 loki Service 的 ClusterIP(不经 DNS)
|
||||
func (d *CollectorDeployer) lokiClusterIP(ctx context.Context, env []string, endpoint string) string {
|
||||
ns := "log"
|
||||
if d.loki != nil && d.loki.Namespace != "" {
|
||||
ns = d.loki.Namespace
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "kubectl", "get", "svc", "loki", "-n", ns,
|
||||
"-o", "jsonpath={.spec.clusterIP}")
|
||||
cmd.Env = env
|
||||
out, _ := cmd.CombinedOutput()
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
|
||||
func (d *CollectorDeployer) uninstallDaemonset(ctx context.Context) error {
|
||||
cluster := d.collector.Cluster
|
||||
if cluster == nil {
|
||||
@@ -259,10 +323,16 @@ func (d *CollectorDeployer) uninstallDaemonset(ctx context.Context) error {
|
||||
}
|
||||
defer cleanup()
|
||||
ns := d.collector.Namespace
|
||||
if ns == "" {
|
||||
ns = "logging"
|
||||
if ns == "" && d.loki != nil {
|
||||
ns = d.loki.Namespace
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "helm", "uninstall", "promtail", "--namespace", ns)
|
||||
if ns == "" {
|
||||
ns = "log"
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "kubectl", "delete",
|
||||
"daemonset/promtail", "configmap/promtail", "serviceaccount/promtail",
|
||||
"clusterrole/opsxloki-promtail", "clusterrolebinding/opsxloki-promtail",
|
||||
"-n", ns, "--ignore-not-found=true")
|
||||
cmd.Env = env
|
||||
cmd.CombinedOutput()
|
||||
return nil
|
||||
|
||||
@@ -1,14 +1,35 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// PromtailConfig 主机 Promtail 配置(推送至 lokiEndpoint,采集 logPath)
|
||||
// 生产级配置:持久化位置文件、批处理、重试、CRI日志格式解析
|
||||
func PromtailConfig(lokiEndpoint, logPath string) string {
|
||||
return fmt.Sprintf(`server:
|
||||
// promtailExternalLabels 组并 cluster 标识与用户自定义标签(JSON)为 external_labels YAML 行
|
||||
func promtailExternalLabels(cluster string, labelsJSON string) string {
|
||||
lines := []string{fmt.Sprintf(" cluster: %q", cluster)}
|
||||
if labelsJSON != "" {
|
||||
var m map[string]string
|
||||
if err := json.Unmarshal([]byte(labelsJSON), &m); err == nil {
|
||||
keys := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
lines = append(lines, fmt.Sprintf(" %s: %q", k, m[k]))
|
||||
}
|
||||
}
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
func PromtailConfig(lokiEndpoint, logPath, cluster, labelsJSON string) string {
|
||||
conf := fmt.Sprintf(`server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
log_level: info
|
||||
@@ -29,6 +50,7 @@ clients:
|
||||
max_retries: 10
|
||||
external_labels:
|
||||
job: host-logs
|
||||
__EXT_LABELS__
|
||||
|
||||
limits_config:
|
||||
readline_rate_enabled: false
|
||||
@@ -57,12 +79,13 @@ scrape_configs:
|
||||
burst: 10000
|
||||
drop: true
|
||||
`, lokiEndpoint, logPath)
|
||||
return strings.Replace(conf, "__EXT_LABELS__", promtailExternalLabels(cluster, labelsJSON), 1)
|
||||
}
|
||||
|
||||
// PromtailHelmValues DaemonSet 模式 Promtail Helm values(生产级)
|
||||
// 包含:cri解析、kubernetes标签、批处理配置、资源限制
|
||||
func PromtailHelmValues(lokiEndpoint string) string {
|
||||
return fmt.Sprintf(`config:
|
||||
func PromtailHelmValues(lokiEndpoint, cluster, labelsJSON string) string {
|
||||
values := fmt.Sprintf(`config:
|
||||
clients:
|
||||
- url: %s/loki/api/v1/push
|
||||
batchwait: 1s
|
||||
@@ -73,7 +96,7 @@ func PromtailHelmValues(lokiEndpoint string) string {
|
||||
max_period: 5m
|
||||
max_retries: 10
|
||||
external_labels:
|
||||
cluster: ${CLUSTER_NAME:-unknown}
|
||||
__EXT_LABELS__
|
||||
positions:
|
||||
filename: /run/promtail/positions.yaml
|
||||
scrape_configs:
|
||||
@@ -164,6 +187,160 @@ extraVolumeMounts:
|
||||
- name: promtail-run
|
||||
mountPath: /run/promtail
|
||||
`, lokiEndpoint)
|
||||
return strings.Replace(values, "__EXT_LABELS__", promtailExternalLabels(cluster, labelsJSON), 1)
|
||||
}
|
||||
|
||||
// PromtailDaemonsetYAML DaemonSet 模式纯 YAML 安装(SA/RBAC/ConfigMap/DaemonSet),
|
||||
// 取代 helm 安装 —— 后端无需安装 helm 二进制。
|
||||
// relabel 体系与主流 helm chart 对齐:namespace/pod/container/node_name + __path__ 按 pod uid 精确匹配。
|
||||
func PromtailDaemonsetYAML(ns, lokiURL, clusterName, labelsJSON string) string {
|
||||
extLabels := promtailExternalLabels(clusterName, labelsJSON)
|
||||
return fmt.Sprintf(`apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: promtail
|
||||
namespace: %s
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: opsxloki-promtail
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "nodes"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: opsxloki-promtail
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: opsxloki-promtail
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: promtail
|
||||
namespace: %s
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: promtail
|
||||
namespace: %s
|
||||
data:
|
||||
promtail.yaml: |
|
||||
server:
|
||||
http_listen_port: 3101
|
||||
grpc_listen_port: 0
|
||||
positions:
|
||||
filename: /run/promtail/positions.yaml
|
||||
clients:
|
||||
- url: %s/loki/api/v1/push
|
||||
batchwait: 1s
|
||||
batchsize: 1048576
|
||||
backoff_config:
|
||||
min_period: 500ms
|
||||
max_period: 5m
|
||||
max_retries: 10
|
||||
external_labels:
|
||||
job: promtail
|
||||
%s
|
||||
scrape_configs:
|
||||
- job_name: kubernetes-pods
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
target_label: container
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node_name
|
||||
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name, __meta_kubernetes_pod_label_app]
|
||||
regex: ^;*([^;]+)(;.*)?$
|
||||
target_label: app
|
||||
- separator: /
|
||||
source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app_kubernetes_io_name]
|
||||
target_label: job
|
||||
- source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
|
||||
separator: /
|
||||
target_label: __path__
|
||||
replacement: /var/log/pods/*$1/*.log
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: promtail
|
||||
namespace: %s
|
||||
labels:
|
||||
app: promtail
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: promtail
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: promtail
|
||||
spec:
|
||||
serviceAccountName: promtail
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
operator: Exists
|
||||
containers:
|
||||
- name: promtail
|
||||
image: grafana/promtail:3.4.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
args: ["-config.file=/etc/promtail/promtail.yaml"]
|
||||
env:
|
||||
- name: HOSTNAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: spec.nodeName
|
||||
ports:
|
||||
- name: http-metrics
|
||||
containerPort: 3101
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: http-metrics
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/promtail
|
||||
- name: run
|
||||
mountPath: /run/promtail
|
||||
- name: pods
|
||||
mountPath: /var/log/pods
|
||||
readOnly: true
|
||||
- name: containers
|
||||
mountPath: /var/lib/docker/containers
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: promtail
|
||||
- name: run
|
||||
hostPath:
|
||||
path: /run/promtail
|
||||
- name: pods
|
||||
hostPath:
|
||||
path: /var/log/pods
|
||||
- name: containers
|
||||
hostPath:
|
||||
path: /var/lib/docker/containers
|
||||
`, ns, ns, ns, lokiURL, extLabels, ns)
|
||||
}
|
||||
|
||||
// FluentbitConfig Sidecar 模式 Fluent Bit 配置(采集 logPath,推送至 lokiHost:lokiPort)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"os/exec"
|
||||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
@@ -8,9 +9,9 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"lokideploy/models"
|
||||
"lokideploy/repository"
|
||||
"lokideploy/utils"
|
||||
"opsxloki/models"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
)
|
||||
|
||||
// LokiVersionList 支持的 Loki 版本列表(按推荐程度排序)
|
||||
@@ -50,6 +51,8 @@ type CreateDeploymentRequest struct {
|
||||
ResourcesCPU string `json:"resourcesCPU"`
|
||||
ResourcesMem string `json:"resourcesMem"`
|
||||
StorageSize string `json:"storageSize"`
|
||||
GrafanaEnabled bool `json:"grafanaEnabled"`
|
||||
GrafanaVersion string `json:"grafanaVersion"`
|
||||
}
|
||||
|
||||
func (s *DeploymentService) Create(req *CreateDeploymentRequest) (*models.LokiDeployment, error) {
|
||||
@@ -77,6 +80,8 @@ func (s *DeploymentService) Create(req *CreateDeploymentRequest) (*models.LokiDe
|
||||
ResourcesCPU: req.ResourcesCPU,
|
||||
ResourcesMem: req.ResourcesMem,
|
||||
StorageSize: req.StorageSize,
|
||||
GrafanaEnabled: req.GrafanaEnabled,
|
||||
GrafanaVersion: req.GrafanaVersion,
|
||||
Status: "pending",
|
||||
CreatedAt: time.Now(),
|
||||
UpdatedAt: time.Now(),
|
||||
@@ -106,6 +111,13 @@ func (s *DeploymentService) Create(req *CreateDeploymentRequest) (*models.LokiDe
|
||||
for i := range db.Storages {
|
||||
if db.Storages[i].ID == d.StorageConfigID {
|
||||
st := db.Storages[i]
|
||||
// 模型内为 ENC: 密文,生成 Loki 配置前解密(否则密文进 ConfigMap 且无法通过对象存储认证)
|
||||
if ak, err := DecryptCred(st.AccessKey); err == nil && ak != "" {
|
||||
st.AccessKey = ak
|
||||
}
|
||||
if sk, err := DecryptCred(st.SecretKey); err == nil && sk != "" {
|
||||
st.SecretKey = sk
|
||||
}
|
||||
d.StorageConfig = &st
|
||||
break
|
||||
}
|
||||
@@ -171,6 +183,57 @@ func (s *DeploymentService) Update(id uint, updates map[string]interface{}) erro
|
||||
if v, ok := updates["endpoint"].(string); ok {
|
||||
db.Deployments[i].Endpoint = v
|
||||
}
|
||||
// 业务字段(来自 PUT /deployments/:id;JSON 数字解码为 float64)
|
||||
if v, ok := updates["name"].(string); ok && v != "" {
|
||||
db.Deployments[i].Name = v
|
||||
}
|
||||
if v, ok := updates["description"].(string); ok {
|
||||
db.Deployments[i].Description = v
|
||||
}
|
||||
if v, ok := updates["version"].(string); ok && v != "" {
|
||||
db.Deployments[i].Version = v
|
||||
}
|
||||
if v, ok := updates["namespace"].(string); ok && v != "" {
|
||||
db.Deployments[i].Namespace = v
|
||||
}
|
||||
if v, ok := updates["resourcesCPU"].(string); ok {
|
||||
db.Deployments[i].ResourcesCPU = v
|
||||
}
|
||||
if v, ok := updates["resourcesMem"].(string); ok {
|
||||
db.Deployments[i].ResourcesMem = v
|
||||
}
|
||||
if v, ok := updates["storageSize"].(string); ok && v != "" {
|
||||
db.Deployments[i].StorageSize = v
|
||||
}
|
||||
if v, ok := updates["authEnabled"].(bool); ok {
|
||||
db.Deployments[i].AuthEnabled = v
|
||||
}
|
||||
if v, ok := updates["grafanaEnabled"].(bool); ok {
|
||||
db.Deployments[i].GrafanaEnabled = v
|
||||
}
|
||||
if v, ok := updates["grafanaVersion"].(string); ok && v != "" {
|
||||
db.Deployments[i].GrafanaVersion = v
|
||||
}
|
||||
if v, ok := updates["grafanaEndpoint"].(string); ok {
|
||||
db.Deployments[i].GrafanaEndpoint = v
|
||||
}
|
||||
if v, ok := updates["storageConfigID"].(float64); ok && v > 0 {
|
||||
db.Deployments[i].StorageConfigID = uint(v)
|
||||
}
|
||||
if v, ok := updates["clusterID"].(float64); ok && v > 0 {
|
||||
id := uint(v)
|
||||
db.Deployments[i].ClusterID = &id
|
||||
}
|
||||
if v, ok := updates["hostID"].(float64); ok && v > 0 {
|
||||
id := uint(v)
|
||||
db.Deployments[i].HostID = &id
|
||||
}
|
||||
if v, ok := updates["retentionDays"].(float64); ok && v > 0 {
|
||||
db.Deployments[i].RetentionDays = int(v)
|
||||
}
|
||||
if v, ok := updates["replicas"].(float64); ok && v > 0 {
|
||||
db.Deployments[i].Replicas = int(v)
|
||||
}
|
||||
db.Deployments[i].UpdatedAt = time.Now()
|
||||
db.RequestSave()
|
||||
return nil
|
||||
@@ -260,6 +323,11 @@ func (s *DeploymentService) StartDeploy(id uint) error {
|
||||
"statusMessage": "Deployment completed successfully",
|
||||
"endpoint": s.getEndpoint(deployment),
|
||||
})
|
||||
// 部署成功自动注册日志查询数据源 + 记录 Grafana 访问地址
|
||||
if fresh, gerr := s.Get(id); gerr == nil {
|
||||
EnsureLogSourceForDeployment(fresh)
|
||||
go s.recordGrafanaEndpoint(fresh)
|
||||
}
|
||||
s.AddLog(id, "success", "Deployment completed successfully", "")
|
||||
}
|
||||
}()
|
||||
@@ -398,6 +466,13 @@ func (s *DeploymentService) loadRelations(d *models.LokiDeployment) {
|
||||
for i := range db.Storages {
|
||||
if db.Storages[i].ID == d.StorageConfigID {
|
||||
st := db.Storages[i]
|
||||
// 模型内为 ENC: 密文,生成 Loki 配置前解密(否则密文进 ConfigMap 且无法通过对象存储认证)
|
||||
if ak, err := DecryptCred(st.AccessKey); err == nil && ak != "" {
|
||||
st.AccessKey = ak
|
||||
}
|
||||
if sk, err := DecryptCred(st.SecretKey); err == nil && sk != "" {
|
||||
st.SecretKey = sk
|
||||
}
|
||||
d.StorageConfig = &st
|
||||
break
|
||||
}
|
||||
@@ -472,3 +547,126 @@ func GetRecommendedVersions() []string {
|
||||
copy(result, LokiVersionList)
|
||||
return result
|
||||
}
|
||||
|
||||
// Preflight 向导预检:聚合各资源真实状态,前端据此推导步骤完成态(状态推导式向导)
|
||||
func (s *DeploymentService) Preflight() map[string]interface{} {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
|
||||
result := map[string]interface{}{
|
||||
"hasStorage": len(db.Storages) > 0,
|
||||
"hasActiveCluster": false,
|
||||
"clusterNames": []string{},
|
||||
"hasActiveHost": false,
|
||||
"hostNames": []string{},
|
||||
"runningLoki": nil,
|
||||
"activeCollectors": 0,
|
||||
"grafanaReady": false,
|
||||
}
|
||||
|
||||
var defaultStorage string
|
||||
for i := range db.Storages {
|
||||
if db.Storages[i].IsDefault {
|
||||
defaultStorage = db.Storages[i].Name
|
||||
break
|
||||
}
|
||||
}
|
||||
if defaultStorage == "" && len(db.Storages) > 0 {
|
||||
defaultStorage = db.Storages[0].Name
|
||||
}
|
||||
result["defaultStorage"] = defaultStorage
|
||||
|
||||
clusters := []string{}
|
||||
for i := range db.Clusters {
|
||||
clusters = append(clusters, db.Clusters[i].Name)
|
||||
if db.Clusters[i].Status == "active" || db.Clusters[i].Status == "running" {
|
||||
result["hasActiveCluster"] = true
|
||||
}
|
||||
}
|
||||
result["clusterNames"] = clusters
|
||||
|
||||
hosts := []string{}
|
||||
for i := range db.Hosts {
|
||||
hosts = append(hosts, db.Hosts[i].Name)
|
||||
if db.Hosts[i].Status == "active" {
|
||||
result["hasActiveHost"] = true
|
||||
}
|
||||
}
|
||||
result["hostNames"] = hosts
|
||||
|
||||
for i := range db.Deployments {
|
||||
d := &db.Deployments[i]
|
||||
if d.Status == "running" {
|
||||
loki := map[string]interface{}{
|
||||
"id": d.ID,
|
||||
"name": d.Name,
|
||||
"mode": d.Mode,
|
||||
"endpoint": d.Endpoint,
|
||||
"namespace": d.Namespace,
|
||||
"grafana": d.GrafanaEnabled,
|
||||
"grafanaEndpoint": d.GrafanaEndpoint,
|
||||
}
|
||||
result["runningLoki"] = loki
|
||||
if d.GrafanaEnabled {
|
||||
result["grafanaReady"] = true
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
for i := range db.Collectors {
|
||||
if db.Collectors[i].Status == "active" {
|
||||
result["activeCollectors"] = result["activeCollectors"].(int) + 1
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// recordGrafanaEndpoint 记录 Grafana 可点击访问地址(k8s: 首节点 IP + NodePort;host: 主机 IP:3000),
|
||||
// 供部署卡片/向导直接展示,用户不用翻日志找端口。
|
||||
func (s *DeploymentService) recordGrafanaEndpoint(d *models.LokiDeployment) {
|
||||
if d == nil || !d.GrafanaEnabled {
|
||||
return
|
||||
}
|
||||
var endpoint string
|
||||
switch d.Mode {
|
||||
case "k8s-helm", "k8s-yaml":
|
||||
if d.Cluster == nil {
|
||||
s.AddLog(d.ID, "warn", "Grafana 地址记录跳过:未关联集群", "")
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
env, cleanup, err := kubeconfigEnv(d.Cluster)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer cleanup()
|
||||
ns := d.Namespace
|
||||
if ns == "" {
|
||||
ns = "log"
|
||||
}
|
||||
ipCmd := exec.CommandContext(ctx, "kubectl", "get", "nodes",
|
||||
"-o", "jsonpath={.items[0].status.addresses[0].address}")
|
||||
ipCmd.Env = env
|
||||
ipOut, _ := ipCmd.CombinedOutput()
|
||||
portCmd := exec.CommandContext(ctx, "kubectl", "get", "svc", "grafana", "-n", ns,
|
||||
"-o", "jsonpath={.spec.ports[0].nodePort}")
|
||||
portCmd.Env = env
|
||||
portOut, _ := portCmd.CombinedOutput()
|
||||
ip := strings.TrimSpace(string(ipOut))
|
||||
port := strings.TrimSpace(string(portOut))
|
||||
s.AddLog(d.ID, "info", fmt.Sprintf("Grafana 地址探测: node=%s port=%s", ip, port), "")
|
||||
if ip != "" && port != "" {
|
||||
endpoint = fmt.Sprintf("http://%s:%s", ip, port)
|
||||
}
|
||||
case "host-binary":
|
||||
if d.Host != nil {
|
||||
endpoint = fmt.Sprintf("http://%s:3000", d.Host.HostIP)
|
||||
}
|
||||
}
|
||||
if endpoint != "" {
|
||||
_ = s.Update(d.ID, map[string]interface{}{"grafanaEndpoint": endpoint})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,438 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"opsxloki/models"
|
||||
)
|
||||
|
||||
// GrafanaDeployer 与 Loki 同环境部署 Grafana 并自动配置 Loki 数据源。
|
||||
// k8s 模式:manifest(Deployment + NodePort Service + 数据源 ConfigMap),与 Loki 同 namespace;
|
||||
// host-binary 模式:tarball + systemd,与 Loki 同机。
|
||||
type GrafanaDeployer struct {
|
||||
deployment *models.LokiDeployment
|
||||
log func(level, message, detail string)
|
||||
}
|
||||
|
||||
const defaultGrafanaVersion = "11.3.0"
|
||||
|
||||
func (d *GrafanaDeployer) version() string {
|
||||
if d.deployment.GrafanaVersion != "" {
|
||||
return d.deployment.GrafanaVersion
|
||||
}
|
||||
return defaultGrafanaVersion
|
||||
}
|
||||
|
||||
// lokiPushURL 数据源指向的 Loki 地址(k8s 走集群内服务名,主机走本机)
|
||||
func (d *GrafanaDeployer) lokiPushURL() string {
|
||||
if d.deployment.Mode == "host-binary" {
|
||||
return "http://127.0.0.1:3100"
|
||||
}
|
||||
ns := d.deployment.Namespace
|
||||
if ns == "" {
|
||||
ns = "log"
|
||||
}
|
||||
return fmt.Sprintf("http://loki.%s.svc.cluster.local:3100", ns)
|
||||
}
|
||||
|
||||
// Deploy 部署入口:按模式分发
|
||||
func (d *GrafanaDeployer) Deploy(ctx context.Context) error {
|
||||
d.log("info", fmt.Sprintf("Deploying Grafana %s (mode: %s)...", d.version(), d.deployment.Mode), "")
|
||||
switch d.deployment.Mode {
|
||||
case "k8s-helm", "k8s-yaml":
|
||||
return d.deployK8s(ctx)
|
||||
case "host-binary":
|
||||
return d.deployHost(ctx)
|
||||
default:
|
||||
return fmt.Errorf("unsupported mode for grafana: %s", d.deployment.Mode)
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== K8s manifest 部署 ====================
|
||||
|
||||
// grafanaManifest 生成 Grafana 资源;lokiURL 显式传入(默认服务名,DNS 异常的集群可回退 ClusterIP)。
|
||||
// 数据卷用 emptyDir:数据源/配置由 ConfigMap 供给,重启即恢复;避免无默认
|
||||
// StorageClass 的集群上 PVC Pending 阻塞(UI 内自建的面板需重新导入,见已知限制)。
|
||||
func (d *GrafanaDeployer) grafanaManifest(ns, lokiURL string) string {
|
||||
version := d.version()
|
||||
|
||||
datasources := fmt.Sprintf(`apiVersion: 1
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
uid: loki-ds
|
||||
access: proxy
|
||||
url: %s
|
||||
isDefault: true
|
||||
editable: false`, lokiURL)
|
||||
|
||||
return fmt.Sprintf(`apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-datasources
|
||||
namespace: %s
|
||||
data:
|
||||
datasources.yaml: |
|
||||
%s
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: %s
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: grafana
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
containers:
|
||||
- name: grafana
|
||||
image: grafana/grafana:%s
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3000
|
||||
env:
|
||||
- name: GF_SECURITY_ADMIN_USER
|
||||
value: admin
|
||||
- name: GF_SECURITY_ADMIN_PASSWORD
|
||||
value: admin
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
volumeMounts:
|
||||
- name: datasources
|
||||
mountPath: /etc/grafana/provisioning/datasources
|
||||
volumes:
|
||||
- name: datasources
|
||||
configMap:
|
||||
name: grafana-datasources
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: grafana
|
||||
namespace: %s
|
||||
labels:
|
||||
app: grafana
|
||||
spec:
|
||||
# NodePort 便于集群外直接访问(端口由集群自动分配,实际端口见部署日志)
|
||||
type: NodePort
|
||||
ports:
|
||||
- name: http
|
||||
port: 3000
|
||||
targetPort: 3000
|
||||
selector:
|
||||
app: grafana
|
||||
`, ns, indentBlock(datasources, " "), ns, version, ns)
|
||||
}
|
||||
|
||||
func (d *GrafanaDeployer) deployK8s(ctx context.Context) error {
|
||||
if d.deployment.Cluster == nil {
|
||||
return fmt.Errorf("please select target cluster")
|
||||
}
|
||||
env, cleanup, err := kubeconfigEnv(d.deployment.Cluster)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
ns := d.deployment.Namespace
|
||||
if ns == "" {
|
||||
ns = "log"
|
||||
}
|
||||
|
||||
lokiURL := d.lokiPushURL()
|
||||
manifestFile, manifestCleanup, err := writeTempFile("grafana-*.yaml", d.grafanaManifest(ns, lokiURL), 0600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer manifestCleanup()
|
||||
|
||||
d.log("info", "Applying Grafana manifest (Deployment + NodePort Service + datasource ConfigMap)...", "")
|
||||
cmd := exec.CommandContext(ctx, "kubectl", "apply", "-f", manifestFile)
|
||||
cmd.Env = env
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
d.log("error", fmt.Sprintf("kubectl apply grafana failed: %v", err), string(out))
|
||||
return fmt.Errorf("kubectl apply grafana failed: %w\n%s", err, string(out))
|
||||
}
|
||||
|
||||
waitCmd := exec.CommandContext(ctx, "kubectl", "rollout", "status", "deployment/grafana",
|
||||
"-n", ns, "--timeout=5m")
|
||||
waitCmd.Env = env
|
||||
waitOut, _ := waitCmd.CombinedOutput()
|
||||
d.log("info", string(waitOut), "")
|
||||
|
||||
// DNS 自愈:部分集群 CoreDNS 对 Service 记录异常(如 watch 失效),
|
||||
// 经 grafana 容器探测 Loki 服务名,不通则回退 ClusterIP 并热更新数据源。
|
||||
dnsProbe := exec.CommandContext(ctx, "kubectl", "exec", "-n", ns, "deploy/grafana", "--",
|
||||
"wget", "-q", "-O", "/dev/null", "-T", "5", lokiURL+"/ready")
|
||||
dnsProbe.Env = env
|
||||
if probeOut, probeErr := dnsProbe.CombinedOutput(); probeErr != nil {
|
||||
d.log("warn", fmt.Sprintf("Grafana 无法解析 Loki 服务名(%s),回退 ClusterIP…", strings.TrimSpace(string(probeOut))), "")
|
||||
ipCmd := exec.CommandContext(ctx, "kubectl", "get", "svc", "loki", "-n", ns,
|
||||
"-o", "jsonpath={.spec.clusterIP}")
|
||||
ipCmd.Env = env
|
||||
ipOut, ipErr := ipCmd.CombinedOutput()
|
||||
clusterIP := strings.TrimSpace(string(ipOut))
|
||||
if ipErr != nil || clusterIP == "" {
|
||||
d.log("error", "获取 Loki ClusterIP 失败,数据源保持服务名形式", "")
|
||||
} else {
|
||||
fallbackURL := fmt.Sprintf("http://%s:3100", clusterIP)
|
||||
fbFile, fbCleanup, ferr := writeTempFile("grafana-fb-*.yaml", d.grafanaManifest(ns, fallbackURL), 0600)
|
||||
if ferr == nil {
|
||||
fbApply := exec.CommandContext(ctx, "kubectl", "apply", "-f", fbFile)
|
||||
fbApply.Env = env
|
||||
fbApply.CombinedOutput()
|
||||
fbCleanup()
|
||||
restart := exec.CommandContext(ctx, "kubectl", "rollout", "restart", "deployment/grafana", "-n", ns)
|
||||
restart.Env = env
|
||||
restart.CombinedOutput()
|
||||
wait2 := exec.CommandContext(ctx, "kubectl", "rollout", "status", "deployment/grafana",
|
||||
"-n", ns, "--timeout=5m")
|
||||
wait2.Env = env
|
||||
w2, _ := wait2.CombinedOutput()
|
||||
d.log("info", string(w2), "")
|
||||
d.log("info", fmt.Sprintf("数据源已回退为 ClusterIP: %s", fallbackURL), "")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 回读实际分配的 NodePort(固定端口易与集群已有服务冲突)
|
||||
portCmd := exec.CommandContext(ctx, "kubectl", "get", "svc", "grafana", "-n", ns,
|
||||
"-o", "jsonpath={.spec.ports[0].nodePort}")
|
||||
portCmd.Env = env
|
||||
portOut, _ := portCmd.CombinedOutput()
|
||||
nodePort := strings.TrimSpace(string(portOut))
|
||||
access := fmt.Sprintf("http://<节点IP>:%s", nodePort)
|
||||
if nodePort == "" {
|
||||
access = "kubectl get svc grafana -n " + ns + " 查看访问端口"
|
||||
}
|
||||
|
||||
d.log("success", fmt.Sprintf("Grafana deployed. 访问: %s (NodePort, 账号 admin/admin), 数据源已指向 %s", access, d.lokiPushURL()), "")
|
||||
|
||||
// 自动导入 Loki 日志面板(打开即可看图;密码被用户修改过则跳过并提示)
|
||||
if err := d.importDashboard(ctx, env, ns); err != nil {
|
||||
d.log("warn", fmt.Sprintf("面板自动导入跳过: %v", err), "")
|
||||
} else {
|
||||
d.log("info", "已导入「Loki 日志」面板(Explore 可用,默认账号 admin/admin)", "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// importDashboard 通过 Grafana API 导入预置的 Loki 日志面板
|
||||
func (d *GrafanaDeployer) importDashboard(ctx context.Context, env []string, ns string) error {
|
||||
payload := grafanaDashboardPayload()
|
||||
|
||||
postDashboard := func(baseURL string) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
|
||||
strings.TrimSuffix(baseURL, "/")+"/api/dashboards/db", strings.NewReader(payload))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte("admin:admin")))
|
||||
resp, err := (&http.Client{Timeout: 15 * time.Second}).Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode >= 400 {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 300))
|
||||
return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(body))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
switch d.deployment.Mode {
|
||||
case "k8s-helm", "k8s-yaml":
|
||||
// 经 NodePort 直连(kube service proxy 的 POST 需额外 RBAC,直连最稳)
|
||||
ipCmd := exec.CommandContext(ctx, "kubectl", "get", "nodes",
|
||||
"-o", "jsonpath={.items[0].status.addresses[0].address}")
|
||||
ipCmd.Env = env
|
||||
ipOut, _ := ipCmd.CombinedOutput()
|
||||
portCmd := exec.CommandContext(ctx, "kubectl", "get", "svc", "grafana", "-n", ns,
|
||||
"-o", "jsonpath={.spec.ports[0].nodePort}")
|
||||
portCmd.Env = env
|
||||
portOut, _ := portCmd.CombinedOutput()
|
||||
ip, port := strings.TrimSpace(string(ipOut)), strings.TrimSpace(string(portOut))
|
||||
if ip == "" || port == "" {
|
||||
return fmt.Errorf("获取 NodePort 失败")
|
||||
}
|
||||
return postDashboard(fmt.Sprintf("http://%s:%s", ip, port))
|
||||
case "host-binary":
|
||||
if d.deployment.Host == nil {
|
||||
return fmt.Errorf("no host")
|
||||
}
|
||||
return postDashboard(fmt.Sprintf("http://%s:3000", d.deployment.Host.HostIP))
|
||||
}
|
||||
return fmt.Errorf("unsupported mode")
|
||||
}
|
||||
|
||||
// ==================== 主机 tarball 部署 ====================
|
||||
|
||||
func (d *GrafanaDeployer) deployHost(ctx context.Context) error {
|
||||
host := d.deployment.Host
|
||||
if host == nil {
|
||||
return fmt.Errorf("please select target host")
|
||||
}
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer client.Close()
|
||||
|
||||
arch, _, _ := client.Execute(ctx, "uname -m")
|
||||
grafanaArch := "amd64"
|
||||
if strings.Contains(arch, "aarch64") || strings.Contains(arch, "arm64") {
|
||||
grafanaArch = "arm64"
|
||||
}
|
||||
version := d.version()
|
||||
|
||||
d.log("info", fmt.Sprintf("Downloading Grafana %s (%s)...", version, grafanaArch), "")
|
||||
url := fmt.Sprintf("https://dl.grafana.com/oss/release/grafana-%s.linux-%s.tar.gz", version, grafanaArch)
|
||||
dl := fmt.Sprintf("curl -fsSL -o /tmp/grafana.tar.gz %s && tar -xzf /tmp/grafana.tar.gz -C /opt/ && rm -rf /opt/grafana && ln -sfn /opt/grafana-v%s /opt/grafana && rm -f /tmp/grafana.tar.gz", url, version)
|
||||
if out, _, err := client.Execute(ctx, dl); err != nil {
|
||||
return fmt.Errorf("download grafana failed: %w\n%s", err, out)
|
||||
}
|
||||
|
||||
// 数据源 provisioning(指向同机 Loki)
|
||||
dsYAML := fmt.Sprintf(`apiVersion: 1
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: %s
|
||||
isDefault: true
|
||||
`, d.lokiPushURL())
|
||||
client.Execute(ctx, "mkdir -p /etc/grafana/provisioning/datasources /var/lib/grafana /var/log/grafana")
|
||||
writeCmd := fmt.Sprintf("cat > /etc/grafana/provisioning/datasources/loki.yaml <<'EOF'\n%s\nEOF", dsYAML)
|
||||
if out, _, err := client.Execute(ctx, writeCmd); err != nil {
|
||||
return fmt.Errorf("write datasource config failed: %w\n%s", err, out)
|
||||
}
|
||||
|
||||
unit := `[Unit]
|
||||
Description=Grafana
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/opt/grafana
|
||||
ExecStart=/opt/grafana/bin/grafana server --homepath /opt/grafana --config /opt/grafana/conf/defaults.ini
|
||||
Restart=always
|
||||
LimitNOFILE=65536
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target`
|
||||
writeUnit := fmt.Sprintf("cat > /etc/systemd/system/grafana.service <<'EOF'\n%s\nEOF", unit)
|
||||
if out, _, err := client.Execute(ctx, writeUnit); err != nil {
|
||||
return fmt.Errorf("write systemd unit failed: %w\n%s", err, out)
|
||||
}
|
||||
if out, _, err := client.Execute(ctx, "systemctl daemon-reload && systemctl enable grafana && systemctl restart grafana"); err != nil {
|
||||
return fmt.Errorf("start grafana failed: %w\n%s", err, out)
|
||||
}
|
||||
|
||||
// 健康检查(最多 10 次)
|
||||
for i := 0; i < 10; i++ {
|
||||
out, _, _ := client.Execute(ctx, "curl -fsSL http://localhost:3000/api/health")
|
||||
if strings.Contains(out, "\"version\"") {
|
||||
d.log("success", fmt.Sprintf("Grafana running on %s:3000 (admin/admin), 数据源已指向本机 Loki", host.HostIP), "")
|
||||
return nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
d.log("warn", "Grafana 服务已启动但健康检查未通过,请稍后确认 http://<主机IP>:3000", "")
|
||||
return nil
|
||||
}
|
||||
|
||||
// CleanupGrafana 卸载 Grafana(停止部署时同步调用)
|
||||
func CleanupGrafana(ctx context.Context, deployment *models.LokiDeployment) {
|
||||
if deployment == nil {
|
||||
return
|
||||
}
|
||||
switch deployment.Mode {
|
||||
case "k8s-helm", "k8s-yaml":
|
||||
if deployment.Cluster == nil {
|
||||
return
|
||||
}
|
||||
env, cleanup, err := kubeconfigEnv(deployment.Cluster)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer cleanup()
|
||||
ns := deployment.Namespace
|
||||
if ns == "" {
|
||||
ns = "log"
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "kubectl", "delete",
|
||||
"deployment/grafana", "service/grafana", "configmap/grafana-datasources",
|
||||
"-n", ns, "--ignore-not-found=true")
|
||||
cmd.Env = env
|
||||
cmd.CombinedOutput()
|
||||
case "host-binary":
|
||||
if deployment.Host == nil {
|
||||
return
|
||||
}
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(deployment.Host))
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer client.Close()
|
||||
client.Execute(ctx, "systemctl stop grafana 2>/dev/null; systemctl disable grafana 2>/dev/null; rm -f /etc/systemd/system/grafana.service; systemctl daemon-reload")
|
||||
}
|
||||
}
|
||||
|
||||
// grafanaDashboardPayload 预置「Loki 日志」面板:日志量(按命名空间)/错误日志/最近日志
|
||||
func grafanaDashboardPayload() string {
|
||||
return `{"dashboard":{
|
||||
"uid":"loki-logs","title":"Loki 日志","tags":["lokideploy"],
|
||||
"timezone":"browser","schemaVersion":39,"refresh":"30s","version":1,
|
||||
"time":{"from":"now-1h","to":"now"},
|
||||
"panels":[
|
||||
{"id":1,"type":"timeseries","title":"日志量(按命名空间)",
|
||||
"datasource":{"type":"loki","uid":"loki-ds"},
|
||||
"gridPos":{"h":8,"w":12,"x":0,"y":0},
|
||||
"fieldConfig":{"defaults":{"unit":"short"},"overrides":[]},
|
||||
"targets":[{"refId":"A","expr":"sum by (namespace) (count_over_time({namespace=\".+\"}[5m]))","queryType":"range"}]},
|
||||
{"id":2,"type":"timeseries","title":"错误日志",
|
||||
"datasource":{"type":"loki","uid":"loki-ds"},
|
||||
"gridPos":{"h":8,"w":12,"x":12,"y":0},
|
||||
"fieldConfig":{"defaults":{"unit":"short","color":{"mode":"palette-classic"}},"overrides":[]},
|
||||
"targets":[{"refId":"A","expr":"sum(count_over_time({namespace=\".+\"} |~ \"(?i)error\" [5m]))","queryType":"range"}]},
|
||||
{"id":3,"type":"logs","title":"最近日志",
|
||||
"datasource":{"type":"loki","uid":"loki-ds"},
|
||||
"gridPos":{"h":10,"w":24,"x":0,"y":8},
|
||||
"targets":[{"refId":"A","expr":"{namespace=\".+\"}","queryType":"range"}]}
|
||||
]
|
||||
},"overwrite":true}`
|
||||
}
|
||||
+10
-18
@@ -3,13 +3,12 @@ package services
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"lokideploy/models"
|
||||
"lokideploy/pkg/crypto"
|
||||
"lokideploy/repository"
|
||||
"lokideploy/utils"
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
)
|
||||
|
||||
type HostService struct{}
|
||||
@@ -112,6 +111,9 @@ func (s *HostService) Update(id uint, updates map[string]interface{}) error {
|
||||
if v, ok := updates["status"].(string); ok {
|
||||
db.Hosts[i].Status = v
|
||||
}
|
||||
if v, ok := updates["statusMessage"].(string); ok {
|
||||
db.Hosts[i].StatusMessage = v
|
||||
}
|
||||
if v, ok := updates["os"].(string); ok {
|
||||
db.Hosts[i].OS = v
|
||||
}
|
||||
@@ -148,14 +150,9 @@ func (s *HostService) Validate(id uint) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
password := host.Password
|
||||
if strings.HasPrefix(password, "ENC:") {
|
||||
password, _ = crypto.Decrypt(password[4:])
|
||||
}
|
||||
|
||||
client, err := NewSSHClient(host.HostIP, host.Port, host.User, password)
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if err != nil {
|
||||
s.Update(id, map[string]interface{}{"status": "error"})
|
||||
s.Update(id, map[string]interface{}{"status": "error", "statusMessage": err.Error()})
|
||||
return
|
||||
}
|
||||
defer client.Close()
|
||||
@@ -175,12 +172,7 @@ func (s *HostService) ExecuteCommand(id uint, command string) (string, string, e
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
password := host.Password
|
||||
if strings.HasPrefix(password, "ENC:") {
|
||||
password, _ = crypto.Decrypt(password[4:])
|
||||
}
|
||||
|
||||
client, err := NewSSHClient(host.HostIP, host.Port, host.User, password)
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
|
||||
@@ -6,7 +6,7 @@ import (
|
||||
"os"
|
||||
"regexp"
|
||||
|
||||
"lokideploy/models"
|
||||
"opsxloki/models"
|
||||
)
|
||||
|
||||
// resolveKubeconfig 取集群最终的 kubeconfig 文本(优先内存字段,其次磁盘文件)
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
// Package services — LDAP 接入:配置存储(SystemSetting 表 JSON)、连接测试、
|
||||
// 用户搜索预览、批量导入与登录验证共用同一套连接/搜索代码。
|
||||
// 设计参照 opsx-web-lite(has_password 脱敏/空密码保留旧值/批量导入冲突策略)。
|
||||
package services
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
|
||||
ldap "github.com/go-ldap/ldap/v3"
|
||||
)
|
||||
|
||||
const ldapSettingKey = "ldap_settings"
|
||||
|
||||
// netDialer LDAP 拨号器(5s 超时,参照 aiops 的超时处理,避免连接挂起)
|
||||
func netDialer() *net.Dialer {
|
||||
return &net.Dialer{Timeout: 5 * time.Second}
|
||||
}
|
||||
|
||||
// LDAPConfig LDAP 配置(存储形态;BindPassword 以 ENC: 前缀 AES-GCM 加密落库)
|
||||
type LDAPConfig struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Host string `json:"host"`
|
||||
Port int `json:"port"`
|
||||
UseSSL bool `json:"useSSL"`
|
||||
BindDN string `json:"bindDN"`
|
||||
BindPassword string `json:"bindPassword"` // 存储为 ENC:密文;对外输出时剔除
|
||||
BaseDN string `json:"baseDN"`
|
||||
UserFilter string `json:"userFilter"`
|
||||
UsernameAttr string `json:"usernameAttr"`
|
||||
NameAttr string `json:"nameAttr"`
|
||||
EmailAttr string `json:"emailAttr"`
|
||||
}
|
||||
|
||||
// LDAPPublicConfig 对外输出形态(密码剔除,附 hasPassword)
|
||||
type LDAPPublicConfig struct {
|
||||
LDAPConfig
|
||||
BindPassword string `json:"bindPassword"` // 恒为空
|
||||
HasPassword bool `json:"hasPassword"`
|
||||
Configured bool `json:"configured"` // host+bindDN+baseDN 齐备(控制用户页导入按钮显隐)
|
||||
}
|
||||
|
||||
func defaultLDAPConfig() *LDAPConfig {
|
||||
return &LDAPConfig{
|
||||
Port: 389,
|
||||
UserFilter: "(objectClass=inetOrgPerson)",
|
||||
UsernameAttr: "uid",
|
||||
NameAttr: "displayName",
|
||||
EmailAttr: "mail",
|
||||
}
|
||||
}
|
||||
|
||||
// GetLDAPConfig 读取配置(解密密码);未配置时返回默认值
|
||||
func GetLDAPConfig() *LDAPConfig {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
var raw string
|
||||
for i := range db.Settings {
|
||||
if db.Settings[i].Key == ldapSettingKey {
|
||||
raw = db.Settings[i].Value
|
||||
break
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
|
||||
cfg := defaultLDAPConfig()
|
||||
if raw != "" {
|
||||
_ = json.Unmarshal([]byte(raw), cfg)
|
||||
}
|
||||
if cfg.Port == 0 {
|
||||
cfg.Port = 389
|
||||
}
|
||||
if cfg.BindPassword != "" {
|
||||
if plain, err := DecryptCred(cfg.BindPassword); err == nil {
|
||||
cfg.BindPassword = plain
|
||||
}
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func GetLDAPPublicConfig() *LDAPPublicConfig {
|
||||
cfg := GetLDAPConfig()
|
||||
hasPwd := cfg.BindPassword != ""
|
||||
return &LDAPPublicConfig{
|
||||
LDAPConfig: LDAPConfig{
|
||||
Enabled: cfg.Enabled,
|
||||
Host: cfg.Host,
|
||||
Port: cfg.Port,
|
||||
UseSSL: cfg.UseSSL,
|
||||
BindDN: cfg.BindDN,
|
||||
BaseDN: cfg.BaseDN,
|
||||
UserFilter: cfg.UserFilter,
|
||||
UsernameAttr: cfg.UsernameAttr,
|
||||
NameAttr: cfg.NameAttr,
|
||||
EmailAttr: cfg.EmailAttr,
|
||||
},
|
||||
HasPassword: hasPwd,
|
||||
Configured: cfg.Host != "" && cfg.BindDN != "" && cfg.BaseDN != "",
|
||||
}
|
||||
}
|
||||
|
||||
// SaveLDAPConfig 保存;bindPassword 为空且已有存量 → 保留旧密文
|
||||
func SaveLDAPConfig(in *LDAPConfig) error {
|
||||
old := GetLDAPConfig()
|
||||
if in.Port == 0 {
|
||||
in.Port = 389
|
||||
}
|
||||
if in.UserFilter == "" {
|
||||
in.UserFilter = old.UserFilter
|
||||
}
|
||||
if in.UsernameAttr == "" {
|
||||
in.UsernameAttr = "uid"
|
||||
}
|
||||
if in.NameAttr == "" {
|
||||
in.NameAttr = "displayName"
|
||||
}
|
||||
if in.EmailAttr == "" {
|
||||
in.EmailAttr = "mail"
|
||||
}
|
||||
if in.BindPassword == "" {
|
||||
in.BindPassword = old.BindPassword // 已是明文(GetLDAPConfig 解密过)
|
||||
} else if enc, err := crypto.EncryptField(in.BindPassword); err == nil {
|
||||
in.BindPassword = enc
|
||||
}
|
||||
|
||||
data, _ := json.Marshal(in)
|
||||
db := repository.GetDB()
|
||||
// NextID/CreateUser 均自拿锁,禁止在持有 db.Lock() 时调用(JsonDB 锁不可重入)
|
||||
newID := db.NextID("settings")
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
for i := range db.Settings {
|
||||
if db.Settings[i].Key == ldapSettingKey {
|
||||
db.Settings[i].Value = string(data)
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
}
|
||||
db.Settings = append(db.Settings, models.SystemSetting{
|
||||
ID: newID, Key: ldapSettingKey, Value: string(data),
|
||||
})
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
|
||||
// ==================== 连接与搜索 ====================
|
||||
|
||||
// dialLDAP 建立连接(5s 拨号超时,防参照项目无超时挂起的问题)
|
||||
func dialLDAP(cfg *LDAPConfig) (*ldap.Conn, error) {
|
||||
scheme := "ldap"
|
||||
if cfg.UseSSL {
|
||||
scheme = "ldaps"
|
||||
}
|
||||
url := fmt.Sprintf("%s://%s:%d", scheme, cfg.Host, cfg.Port)
|
||||
conn, err := ldap.DialURL(url, ldap.DialWithDialer(netDialer()))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("无法连接 LDAP 服务器 %s: %w", url, err)
|
||||
}
|
||||
conn.SetTimeout(10 * time.Second)
|
||||
if err := conn.Bind(cfg.BindDN, cfg.BindPassword); err != nil {
|
||||
conn.Close()
|
||||
return nil, fmt.Errorf("管理员绑定失败(检查 Bind DN / 密码): %w", err)
|
||||
}
|
||||
return conn, nil
|
||||
}
|
||||
|
||||
// ldapUserEntry LDAP 用户条目(导入预览与登录共用)
|
||||
type ldapUserEntry struct {
|
||||
DN string
|
||||
Username string
|
||||
DisplayName string
|
||||
Email string
|
||||
}
|
||||
|
||||
// searchLDAPUsers 按 ouDN(空用 baseDN)WholeSubtree 搜索用户,sizeLimit 上限
|
||||
func searchLDAPUsers(cfg *LDAPConfig, ouDN string, sizeLimit int) ([]ldapUserEntry, error) {
|
||||
if ouDN == "" {
|
||||
ouDN = cfg.BaseDN
|
||||
}
|
||||
conn, err := dialLDAP(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
filter := cfg.UserFilter
|
||||
if !strings.HasPrefix(filter, "(") {
|
||||
filter = "(" + filter + ")"
|
||||
}
|
||||
req := ldap.NewSearchRequest(ouDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
|
||||
sizeLimit, 0, false, filter,
|
||||
[]string{"dn", cfg.UsernameAttr, cfg.NameAttr, "cn", cfg.EmailAttr}, nil)
|
||||
res, err := conn.Search(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("搜索失败(BaseDN/过滤器可能有误): %w", err)
|
||||
}
|
||||
|
||||
users := make([]ldapUserEntry, 0, len(res.Entries))
|
||||
seen := map[string]bool{}
|
||||
for _, e := range res.Entries {
|
||||
u := ldapUserEntry{
|
||||
DN: e.DN,
|
||||
Username: e.GetAttributeValue(cfg.UsernameAttr),
|
||||
DisplayName: e.GetAttributeValue(cfg.NameAttr),
|
||||
Email: e.GetAttributeValue(cfg.EmailAttr),
|
||||
}
|
||||
if u.DisplayName == "" {
|
||||
u.DisplayName = e.GetAttributeValue("cn")
|
||||
}
|
||||
if u.Username == "" || seen[u.DN] {
|
||||
continue // 无用户名属性的条目(如 ou 本身)跳过
|
||||
}
|
||||
seen[u.DN] = true
|
||||
users = append(users, u)
|
||||
}
|
||||
return users, nil
|
||||
}
|
||||
|
||||
// searchLDAPUserByUsername 登录用:按用户名搜索单个条目
|
||||
func searchLDAPUserByUsername(cfg *LDAPConfig, username string) (*ldapUserEntry, error) {
|
||||
conn, err := dialLDAP(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
filter := fmt.Sprintf("(&%s(%s=%s))", normalizeFilter(cfg.UserFilter),
|
||||
cfg.UsernameAttr, ldap.EscapeFilter(username))
|
||||
req := ldap.NewSearchRequest(cfg.BaseDN, ldap.ScopeWholeSubtree, ldap.NeverDerefAliases,
|
||||
2, 0, false, filter,
|
||||
[]string{"dn", cfg.UsernameAttr, cfg.NameAttr, "cn", cfg.EmailAttr}, nil)
|
||||
res, err := conn.Search(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(res.Entries) == 0 {
|
||||
return nil, utils.ErrInvalidCredentials
|
||||
}
|
||||
if len(res.Entries) > 1 {
|
||||
return nil, fmt.Errorf("LDAP 中存在多个匹配 %s 的用户", username)
|
||||
}
|
||||
e := res.Entries[0]
|
||||
return &ldapUserEntry{
|
||||
DN: e.DN,
|
||||
Username: e.GetAttributeValue(cfg.UsernameAttr),
|
||||
DisplayName: e.GetAttributeValue(cfg.NameAttr),
|
||||
Email: e.GetAttributeValue(cfg.EmailAttr),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// verifyLDAPPassword 用户级验证:bind 到用户 DN 校验密码
|
||||
func verifyLDAPPassword(cfg *LDAPConfig, username, password string) (*ldapUserEntry, error) {
|
||||
entry, err := searchLDAPUserByUsername(cfg, username)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
scheme := "ldap"
|
||||
if cfg.UseSSL {
|
||||
scheme = "ldaps"
|
||||
}
|
||||
conn, err := ldap.DialURL(fmt.Sprintf("%s://%s:%d", scheme, cfg.Host, cfg.Port),
|
||||
ldap.DialWithDialer(netDialer()))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer conn.Close()
|
||||
if err := conn.Bind(entry.DN, password); err != nil {
|
||||
return nil, utils.ErrInvalidCredentials
|
||||
}
|
||||
return entry, nil
|
||||
}
|
||||
|
||||
func normalizeFilter(f string) string {
|
||||
if !strings.HasPrefix(f, "(") {
|
||||
return "(" + f + ")"
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// TestLDAPConnection 测试连接:管理 bind + baseDN 校验 + 可选用户级验证
|
||||
func TestLDAPConnection(cfg *LDAPConfig, testUsername, testPassword string) map[string]interface{} {
|
||||
if cfg.Host == "" || cfg.BindDN == "" || cfg.BaseDN == "" {
|
||||
return map[string]interface{}{"success": false, "message": "请先填写服务器地址 / Bind DN / Base DN"}
|
||||
}
|
||||
// 表单密码为空回退已存密码
|
||||
if cfg.BindPassword == "" {
|
||||
cfg.BindPassword = GetLDAPConfig().BindPassword
|
||||
}
|
||||
|
||||
conn, err := dialLDAP(cfg)
|
||||
if err != nil {
|
||||
return map[string]interface{}{"success": false, "message": err.Error()}
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
// baseDN 可读性校验
|
||||
req := ldap.NewSearchRequest(cfg.BaseDN, ldap.ScopeBaseObject, ldap.NeverDerefAliases,
|
||||
1, 0, false, "(objectClass=*)", []string{"dn"}, nil)
|
||||
if _, err := conn.Search(req); err != nil {
|
||||
return map[string]interface{}{"success": false, "message": fmt.Sprintf("服务器连通,但 Base DN 搜索失败: %v", err)}
|
||||
}
|
||||
|
||||
result := map[string]interface{}{
|
||||
"success": true,
|
||||
"message": fmt.Sprintf("连接成功:服务器可达,管理员绑定与 Base DN 校验通过 (%s)", cfg.BaseDN),
|
||||
}
|
||||
if testUsername != "" {
|
||||
if testPassword == "" {
|
||||
return map[string]interface{}{"success": false, "message": "请填写测试账号的密码"}
|
||||
}
|
||||
entry, err := verifyLDAPPassword(cfg, testUsername, testPassword)
|
||||
if err != nil {
|
||||
return map[string]interface{}{"success": false, "message": fmt.Sprintf("服务器连接正常,但用户「%s」验证失败: %v", testUsername, err)}
|
||||
}
|
||||
result["message"] = fmt.Sprintf("连接成功,用户「%s」验证通过 (DN: %s)", testUsername, entry.DN)
|
||||
result["userInfo"] = map[string]string{
|
||||
"dn": entry.DN, "username": entry.Username,
|
||||
"displayName": entry.DisplayName, "email": entry.Email,
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
"opsxloki/repository"
|
||||
)
|
||||
|
||||
// LDAPImportUser 批量导入请求中的用户条目(数据来自预览接口)
|
||||
type LDAPImportUser struct {
|
||||
DN string `json:"dn"`
|
||||
Username string `json:"username" binding:"required"`
|
||||
Nickname string `json:"nickname"`
|
||||
Email string `json:"email"`
|
||||
}
|
||||
|
||||
// LDAPImportResult 导入结果(对齐 opsx-web-lite 契约)
|
||||
type LDAPImportResult struct {
|
||||
SuccessCount int `json:"successCount"`
|
||||
SkippedCount int `json:"skippedCount"`
|
||||
UpdatedCount int `json:"updatedCount"`
|
||||
FailedCount int `json:"failedCount"`
|
||||
FailedUsers []map[string]interface{} `json:"failedUsers"`
|
||||
}
|
||||
|
||||
// BatchImportLDAPUsers 批量导入 LDAP 用户:
|
||||
// 已存在且 LDAP 用户 → 跳过;本地同名 → 更新昵称并转 LDAP;否则新建(user 角色,密码占位)。
|
||||
func BatchImportLDAPUsers(users []LDAPImportUser) *LDAPImportResult {
|
||||
result := &LDAPImportResult{FailedUsers: []map[string]interface{}{}}
|
||||
db := repository.GetDB()
|
||||
|
||||
for _, in := range users {
|
||||
if in.Username == "" {
|
||||
continue
|
||||
}
|
||||
// 查找本地同名用户(读锁短持有;NextID/CreateUser 不可在持锁时调用)
|
||||
db.RLock()
|
||||
idx := -1
|
||||
for i := range db.Users {
|
||||
if db.Users[i].Username == in.Username {
|
||||
idx = i
|
||||
break
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
|
||||
if idx >= 0 {
|
||||
db.Lock()
|
||||
if db.Users[idx].AuthSource == "ldap" {
|
||||
db.Unlock()
|
||||
result.SkippedCount++
|
||||
continue
|
||||
}
|
||||
db.Users[idx].AuthSource = "ldap" // 本地同名 → 转 LDAP 并同步昵称
|
||||
if in.Nickname != "" {
|
||||
db.Users[idx].Nickname = in.Nickname
|
||||
}
|
||||
db.Users[idx].UpdatedAt = time.Now()
|
||||
db.RequestSave()
|
||||
db.Unlock()
|
||||
result.UpdatedCount++
|
||||
continue
|
||||
}
|
||||
|
||||
hash, err := crypto.HashPassword("ldap-user-" + in.Username + "-no-local-login")
|
||||
if err != nil {
|
||||
result.FailedCount++
|
||||
result.FailedUsers = append(result.FailedUsers, map[string]interface{}{
|
||||
"username": in.Username, "error": err.Error(),
|
||||
})
|
||||
continue
|
||||
}
|
||||
u := &models.User{
|
||||
Username: in.Username,
|
||||
Password: hash,
|
||||
Nickname: in.Nickname,
|
||||
Role: "user",
|
||||
AuthSource: "ldap",
|
||||
}
|
||||
if err := db.CreateUser(u); err != nil {
|
||||
result.FailedCount++
|
||||
result.FailedUsers = append(result.FailedUsers, map[string]interface{}{
|
||||
"username": in.Username, "error": err.Error(),
|
||||
})
|
||||
continue
|
||||
}
|
||||
result.SuccessCount++
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// SearchLDAPUsersForImport 预览入口(服务层薄封装,便于 controller 调用)
|
||||
func SearchLDAPUsersForImport(cfg *LDAPConfig, ouDN string) ([]map[string]interface{}, error) {
|
||||
users, err := searchLDAPUsers(cfg, ouDN, 500)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]map[string]interface{}, 0, len(users))
|
||||
for _, u := range users {
|
||||
out = append(out, map[string]interface{}{
|
||||
"dn": u.DN, "username": u.Username,
|
||||
"displayName": u.DisplayName, "email": u.Email,
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"opsxloki/models"
|
||||
"opsxloki/repository"
|
||||
)
|
||||
|
||||
func ListLogPolicyGroups() []models.LogPolicyGroup {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
groups := make([]models.LogPolicyGroup, len(db.LogPolicyGroups))
|
||||
for i, group := range db.LogPolicyGroups {
|
||||
groups[i] = group
|
||||
groups[i].UserIDs = append([]uint(nil), group.UserIDs...)
|
||||
groups[i].Labels = make(map[string]string, len(group.Labels))
|
||||
for key, value := range group.Labels { groups[i].Labels[key] = value }
|
||||
}
|
||||
return groups
|
||||
}
|
||||
|
||||
func validatePolicyGroup(db *repository.JsonDB, group models.LogPolicyGroup, excludeID uint) error {
|
||||
group.Name = strings.TrimSpace(group.Name)
|
||||
if group.Name == "" {
|
||||
return fmt.Errorf("策略组名称必填")
|
||||
}
|
||||
if group.SourceID == 0 {
|
||||
return fmt.Errorf("请选择数据源")
|
||||
}
|
||||
foundSource := false
|
||||
for _, source := range db.LogSources {
|
||||
if source.ID == group.SourceID {
|
||||
foundSource = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !foundSource {
|
||||
return fmt.Errorf("数据源不存在")
|
||||
}
|
||||
if len(group.UserIDs) == 0 {
|
||||
return fmt.Errorf("请选择至少一个用户")
|
||||
}
|
||||
for key := range group.Labels {
|
||||
if !ValidLogLabelName(key) {
|
||||
return fmt.Errorf("无效标签名: %s", key)
|
||||
}
|
||||
}
|
||||
seen := map[uint]bool{}
|
||||
for _, id := range group.UserIDs {
|
||||
if id == 0 || seen[id] {
|
||||
return fmt.Errorf("用户 ID 无效或重复")
|
||||
}
|
||||
seen[id] = true
|
||||
found := false
|
||||
for _, user := range db.Users {
|
||||
if user.ID == id && user.Role != "admin" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return fmt.Errorf("用户 %d 不存在或不是普通用户", id)
|
||||
}
|
||||
}
|
||||
for _, other := range db.LogPolicyGroups {
|
||||
if other.ID == excludeID || other.SourceID != group.SourceID {
|
||||
continue
|
||||
}
|
||||
if other.Name == group.Name {
|
||||
return fmt.Errorf("此数据源已有同名策略组")
|
||||
}
|
||||
for _, id := range other.UserIDs {
|
||||
if seen[id] {
|
||||
return fmt.Errorf("用户 %d 已属于此数据源的另一个策略组", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Group membership replaces legacy per-user grants for the same source.
|
||||
func removeDirectGrants(db *repository.JsonDB, sourceID uint, userIDs []uint) {
|
||||
members := map[uint]bool{}
|
||||
for _, id := range userIDs {
|
||||
members[id] = true
|
||||
}
|
||||
for i := range db.LogSources {
|
||||
if db.LogSources[i].ID != sourceID {
|
||||
continue
|
||||
}
|
||||
kept := db.LogSources[i].Access[:0]
|
||||
for _, rule := range db.LogSources[i].Access {
|
||||
if !members[rule.UserID] {
|
||||
kept = append(kept, rule)
|
||||
}
|
||||
}
|
||||
db.LogSources[i].Access = kept
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func CreateLogPolicyGroup(group models.LogPolicyGroup) (*models.LogPolicyGroup, error) {
|
||||
db := repository.GetDB()
|
||||
id := db.NextID("logPolicyGroups")
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
if err := validatePolicyGroup(db, group, 0); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
group.ID = id
|
||||
group.Name = strings.TrimSpace(group.Name)
|
||||
group.CreatedAt = time.Now()
|
||||
group.UpdatedAt = group.CreatedAt
|
||||
removeDirectGrants(db, group.SourceID, group.UserIDs)
|
||||
db.LogPolicyGroups = append(db.LogPolicyGroups, group)
|
||||
db.RequestSave()
|
||||
return &group, nil
|
||||
}
|
||||
|
||||
func UpdateLogPolicyGroup(id uint, group models.LogPolicyGroup) (*models.LogPolicyGroup, error) {
|
||||
db := repository.GetDB()
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
for i := range db.LogPolicyGroups {
|
||||
if db.LogPolicyGroups[i].ID != id {
|
||||
continue
|
||||
}
|
||||
if err := validatePolicyGroup(db, group, id); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
group.ID = id
|
||||
group.Name = strings.TrimSpace(group.Name)
|
||||
group.CreatedAt = db.LogPolicyGroups[i].CreatedAt
|
||||
group.UpdatedAt = time.Now()
|
||||
removeDirectGrants(db, group.SourceID, group.UserIDs)
|
||||
db.LogPolicyGroups[i] = group
|
||||
db.RequestSave()
|
||||
return &group, nil
|
||||
}
|
||||
return nil, fmt.Errorf("策略组不存在")
|
||||
}
|
||||
|
||||
func DeleteLogPolicyGroup(id uint) error {
|
||||
db := repository.GetDB()
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
for i := range db.LogPolicyGroups {
|
||||
if db.LogPolicyGroups[i].ID == id {
|
||||
db.LogPolicyGroups = append(db.LogPolicyGroups[:i], db.LogPolicyGroups[i+1:]...)
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("策略组不存在")
|
||||
}
|
||||
|
||||
func policyGroupAccess(groups []models.LogPolicyGroup, sourceID, userID uint) *models.LogSourceAccess {
|
||||
for _, group := range groups {
|
||||
if group.SourceID != sourceID {
|
||||
continue
|
||||
}
|
||||
for _, id := range group.UserIDs {
|
||||
if id == userID {
|
||||
return &models.LogSourceAccess{UserID: userID, Labels: group.Labels}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,780 @@
|
||||
package services
|
||||
|
||||
// 日志查询代理 — 后端代表前端访问 Loki HTTP API。
|
||||
// cluster-svc 型经 kube-apiserver service proxy(不依赖 ClusterIP 直通与集群 DNS);
|
||||
// url 型直连。契约参照 aiops 日志中心(query/labels/values/histogram + cursor 翻页)。
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
"opsxloki/repository"
|
||||
)
|
||||
|
||||
const maxLogQueryLimit = 1000
|
||||
|
||||
// ==================== 数据源 CRUD ====================
|
||||
|
||||
func GetLogSource(id uint) *models.LogSource {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
for i := range db.LogSources {
|
||||
if db.LogSources[i].ID == id {
|
||||
s := db.LogSources[i]
|
||||
return &s
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ListLogSources() []map[string]interface{} {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
out := make([]map[string]interface{}, 0, len(db.LogSources))
|
||||
for i := range db.LogSources {
|
||||
out = append(out, db.LogSources[i].Sanitized())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ListVisibleLogSources(userID uint, admin bool) []map[string]interface{} {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
defer db.RUnlock()
|
||||
out := make([]map[string]interface{}, 0)
|
||||
for i := range db.LogSources {
|
||||
s := &db.LogSources[i]
|
||||
if admin || policyGroupAccess(db.LogPolicyGroups, s.ID, userID) != nil || sourceAccess(s, userID) != nil {
|
||||
item := s.Sanitized()
|
||||
if !admin {
|
||||
item = map[string]interface{}{"id": s.ID, "name": s.Name, "isDefault": s.IsDefault}
|
||||
}
|
||||
out = append(out, item)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sourceAccess(src *models.LogSource, userID uint) *models.LogSourceAccess {
|
||||
for i := range src.Access {
|
||||
if src.Access[i].UserID == userID {
|
||||
return &src.Access[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var labelNamePattern = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]*$`)
|
||||
|
||||
func ValidLogLabelName(name string) bool { return labelNamePattern.MatchString(name) }
|
||||
|
||||
// AuthorizedLogQuery constrains the Loki stream selector server-side. Non-admins must be explicitly granted access.
|
||||
func AuthorizedLogQuery(src *models.LogSource, userID uint, admin bool, query string) (string, error) {
|
||||
if admin {
|
||||
return query, nil
|
||||
}
|
||||
db := repository.GetDB()
|
||||
var access *models.LogSourceAccess
|
||||
if db != nil {
|
||||
db.RLock()
|
||||
access = policyGroupAccess(db.LogPolicyGroups, src.ID, userID)
|
||||
db.RUnlock()
|
||||
}
|
||||
if access == nil {
|
||||
access = sourceAccess(src, userID)
|
||||
}
|
||||
if access == nil {
|
||||
return "", fmt.Errorf("无权查询此数据源")
|
||||
}
|
||||
q := strings.TrimSpace(query)
|
||||
if !strings.HasPrefix(q, "{") {
|
||||
return "", fmt.Errorf("普通用户查询必须以日志流选择器开头")
|
||||
}
|
||||
close := strings.IndexByte(q, '}')
|
||||
if close < 0 || strings.Contains(q[close+1:], "{") {
|
||||
return "", fmt.Errorf("查询只能包含一个日志流选择器")
|
||||
}
|
||||
if len(access.Labels) == 0 {
|
||||
return q, nil
|
||||
}
|
||||
keys := make([]string, 0, len(access.Labels))
|
||||
for k := range access.Labels {
|
||||
if !labelNamePattern.MatchString(k) {
|
||||
return "", fmt.Errorf("访问策略包含无效标签名")
|
||||
}
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
matchers := make([]string, 0, len(keys))
|
||||
for _, k := range keys {
|
||||
v, _ := json.Marshal(access.Labels[k])
|
||||
matchers = append(matchers, k+"="+string(v))
|
||||
}
|
||||
inner := strings.TrimSpace(q[1:close])
|
||||
if inner != "" {
|
||||
inner += ","
|
||||
}
|
||||
return "{" + inner + strings.Join(matchers, ",") + "}" + q[close+1:], nil
|
||||
}
|
||||
|
||||
func CreateLogSource(s *models.LogSource) (*models.LogSource, error) {
|
||||
if err := encryptSourceCredentials(s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
db := repository.GetDB()
|
||||
newID := db.NextID("logSources")
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
s.ID = newID
|
||||
if s.IsDefault {
|
||||
for i := range db.LogSources {
|
||||
db.LogSources[i].IsDefault = false
|
||||
}
|
||||
}
|
||||
// 首个数据源自动设默认
|
||||
if len(db.LogSources) == 0 {
|
||||
s.IsDefault = true
|
||||
}
|
||||
db.LogSources = append(db.LogSources, *s)
|
||||
db.RequestSave()
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func encryptSourceCredentials(s *models.LogSource) error {
|
||||
if s.BasicPass != "" {
|
||||
v, err := crypto.EncryptField(s.BasicPass)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.BasicPass = v
|
||||
}
|
||||
if s.BearerToken != "" {
|
||||
v, err := crypto.EncryptField(s.BearerToken)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.BearerToken = v
|
||||
}
|
||||
for name, value := range s.Headers {
|
||||
if !validSourceHeader(name) {
|
||||
return fmt.Errorf("自定义 Header 名称无效: %s", name)
|
||||
}
|
||||
v, err := crypto.EncryptField(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.Headers[name] = v
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validSourceHeader(name string) bool {
|
||||
if name == "" || strings.EqualFold(name, "Host") || strings.EqualFold(name, "Authorization") {
|
||||
return false
|
||||
}
|
||||
for _, c := range name {
|
||||
if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func UpdateLogSource(id uint, patch map[string]interface{}) error {
|
||||
db := repository.GetDB()
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
for i := range db.LogSources {
|
||||
if db.LogSources[i].ID != id {
|
||||
continue
|
||||
}
|
||||
if v, ok := patch["name"].(string); ok && v != "" {
|
||||
db.LogSources[i].Name = v
|
||||
}
|
||||
if v, ok := patch["namespace"].(string); ok {
|
||||
db.LogSources[i].Namespace = v
|
||||
}
|
||||
if v, ok := patch["service"].(string); ok {
|
||||
db.LogSources[i].Service = v
|
||||
}
|
||||
if v, ok := patch["port"].(float64); ok {
|
||||
db.LogSources[i].Port = int(v)
|
||||
}
|
||||
if v, ok := patch["url"].(string); ok {
|
||||
db.LogSources[i].URL = v
|
||||
}
|
||||
if v, ok := patch["authType"].(string); ok {
|
||||
db.LogSources[i].AuthType = v
|
||||
}
|
||||
if v, ok := patch["basicUser"].(string); ok {
|
||||
db.LogSources[i].BasicUser = v
|
||||
}
|
||||
if v, ok := patch["bearerToken"].(string); ok {
|
||||
if v != "" {
|
||||
enc, err := crypto.EncryptField(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
db.LogSources[i].BearerToken = enc
|
||||
}
|
||||
}
|
||||
if v, ok := patch["skipTLSVerify"].(bool); ok {
|
||||
db.LogSources[i].SkipTLSVerify = v
|
||||
}
|
||||
if raw, ok := patch["headers"]; ok {
|
||||
data, err := json.Marshal(raw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var headers map[string]string
|
||||
if err := json.Unmarshal(data, &headers); err != nil {
|
||||
return err
|
||||
}
|
||||
for name, value := range headers {
|
||||
if !validSourceHeader(name) {
|
||||
return fmt.Errorf("自定义 Header 名称无效: %s", name)
|
||||
}
|
||||
if value == "" {
|
||||
if old, ok := db.LogSources[i].Headers[name]; ok {
|
||||
headers[name] = old
|
||||
}
|
||||
continue
|
||||
}
|
||||
enc, err := crypto.EncryptField(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
headers[name] = enc
|
||||
}
|
||||
db.LogSources[i].Headers = headers
|
||||
}
|
||||
if v, ok := patch["basicPass"].(string); ok && v != "" {
|
||||
if enc, err := crypto.EncryptField(v); err == nil {
|
||||
db.LogSources[i].BasicPass = enc
|
||||
}
|
||||
}
|
||||
if v, ok := patch["isDefault"].(bool); ok && v {
|
||||
for j := range db.LogSources {
|
||||
db.LogSources[j].IsDefault = db.LogSources[j].ID == id
|
||||
}
|
||||
}
|
||||
if raw, ok := patch["access"]; ok {
|
||||
data, err := json.Marshal(raw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var access []models.LogSourceAccess
|
||||
if err := json.Unmarshal(data, &access); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := ValidateSourceAccess(access); err != nil {
|
||||
return err
|
||||
}
|
||||
db.LogSources[i].Access = access
|
||||
}
|
||||
db.LogSources[i].UpdatedAt = time.Now()
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("数据源不存在")
|
||||
}
|
||||
|
||||
func ValidateSourceAccess(access []models.LogSourceAccess) error {
|
||||
seen := map[uint]bool{}
|
||||
for _, rule := range access {
|
||||
if rule.UserID == 0 || seen[rule.UserID] {
|
||||
return fmt.Errorf("访问策略的用户 ID 无效或重复")
|
||||
}
|
||||
seen[rule.UserID] = true
|
||||
for k := range rule.Labels {
|
||||
if !labelNamePattern.MatchString(k) {
|
||||
return fmt.Errorf("访问策略包含无效标签名: %s", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func DeleteLogSource(id uint) error {
|
||||
db := repository.GetDB()
|
||||
db.Lock()
|
||||
defer db.Unlock()
|
||||
for i := range db.LogSources {
|
||||
if db.LogSources[i].ID == id {
|
||||
db.LogSources = append(db.LogSources[:i], db.LogSources[i+1:]...)
|
||||
groups := db.LogPolicyGroups[:0]
|
||||
for _, group := range db.LogPolicyGroups {
|
||||
if group.SourceID != id {
|
||||
groups = append(groups, group)
|
||||
}
|
||||
}
|
||||
db.LogPolicyGroups = groups
|
||||
db.RequestSave()
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("数据源不存在")
|
||||
}
|
||||
|
||||
// EnsureLogSourceForDeployment 部署成功后自动注册查询数据源(host-binary→url;k8s→cluster-svc)
|
||||
func EnsureLogSourceForDeployment(d *models.LokiDeployment) {
|
||||
if d == nil || d.Endpoint == "" {
|
||||
return
|
||||
}
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
for i := range db.LogSources {
|
||||
if db.LogSources[i].Name == d.Name {
|
||||
db.RUnlock()
|
||||
return
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
|
||||
svc := "loki"
|
||||
port := 3100
|
||||
src := &models.LogSource{
|
||||
Name: d.Name, TargetType: "cluster-svc",
|
||||
ClusterID: d.ClusterID, Namespace: d.Namespace, Service: svc, Port: port,
|
||||
AuthType: "none",
|
||||
}
|
||||
if d.Mode == "host-binary" {
|
||||
src.TargetType = "url"
|
||||
src.URL = d.Endpoint
|
||||
src.ClusterID = nil
|
||||
}
|
||||
_, _ = CreateLogSource(src)
|
||||
}
|
||||
|
||||
// ==================== Loki HTTP 访问 ====================
|
||||
|
||||
// lokiProxyGET 对数据源执行 GET;cluster-svc 走 kubectl --raw service proxy。
|
||||
func lokiProxyGET(ctx context.Context, src *models.LogSource, path string, query url.Values) ([]byte, int, error) {
|
||||
// cluster-svc:kubectl get --raw /api/v1/namespaces/{ns}/services/{svc}:{port}/proxy{path}?q
|
||||
if src.TargetType == "cluster-svc" {
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
var cluster *models.Cluster
|
||||
for i := range db.Clusters {
|
||||
if src.ClusterID != nil && db.Clusters[i].ID == *src.ClusterID {
|
||||
c := db.Clusters[i]
|
||||
cluster = &c
|
||||
break
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
if cluster == nil {
|
||||
return nil, 500, fmt.Errorf("数据源关联的集群不存在")
|
||||
}
|
||||
env, cleanup, err := kubeconfigEnv(cluster)
|
||||
if err != nil {
|
||||
return nil, 500, err
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
proxyPath := fmt.Sprintf("/api/v1/namespaces/%s/services/%s:%d/proxy%s",
|
||||
src.Namespace, src.Service, src.Port, path)
|
||||
full := proxyPath
|
||||
if qs := query.Encode(); qs != "" {
|
||||
full += "?" + qs
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "kubectl", "get", "--raw", full)
|
||||
cmd.Env = env
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return nil, 502, fmt.Errorf("kubectl 代理查询失败: %v\n%s", err, truncateStr(string(out), 300))
|
||||
}
|
||||
return out, 200, nil
|
||||
}
|
||||
|
||||
// url 直连
|
||||
if src.URL == "" {
|
||||
return nil, 500, fmt.Errorf("数据源未配置访问地址")
|
||||
}
|
||||
full := strings.TrimSuffix(src.URL, "/") + path
|
||||
if qs := query.Encode(); qs != "" {
|
||||
full += "?" + qs
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, full, nil)
|
||||
if err != nil {
|
||||
return nil, 500, err
|
||||
}
|
||||
applyLokiAuth(req, src)
|
||||
client := &http.Client{Timeout: 60 * time.Second, Transport: &http.Transport{
|
||||
TLSClientConfig: &tls.Config{InsecureSkipVerify: src.SkipTLSVerify},
|
||||
}}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, 502, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, 32<<20))
|
||||
if err != nil {
|
||||
return nil, 502, err
|
||||
}
|
||||
if resp.StatusCode >= 400 {
|
||||
return body, resp.StatusCode, fmt.Errorf("Loki 返回 HTTP %d: %s", resp.StatusCode, truncateStr(string(body), 300))
|
||||
}
|
||||
return body, resp.StatusCode, nil
|
||||
}
|
||||
|
||||
func applyLokiAuth(req *http.Request, src *models.LogSource) {
|
||||
switch src.AuthType {
|
||||
case "basic":
|
||||
if pass, err := DecryptCred(src.BasicPass); err == nil {
|
||||
req.Header.Set("Authorization", "Basic "+
|
||||
base64.StdEncoding.EncodeToString([]byte(src.BasicUser+":"+pass)))
|
||||
}
|
||||
case "bearer":
|
||||
if token, err := DecryptCred(src.BearerToken); err == nil {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
}
|
||||
for name, encrypted := range src.Headers {
|
||||
if validSourceHeader(name) {
|
||||
if value, err := DecryptCred(encrypted); err == nil {
|
||||
req.Header.Set(name, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func truncateStr(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "..."
|
||||
}
|
||||
|
||||
// ==================== 查询实现(契约对齐 aiops) ====================
|
||||
|
||||
type LogEntryRow struct {
|
||||
Timestamp string `json:"timestamp"`
|
||||
TimestampNs string `json:"timestampNs"`
|
||||
Level string `json:"level"`
|
||||
Message string `json:"message"`
|
||||
Labels map[string]string `json:"labels"`
|
||||
}
|
||||
|
||||
type LogQueryResult struct {
|
||||
Entries []LogEntryRow `json:"entries"`
|
||||
HasMore bool `json:"hasMore"`
|
||||
NextCursor string `json:"nextCursor"`
|
||||
}
|
||||
|
||||
// QueryLokiLogs 日志查询:flatten streams → 统一行;cursor(最后一条纳秒时间戳)向后翻页
|
||||
func QueryLokiLogs(ctx context.Context, src *models.LogSource, query string, start, end int64, limit int, direction, cursor string) (*LogQueryResult, error) {
|
||||
if limit <= 0 {
|
||||
limit = 200
|
||||
}
|
||||
if limit > maxLogQueryLimit {
|
||||
limit = maxLogQueryLimit
|
||||
}
|
||||
if direction == "" {
|
||||
direction = "backward"
|
||||
}
|
||||
if end <= 0 {
|
||||
end = time.Now().UnixNano()
|
||||
}
|
||||
if start <= 0 {
|
||||
start = end - int64(time.Hour) // 默认最近 1 小时
|
||||
}
|
||||
if cursor != "" {
|
||||
if c, err := strconv.ParseInt(cursor, 10, 64); err == nil {
|
||||
if direction == "forward" {
|
||||
start = c + 1
|
||||
} else {
|
||||
end = c - 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
q := url.Values{}
|
||||
q.Set("query", query)
|
||||
q.Set("start", strconv.FormatInt(start, 10))
|
||||
q.Set("end", strconv.FormatInt(end, 10))
|
||||
q.Set("limit", strconv.Itoa(limit))
|
||||
q.Set("direction", direction)
|
||||
|
||||
body, _, err := lokiProxyGET(ctx, src, "/loki/api/v1/query_range", q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp struct {
|
||||
Status string `json:"status"`
|
||||
Data struct {
|
||||
ResultType string `json:"resultType"`
|
||||
Result []struct {
|
||||
Stream map[string]string `json:"stream"`
|
||||
Values [][2]string `json:"values"`
|
||||
} `json:"result"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return nil, fmt.Errorf("解析 Loki 响应失败: %w", err)
|
||||
}
|
||||
|
||||
entries := make([]LogEntryRow, 0, limit)
|
||||
var lastTs string
|
||||
for _, stream := range resp.Data.Result {
|
||||
for _, v := range stream.Values {
|
||||
entries = append(entries, LogEntryRow{
|
||||
Timestamp: formatLokiTimestamp(v[0]),
|
||||
TimestampNs: v[0],
|
||||
Level: detectLevel(stream.Stream, v[1]),
|
||||
Message: v[1],
|
||||
Labels: stream.Stream,
|
||||
})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(entries, func(i, j int) bool {
|
||||
a, _ := strconv.ParseInt(entries[i].TimestampNs, 10, 64)
|
||||
b, _ := strconv.ParseInt(entries[j].TimestampNs, 10, 64)
|
||||
if direction == "forward" {
|
||||
return a < b
|
||||
}
|
||||
return a > b
|
||||
})
|
||||
if len(entries) > limit {
|
||||
entries = entries[:limit]
|
||||
}
|
||||
if len(entries) > 0 {
|
||||
lastTs = entries[len(entries)-1].TimestampNs
|
||||
}
|
||||
|
||||
nextCursor := ""
|
||||
if len(entries) >= limit && lastTs != "" {
|
||||
if ts, err := strconv.ParseInt(strings.Split(lastTs, ".")[0], 10, 64); err == nil {
|
||||
nextCursor = strconv.FormatInt(ts, 10)
|
||||
}
|
||||
}
|
||||
return &LogQueryResult{Entries: entries, HasMore: nextCursor != "", NextCursor: nextCursor}, nil
|
||||
}
|
||||
|
||||
// formatLokiTimestamp Loki 时间串(纳秒字符串/科学计数法)→ RFC3339
|
||||
func formatLokiTimestamp(ts string) string {
|
||||
if i := strings.IndexAny(ts, ".eE"); i > 0 {
|
||||
ts = ts[:i]
|
||||
}
|
||||
n, err := strconv.ParseInt(ts, 10, 64)
|
||||
if err != nil {
|
||||
return ts
|
||||
}
|
||||
// 自适应纳秒/微秒/毫秒/秒
|
||||
switch {
|
||||
case n > 1e18:
|
||||
return time.Unix(0, n).Format(time.RFC3339)
|
||||
case n > 1e15:
|
||||
return time.Unix(0, n*1e3).Format(time.RFC3339)
|
||||
case n > 1e12:
|
||||
return time.Unix(0, n*1e6).Format(time.RFC3339)
|
||||
case n > 1e9:
|
||||
return time.Unix(0, n*1e9).Format(time.RFC3339)
|
||||
default:
|
||||
return time.Unix(n, 0).Format(time.RFC3339)
|
||||
}
|
||||
}
|
||||
|
||||
// detectLevel 优先取 level 标签;否则按行内容推断
|
||||
func detectLevel(labels map[string]string, line string) string {
|
||||
if lv, ok := labels["level"]; ok && lv != "" {
|
||||
return strings.ToLower(lv)
|
||||
}
|
||||
l := strings.ToLower(line)
|
||||
for _, p := range []string{"error", "err", "fatal", "panic"} {
|
||||
if strings.Contains(l, p) {
|
||||
return "error"
|
||||
}
|
||||
}
|
||||
if strings.Contains(l, "warn") {
|
||||
return "warn"
|
||||
}
|
||||
return "info"
|
||||
}
|
||||
|
||||
// LokiLabels 标签名列表
|
||||
func LokiLabels(ctx context.Context, src *models.LogSource, start, end int64) ([]string, error) {
|
||||
q := url.Values{}
|
||||
if end > 0 {
|
||||
q.Set("end", strconv.FormatInt(end, 10))
|
||||
}
|
||||
if start > 0 {
|
||||
q.Set("start", strconv.FormatInt(start, 10))
|
||||
}
|
||||
body, _, err := lokiProxyGET(ctx, src, "/loki/api/v1/labels", q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp struct {
|
||||
Data []string `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp.Data, nil
|
||||
}
|
||||
|
||||
// LokiLabelValues 标签值列表(query 只取 {...} selector 部分)
|
||||
func LokiLabelValues(ctx context.Context, src *models.LogSource, name, selector string, start, end int64) ([]string, error) {
|
||||
q := url.Values{}
|
||||
if s := extractStreamSelector(selector); s != "" && s != "{}" {
|
||||
q.Set("query", s)
|
||||
}
|
||||
if end > 0 {
|
||||
q.Set("end", strconv.FormatInt(end, 10))
|
||||
}
|
||||
if start > 0 {
|
||||
q.Set("start", strconv.FormatInt(start, 10))
|
||||
}
|
||||
body, _, err := lokiProxyGET(ctx, src, "/loki/api/v1/label/"+url.PathEscape(name)+"/values", q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp struct {
|
||||
Data []string `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp.Data, nil
|
||||
}
|
||||
|
||||
// extractStreamSelector 从 LogQL 中提取 {...} 部分(labels API 仅接受 stream selector)
|
||||
func extractStreamSelector(query string) string {
|
||||
i := strings.Index(query, "{")
|
||||
if i < 0 {
|
||||
return ""
|
||||
}
|
||||
j := strings.Index(query[i:], "}")
|
||||
if j < 0 {
|
||||
return ""
|
||||
}
|
||||
return query[i : i+j+1]
|
||||
}
|
||||
|
||||
// LokiHistogram 日志量直方图:服务端包 sum(count_over_time(q[step]))
|
||||
func LokiHistogram(ctx context.Context, src *models.LogSource, query string, start, end int64) ([]map[string]interface{}, error) {
|
||||
if end <= 0 {
|
||||
end = time.Now().UnixNano()
|
||||
}
|
||||
if start <= 0 {
|
||||
start = end - int64(time.Hour)
|
||||
}
|
||||
span := end - start
|
||||
step := pickHistogramStep(span)
|
||||
wrapped := fmt.Sprintf("sum(count_over_time(%s[%s]))", query, step)
|
||||
|
||||
q := url.Values{}
|
||||
q.Set("query", wrapped)
|
||||
q.Set("start", strconv.FormatInt(start, 10))
|
||||
q.Set("end", strconv.FormatInt(end, 10))
|
||||
q.Set("step", step)
|
||||
body, _, err := lokiProxyGET(ctx, src, "/loki/api/v1/query_range", q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var resp struct {
|
||||
Data struct {
|
||||
Result []struct {
|
||||
Values [][2]interface{} `json:"values"`
|
||||
} `json:"result"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
buckets := []map[string]interface{}{}
|
||||
if len(resp.Data.Result) > 0 {
|
||||
for _, v := range resp.Data.Result[0].Values {
|
||||
// metric API 的 ts 为秒级浮点数(如 1.79e+09),query 日志的才是纳秒字符串
|
||||
tsf, _ := strconv.ParseFloat(fmt.Sprintf("%v", v[0]), 64)
|
||||
cnt, _ := strconv.ParseFloat(fmt.Sprintf("%v", v[1]), 64)
|
||||
buckets = append(buckets, map[string]interface{}{
|
||||
"timestamp": time.Unix(int64(tsf), 0).Format(time.RFC3339),
|
||||
"count": int64(cnt),
|
||||
})
|
||||
}
|
||||
}
|
||||
return buckets, nil
|
||||
}
|
||||
|
||||
// pickHistogramStep 按时间跨度自适应步长(aiops 同款)
|
||||
func pickHistogramStep(spanNano int64) string {
|
||||
span := time.Duration(spanNano)
|
||||
switch {
|
||||
case span <= 15*time.Minute:
|
||||
return "30s"
|
||||
case span <= time.Hour:
|
||||
return "1m"
|
||||
case span <= 6*time.Hour:
|
||||
return "5m"
|
||||
case span <= 24*time.Hour:
|
||||
return "15m"
|
||||
case span <= 7*24*time.Hour:
|
||||
return "1h"
|
||||
default:
|
||||
return "6h"
|
||||
}
|
||||
}
|
||||
|
||||
// DeploymentReady 部署级就绪探测:按部署名匹配自动注册的数据源,经其访问 Loki /ready
|
||||
func DeploymentReady(ctx context.Context, deploymentName string) (bool, string) {
|
||||
if deploymentName == "" {
|
||||
return false, ""
|
||||
}
|
||||
db := repository.GetDB()
|
||||
db.RLock()
|
||||
var src *models.LogSource
|
||||
for i := range db.LogSources {
|
||||
if db.LogSources[i].Name == deploymentName {
|
||||
s := db.LogSources[i]
|
||||
src = &s
|
||||
break
|
||||
}
|
||||
}
|
||||
db.RUnlock()
|
||||
if src == nil {
|
||||
return false, "未注册数据源"
|
||||
}
|
||||
if LokiReady(ctx, src) {
|
||||
return true, ""
|
||||
}
|
||||
return false, "Loki 未就绪"
|
||||
}
|
||||
|
||||
// LokiReady Loki /ready 探活(部署卡片/查询页状态点)
|
||||
func LokiReady(ctx context.Context, src *models.LogSource) bool {
|
||||
body, code, err := lokiProxyGET(ctx, src, "/ready", nil)
|
||||
if err != nil || code != 200 {
|
||||
return false
|
||||
}
|
||||
return strings.Contains(strings.ToLower(string(body)), "ready")
|
||||
}
|
||||
|
||||
// TestLogSource 数据源连通性(/loki/api/v1/labels 探活)
|
||||
func TestLogSource(ctx context.Context, src *models.LogSource) map[string]interface{} {
|
||||
labels, err := LokiLabels(ctx, src, 0, 0)
|
||||
if err != nil {
|
||||
return map[string]interface{}{"success": false, "message": err.Error()}
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"success": true,
|
||||
"message": fmt.Sprintf("连接成功,发现 %d 个标签", len(labels)),
|
||||
"labels": labels,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"opsxloki/models"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAuthorizedLogQuery(t *testing.T) {
|
||||
src := &models.LogSource{Access: []models.LogSourceAccess{{UserID: 7, Labels: map[string]string{"namespace": "prod", "pod": "loki-0"}}}}
|
||||
if _, err := AuthorizedLogQuery(src, 8, false, `{app="api"}`); err == nil {
|
||||
t.Fatal("ungranted user queried source")
|
||||
}
|
||||
q, err := AuthorizedLogQuery(src, 7, false, `{app="api"} |= "error"`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, part := range []string{`app="api"`, `namespace="prod"`, `pod="loki-0"`, `|= "error"`} {
|
||||
if !strings.Contains(q, part) {
|
||||
t.Fatalf("missing %q in %q", part, q)
|
||||
}
|
||||
}
|
||||
if _, err := AuthorizedLogQuery(src, 7, false, `sum(count_over_time({app="api"}[1m]))`); err == nil {
|
||||
t.Fatal("metric query bypassed stream restriction")
|
||||
}
|
||||
if _, err := AuthorizedLogQuery(src, 7, false, `{app="api"} or {app="other"}`); err == nil {
|
||||
t.Fatal("second selector bypassed stream restriction")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolicyGroupAccess(t *testing.T) {
|
||||
groups := []models.LogPolicyGroup{{SourceID: 2, Labels: map[string]string{"namespace": "log", "app": "api"}, UserIDs: []uint{7, 8}}}
|
||||
if got := policyGroupAccess(groups, 2, 8); got == nil || got.Labels["app"] != "api" {
|
||||
t.Fatal("group member should inherit exact label scope")
|
||||
}
|
||||
if got := policyGroupAccess(groups, 3, 8); got != nil {
|
||||
t.Fatal("group must not grant access to another source")
|
||||
}
|
||||
if got := policyGroupAccess(groups, 2, 9); got != nil {
|
||||
t.Fatal("nonmember gained access")
|
||||
}
|
||||
}
|
||||
@@ -10,8 +10,8 @@ import (
|
||||
"strings"
|
||||
"text/template"
|
||||
|
||||
"lokideploy/models"
|
||||
"lokideploy/pkg/crypto"
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
)
|
||||
|
||||
// LokiDeployer 负责 Loki 的三种部署模式:k8s-helm / k8s-yaml / host-binary
|
||||
@@ -34,16 +34,30 @@ func (d *LokiDeployer) Deploy(ctx context.Context) error {
|
||||
d.log("warn", err.Error(), "")
|
||||
}
|
||||
|
||||
var err error
|
||||
switch d.deployment.Mode {
|
||||
case "k8s-helm":
|
||||
return d.deployHelm(ctx)
|
||||
err = d.deployHelm(ctx)
|
||||
case "k8s-yaml":
|
||||
return d.deployYaml(ctx)
|
||||
err = d.deployYaml(ctx)
|
||||
case "host-binary":
|
||||
return d.deployBinary(ctx)
|
||||
err = d.deployBinary(ctx)
|
||||
default:
|
||||
return fmt.Errorf("unsupported deployment mode: %s", d.deployment.Mode)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Loki 就绪后,按需同环境部署 Grafana(数据源自动指向本 Loki)
|
||||
if d.deployment.GrafanaEnabled {
|
||||
g := &GrafanaDeployer{deployment: d.deployment, log: d.log}
|
||||
if gerr := g.Deploy(ctx); gerr != nil {
|
||||
d.log("error", fmt.Sprintf("Grafana deployment failed: %v", gerr), "")
|
||||
return fmt.Errorf("grafana deployment failed: %w", gerr)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ==================== Helm 部署 (grafana/loki v6+) ====================
|
||||
@@ -169,12 +183,7 @@ func (d *LokiDeployer) deployBinary(ctx context.Context) error {
|
||||
return fmt.Errorf("please select target host")
|
||||
}
|
||||
host := d.deployment.Host
|
||||
password := host.Password
|
||||
if strings.HasPrefix(password, "ENC:") {
|
||||
password, _ = DecryptCred(password)
|
||||
}
|
||||
|
||||
client, err := NewSSHClient(host.HostIP, host.Port, host.User, password)
|
||||
client, err := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -696,7 +705,6 @@ common:
|
||||
filesystem:
|
||||
chunks_directory: /var/lib/loki/chunks
|
||||
rules_directory: /var/lib/loki/rules
|
||||
admin_api_directory: /var/lib/loki/admin
|
||||
replication_factor: {{ .ReplicationFactor }}
|
||||
ring:
|
||||
instance_addr: {{ .RingInstanceAddr }}
|
||||
@@ -730,20 +738,11 @@ ingester:
|
||||
chunk_target_size: 1048576
|
||||
chunk_encoding: snappy
|
||||
max_chunk_age: 2h
|
||||
max_transfer_retries: 0
|
||||
autoforget_unhealthy: true
|
||||
|
||||
querier:
|
||||
max_concurrent: 10
|
||||
query_ingesters_within: 2h
|
||||
engine:
|
||||
timeout: 3m
|
||||
max_look_back_period: 30s
|
||||
|
||||
query_frontend:
|
||||
max_outstanding_per_tenant: 4096
|
||||
compress_responses: true
|
||||
scheduler_address: ""
|
||||
|
||||
query_scheduler:
|
||||
max_outstanding_requests_per_tenant: 4096
|
||||
@@ -786,6 +785,9 @@ schema_config:
|
||||
ruler:
|
||||
storage:
|
||||
type: {{ .RulerStoreType }}
|
||||
{{- if .RulerStorageBlock }}
|
||||
{{ .RulerStorageBlock }}
|
||||
{{- end }}
|
||||
{{- if eq .RulerStoreType "local" }}
|
||||
local:
|
||||
directory: /var/lib/loki/rules
|
||||
@@ -802,12 +804,7 @@ frontend:
|
||||
max_outstanding_per_tenant: 4096
|
||||
compress_responses: true
|
||||
|
||||
frontend_worker:
|
||||
parallelism: 10
|
||||
|
||||
table_manager:
|
||||
retention_deletes_enabled: true
|
||||
retention_period: {{ .RetentionPeriod }}h
|
||||
frontend_worker: {}
|
||||
|
||||
chunk_store_config:
|
||||
chunk_cache_config:
|
||||
@@ -831,6 +828,7 @@ type lokiConfigData struct {
|
||||
RingInstanceAddr string // 单副本 127.0.0.1;多副本 0.0.0.0(须可被对端寻址)
|
||||
RingKVStore string // 单副本 inmemory;多副本 memberlist
|
||||
MemberlistBlock string // 多副本时非空,注入 memberlist 顶级配置
|
||||
RulerStorageBlock string // 对象存储时非空,注入 ruler.storage.s3/gcs/azure 嵌套配置
|
||||
}
|
||||
|
||||
func (d *LokiDeployer) generateLokiConfig() string {
|
||||
@@ -870,7 +868,7 @@ func (d *LokiDeployer) generateLokiConfig() string {
|
||||
ringStore = "memberlist"
|
||||
ns := d.deployment.Namespace
|
||||
if ns == "" {
|
||||
ns = "loki"
|
||||
ns = "log"
|
||||
}
|
||||
memberlistBlock = fmt.Sprintf(`memberlist:
|
||||
join_members:
|
||||
@@ -892,6 +890,7 @@ func (d *LokiDeployer) generateLokiConfig() string {
|
||||
RingInstanceAddr: ringAddr,
|
||||
RingKVStore: ringStore,
|
||||
MemberlistBlock: memberlistBlock,
|
||||
RulerStorageBlock: d.buildRulerStorageBlock(),
|
||||
}
|
||||
|
||||
tmpl, err := template.New("loki-config").Parse(lokiConfigTemplate)
|
||||
@@ -962,6 +961,63 @@ func (d *LokiDeployer) buildStorageConfigBlock() string {
|
||||
}
|
||||
}
|
||||
|
||||
// buildRulerStorageBlock 生成 ruler.storage 下与存储类型同名的嵌套配置
|
||||
// (Loki 3.x 的 ruler 存储客户端不回退读取 storage_config,缺少此块会以空 bucket 启动失败)
|
||||
func (d *LokiDeployer) buildRulerStorageBlock() string {
|
||||
if d.deployment.StorageConfig == nil {
|
||||
return ""
|
||||
}
|
||||
sc := d.deployment.StorageConfig
|
||||
switch sc.StorageType {
|
||||
case "s3":
|
||||
endpoint := sc.Endpoint
|
||||
if endpoint == "" {
|
||||
endpoint = fmt.Sprintf("s3.%s.amazonaws.com", sc.Region)
|
||||
}
|
||||
insecure := "false"
|
||||
if sc.Insecure {
|
||||
insecure = "true"
|
||||
}
|
||||
return fmt.Sprintf(` s3:
|
||||
s3: s3://%s/%s
|
||||
endpoint: %s
|
||||
region: %s
|
||||
access_key_id: %s
|
||||
secret_access_key: %s
|
||||
insecure: %s
|
||||
s3forcepathstyle: true`, sc.Bucket, sc.PathPrefix, endpoint, sc.Region, sc.AccessKey, sc.SecretKey, insecure)
|
||||
case "oss":
|
||||
return fmt.Sprintf(` s3:
|
||||
s3: s3://%s/%s
|
||||
endpoint: %s
|
||||
access_key_id: %s
|
||||
secret_access_key: %s
|
||||
s3forcepathstyle: true`, sc.Bucket, sc.PathPrefix, sc.Endpoint, sc.AccessKey, sc.SecretKey)
|
||||
case "minio":
|
||||
insecure := "false"
|
||||
if sc.Insecure {
|
||||
insecure = "true"
|
||||
}
|
||||
return fmt.Sprintf(` s3:
|
||||
s3: s3://%s/%s
|
||||
endpoint: %s
|
||||
access_key_id: %s
|
||||
secret_access_key: %s
|
||||
insecure: %s
|
||||
s3forcepathstyle: true`, sc.Bucket, sc.PathPrefix, sc.Endpoint, sc.AccessKey, sc.SecretKey, insecure)
|
||||
case "gcs":
|
||||
return fmt.Sprintf(` gcs:
|
||||
bucket_name: %s`, sc.Bucket)
|
||||
case "azure":
|
||||
return fmt.Sprintf(` azure:
|
||||
container_name: %s
|
||||
account_name: %s
|
||||
account_key: %s`, sc.Bucket, sc.AccessKey, sc.SecretKey)
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// generateLokiConfigFallback 当模板解析失败时的降级配置
|
||||
func (d *LokiDeployer) generateLokiConfigFallback(retentionHours int) string {
|
||||
return fmt.Sprintf(`auth_enabled: %v
|
||||
@@ -996,8 +1052,6 @@ ingester:
|
||||
max_chunk_age: 2h
|
||||
querier:
|
||||
max_concurrent: 10
|
||||
query_frontend:
|
||||
max_outstanding_per_tenant: 4096
|
||||
compactor:
|
||||
working_directory: /var/lib/loki/compactor
|
||||
retention_enabled: true
|
||||
@@ -1033,7 +1087,7 @@ func (d *LokiDeployer) effectiveNamespace() string {
|
||||
if d.deployment.Namespace != "" {
|
||||
return d.deployment.Namespace
|
||||
}
|
||||
return "loki"
|
||||
return "log"
|
||||
}
|
||||
|
||||
func (d *LokiDeployer) effectiveReleaseName() string {
|
||||
@@ -1075,6 +1129,11 @@ func (d *LokiDeployer) log(level, message, detail string) {
|
||||
// ==================== 清理 / 卸载 ====================
|
||||
|
||||
func CleanupLoki(ctx context.Context, deployment *models.LokiDeployment) {
|
||||
// 同环境部署的 Grafana 一并清理
|
||||
if deployment.GrafanaEnabled {
|
||||
CleanupGrafana(ctx, deployment)
|
||||
}
|
||||
|
||||
if deployment.Mode == "k8s-helm" && deployment.Cluster != nil {
|
||||
kubeconfig, err := resolveKubeconfig(deployment.Cluster)
|
||||
if err != nil {
|
||||
@@ -1089,7 +1148,7 @@ func CleanupLoki(ctx context.Context, deployment *models.LokiDeployment) {
|
||||
env := append(os.Environ(), fmt.Sprintf("KUBECONFIG=%s", kubeFile))
|
||||
ns := deployment.Namespace
|
||||
if ns == "" {
|
||||
ns = "loki"
|
||||
ns = "log"
|
||||
}
|
||||
name := deployment.Name
|
||||
if name == "" {
|
||||
@@ -1121,7 +1180,7 @@ func CleanupLoki(ctx context.Context, deployment *models.LokiDeployment) {
|
||||
env := append(os.Environ(), fmt.Sprintf("KUBECONFIG=%s", kubeFile))
|
||||
ns := deployment.Namespace
|
||||
if ns == "" {
|
||||
ns = "loki"
|
||||
ns = "log"
|
||||
}
|
||||
|
||||
// 删除所有相关资源
|
||||
@@ -1140,11 +1199,7 @@ func CleanupLoki(ctx context.Context, deployment *models.LokiDeployment) {
|
||||
|
||||
if deployment.Mode == "host-binary" && deployment.Host != nil {
|
||||
host := deployment.Host
|
||||
password := host.Password
|
||||
if strings.HasPrefix(password, "ENC:") {
|
||||
password, _ = DecryptCred(password)
|
||||
}
|
||||
client, _ := NewSSHClient(host.HostIP, host.Port, host.User, password)
|
||||
client, _ := NewSSHClientWithAuth(HostSSHAuth(host))
|
||||
if client != nil {
|
||||
defer client.Close()
|
||||
client.Execute(ctx, "systemctl stop loki 2>/dev/null || true")
|
||||
|
||||
+16
-3
@@ -14,9 +14,9 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"lokideploy/config"
|
||||
"lokideploy/models"
|
||||
"lokideploy/pkg/crypto"
|
||||
"opsxloki/config"
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
|
||||
"golang.org/x/crypto/ssh"
|
||||
)
|
||||
@@ -160,6 +160,19 @@ func buildAuthMethods(auth *models.SSHAuthConfig) []ssh.AuthMethod {
|
||||
return methods
|
||||
}
|
||||
|
||||
// HostSSHAuth 从 Host 模型构建完整 SSH 认证配置(密钥优先、密码后备)。
|
||||
// 凭证保持 ENC: 原样传入,由 buildAuthMethods 统一解密。
|
||||
func HostSSHAuth(host *models.Host) *models.SSHAuthConfig {
|
||||
return &models.SSHAuthConfig{
|
||||
Host: host.HostIP,
|
||||
Port: host.Port,
|
||||
User: host.User,
|
||||
Password: host.Password,
|
||||
PrivateKey: host.PrivateKey,
|
||||
KeyPhrase: host.KeyPhrase,
|
||||
}
|
||||
}
|
||||
|
||||
// NewSSHClient 创建SSH客户端(支持密钥+密码认证,TOFU主机密钥验证)
|
||||
func NewSSHClient(host string, port int, user, password string) (*SSHClient, error) {
|
||||
auth := &SSHAuthConfig{
|
||||
|
||||
@@ -4,10 +4,10 @@ import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"lokideploy/models"
|
||||
"lokideploy/pkg/crypto"
|
||||
"lokideploy/repository"
|
||||
"lokideploy/utils"
|
||||
"opsxloki/models"
|
||||
"opsxloki/pkg/crypto"
|
||||
"opsxloki/repository"
|
||||
"opsxloki/utils"
|
||||
)
|
||||
|
||||
type StorageService struct{}
|
||||
|
||||
@@ -0,0 +1,216 @@
|
||||
package services
|
||||
|
||||
import (
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// StorageTestResult 存储连接测试结果(逐级:可达 → 认证 → 桶)
|
||||
type StorageTestResult struct {
|
||||
Reachable bool `json:"reachable"`
|
||||
Authenticated bool `json:"authenticated"`
|
||||
BucketExists bool `json:"bucketExists"`
|
||||
BucketCreated bool `json:"bucketCreated"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// StorageTestParams 测试参数(与创建存储配置一致,额外支持自动建桶)
|
||||
type StorageTestParams struct {
|
||||
StorageType string
|
||||
Endpoint string
|
||||
Bucket string
|
||||
Region string
|
||||
AccessKey string
|
||||
SecretKey string
|
||||
Insecure bool
|
||||
CreateIfMissing bool
|
||||
}
|
||||
|
||||
// ==================== S3 SigV4(零依赖实现,兼容 MinIO/RustFS/S3/OSS S3 兼容端点) ====================
|
||||
|
||||
func hmacSHA256(key []byte, data string) []byte {
|
||||
h := hmac.New(sha256.New, key)
|
||||
h.Write([]byte(data))
|
||||
return h.Sum(nil)
|
||||
}
|
||||
|
||||
func sha256Hex(s string) string {
|
||||
sum := sha256.Sum256([]byte(s))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// s3Request 发起一次 SigV4 签名的 S3 请求(路径风格),path 为 "/" 或 "/<bucket>"
|
||||
func s3Request(method, endpoint, region, accessKey, secretKey, path string, insecure bool) (*http.Response, error) {
|
||||
// endpoint 解析:带 scheme 用之;裸主机名默认 https,insecure 时 http
|
||||
base := strings.TrimSpace(endpoint)
|
||||
scheme := ""
|
||||
if strings.HasPrefix(base, "http://") {
|
||||
scheme, base = "http", strings.TrimPrefix(base, "http://")
|
||||
} else if strings.HasPrefix(base, "https://") {
|
||||
scheme, base = "https", strings.TrimPrefix(base, "https://")
|
||||
} else {
|
||||
if insecure {
|
||||
scheme = "http"
|
||||
} else {
|
||||
scheme = "https"
|
||||
}
|
||||
}
|
||||
base = strings.TrimSuffix(base, "/")
|
||||
if region == "" {
|
||||
region = "us-east-1"
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
amzDate := now.Format("20060102T150405Z")
|
||||
dateStamp := now.Format("20060102")
|
||||
payloadHash := sha256Hex("") // 空 body
|
||||
|
||||
// 规范请求(SigV4: host/x-amz-content-sha256/x-amz-date 参与签名)
|
||||
canonicalURI := path
|
||||
if canonicalURI == "" {
|
||||
canonicalURI = "/"
|
||||
}
|
||||
canonicalRequest := strings.Join([]string{
|
||||
method,
|
||||
canonicalURI,
|
||||
"", // query
|
||||
"host:" + base + "\n" +
|
||||
"x-amz-content-sha256:" + payloadHash + "\n" +
|
||||
"x-amz-date:" + amzDate + "\n",
|
||||
"host;x-amz-content-sha256;x-amz-date",
|
||||
payloadHash,
|
||||
}, "\n")
|
||||
|
||||
scope := dateStamp + "/" + region + "/s3/aws4_request"
|
||||
stringToSign := strings.Join([]string{
|
||||
"AWS4-HMAC-SHA256",
|
||||
amzDate,
|
||||
scope,
|
||||
sha256Hex(canonicalRequest),
|
||||
}, "\n")
|
||||
|
||||
kDate := hmacSHA256([]byte("AWS4"+secretKey), dateStamp)
|
||||
kRegion := hmacSHA256(kDate, region)
|
||||
kService := hmacSHA256(kRegion, "s3")
|
||||
kSigning := hmacSHA256(kService, "aws4_request")
|
||||
signature := hex.EncodeToString(hmacSHA256(kSigning, stringToSign))
|
||||
|
||||
req, err := http.NewRequest(method, scheme+"://"+base+canonicalURI, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Host = base
|
||||
req.Header.Set("x-amz-date", amzDate)
|
||||
req.Header.Set("x-amz-content-sha256", payloadHash)
|
||||
req.Header.Set("Authorization", fmt.Sprintf(
|
||||
"AWS4-HMAC-SHA256 Credential=%s/%s, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=%s",
|
||||
accessKey, scope, signature))
|
||||
|
||||
client := &http.Client{Timeout: 10 * time.Second}
|
||||
return client.Do(req)
|
||||
}
|
||||
|
||||
// s3ErrBrief 从 S3 错误响应 XML 中提取 Code/Message,避免把整段 XML 暴露给用户
|
||||
func s3ErrBrief(body []byte) string {
|
||||
var e struct {
|
||||
Code string `xml:"Code"`
|
||||
Message string `xml:"Message"`
|
||||
}
|
||||
if err := xml.Unmarshal(body, &e); err == nil && (e.Code != "" || e.Message != "") {
|
||||
if e.Code != "" && e.Message != "" {
|
||||
return e.Code + ": " + e.Message
|
||||
}
|
||||
return e.Code + e.Message
|
||||
}
|
||||
brief := strings.TrimSpace(string(body))
|
||||
if len(brief) > 120 {
|
||||
brief = brief[:120] + "..."
|
||||
}
|
||||
return brief
|
||||
}
|
||||
|
||||
// ==================== 测试连接 ====================
|
||||
|
||||
// TestStorageConnection 逐级测试对象存储:可达性 → 凭证 → 桶存在性(可选自动创建)。
|
||||
// OSS 走其 S3 兼容端点;不覆盖 OSS 原生协议的独有能力。
|
||||
func TestStorageConnection(p StorageTestParams) StorageTestResult {
|
||||
// ① ListBuckets:同时验证 网络可达 + AK/SK 有效
|
||||
resp, err := s3Request("GET", p.Endpoint, p.Region, p.AccessKey, p.SecretKey, "/", p.Insecure)
|
||||
if err != nil {
|
||||
return StorageTestResult{Message: fmt.Sprintf("无法连接对象存储:%v", err)}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode == http.StatusForbidden || resp.StatusCode == http.StatusUnauthorized {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
return StorageTestResult{
|
||||
Reachable: true,
|
||||
Message: fmt.Sprintf("连接成功但认证失败(HTTP %d),请检查 Access Key / Secret Key。%s", resp.StatusCode, s3ErrBrief(body)),
|
||||
}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
return StorageTestResult{
|
||||
Reachable: true,
|
||||
Message: fmt.Sprintf("对象存储返回异常(HTTP %d):%s", resp.StatusCode, s3ErrBrief(body)),
|
||||
}
|
||||
}
|
||||
|
||||
// 解析 ListBuckets,顺带确认目标桶是否在列表中
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
var listResp struct {
|
||||
Buckets struct {
|
||||
Bucket []struct {
|
||||
Name string `xml:"Name"`
|
||||
} `xml:"Bucket"`
|
||||
} `xml:"Buckets"`
|
||||
}
|
||||
_ = xml.Unmarshal(body, &listResp)
|
||||
found := false
|
||||
for _, b := range listResp.Buckets.Bucket {
|
||||
if b.Name == p.Bucket {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
result := StorageTestResult{Reachable: true, Authenticated: true, BucketExists: found}
|
||||
|
||||
if found {
|
||||
result.Message = "连接成功,凭证有效,桶已存在"
|
||||
return result
|
||||
}
|
||||
|
||||
// ② 桶不存在:按需创建
|
||||
if !p.CreateIfMissing {
|
||||
result.Message = fmt.Sprintf("连接成功,凭证有效,但桶「%s」不存在;勾选「不存在时自动创建」可一键创建", p.Bucket)
|
||||
return result
|
||||
}
|
||||
|
||||
putResp, err := s3Request("PUT", p.Endpoint, p.Region, p.AccessKey, p.SecretKey, "/"+url.PathEscape(p.Bucket), p.Insecure)
|
||||
if err != nil {
|
||||
result.Message = fmt.Sprintf("创建桶失败:%v", err)
|
||||
return result
|
||||
}
|
||||
defer putResp.Body.Close()
|
||||
|
||||
// 200=创建成功;409 BucketAlreadyOwnedByYou 视为已存在
|
||||
if putResp.StatusCode == http.StatusOK || putResp.StatusCode == http.StatusConflict {
|
||||
result.BucketExists = true
|
||||
result.BucketCreated = true
|
||||
result.Message = fmt.Sprintf("连接成功,桶「%s」不存在,已自动创建 ✓", p.Bucket)
|
||||
return result
|
||||
}
|
||||
|
||||
putBody, _ := io.ReadAll(io.LimitReader(putResp.Body, 512))
|
||||
result.Message = fmt.Sprintf("凭证有效,但创建桶失败(HTTP %d):%s", putResp.StatusCode, s3ErrBrief(putBody))
|
||||
return result
|
||||
}
|
||||
Reference in new issue
Block a user