40 Commits
3fbafa0a54
...
6c2e30028c
| Author | SHA1 | Message | Date |
|---|---|---|---|
|
|
6c2e30028c |
Merge branch 'refs/heads/Spring-AI-1.1.2'
|
2 months ago |
|
|
d33cb56e8a |
优化
|
2 months ago |
|
|
0838dca61a |
fix(auth): 修复后台会话长期不失效,JWT 时间配置改用可读单位
问题
- 关闭后台页面数天后重新打开,可跳过登录页直接进入后台
- 根因:refresh token 有效期 7 天且滚动续期无上限;refresh Cookie 为持久化 Cookie
(maxAge 硬编码 7 天)在浏览器关闭后不会删除,重开页面时 App.vue 启动自检发现
/auth/me 返回 401,静默调 /auth/refresh 换发新 token 并重置窗口 —— 只要 7 天内
来访一次即等效永久会话
会话有效期修复
- application.yml:refresh-expiration 由 7 天改为 8h,作为会话时长的唯一配置项,
所有环境统一生效(不在 dev/prod 覆盖)
- auth/AuthController:refresh Cookie 的 Max-Age 由硬编码 7*24*60*60 改为读
jwt.refresh-expiration,消除「改配置不改 Cookie」造成的有效期漂移
- App.vue:刷新 token 成功后二次校验 /auth/me,失败即登出;修复原先
.catch(() => {}) 丢弃返回值导致用户带着空 currentUser 进入后台、权限判断全部
失效、直到某个请求 401 才被踢出的漏洞
JWT 时间配置改用 Duration 可读格式
- expiration / refresh-expiration / sdk-expiration 均绑定为 java.time.Duration,
值写成 15m / 8h / 2h;纯数字仍按毫秒解析,向后兼容旧配置
- JwtTokenProvider 兜底默认值由 604800000(7 天,与 yml 的 8 小时并不一致)修正为 8h
- SdkJwtTokenProvider:默认有效期改 Duration,钳制边界 MIN/MAX_EXPIRATION 改为
Duration.ofMinutes(5) / Duration.ofDays(1),并公开 clampExpirationMillis 供控制器复用
SDK 配置接通
- controller/AuthController(SDK):默认 ttl 改读 jwt.sdk-expiration。此前硬编码
7200000L,而唯一读该配置的 3 参 generateToken 无任何调用方,属改了不生效的死配置
- 控制器内不再出现 300000L / 86400000L 等毫秒魔数,expiresIn 与 token 实际有效期
保持一致(对外的 ttl / expiresIn 单位仍为秒,契约不变)
验证
- 实测 Duration 绑定链路:15m→PT15M(900s)、8h→PT8H(28800s)、2h→PT2H(7200s),
纯数字 28800000→PT8H(28800s),确认向后兼容
- mvn clean package -P prod 构建通过(含前端构建)
文档
- CLAUDE.md 同步会话有效期约定、Duration 单位规范、SDK Token 有效期约定
已知残留
- 仍保留滚动续期,页面持续活跃的用户不会掉线;彻底杜绝需引入绝对过期上限或服务端
落库记录最后活动时间,本次未做
|
2 months ago |
|
|
2001dc3a0f |
fix(doc): 修复重新处理文档会按 2000 字截断预览重建导致内容丢失
问题
- knowledge_document.content 上传时被截断到 2000 字符(有意的预览设计),
但 reprocessDocument 把它当原文重建分块,而 DocumentProcessingService 以
cleanBeforeFirstAttempt=true 运行(先删光旧向量)——对超过 2000 字符的文档
执行 POST /document/batch/reprocess 会永久丢失其余内容
修复
- content 改存原文全文(列已是 TEXT,无需 DDL),并写 extra_config.contentComplete
作为完整性标记
- 实体 content 加 @JsonIgnore:全文任何接口都不返回(否则 GET /document/list
无字段投影,每行都会带上整篇正文);文档详情接口单独组装响应,返回 2000 字预览
(键名仍是 content)与 contentTruncated 标志,前端据此提示截断
- reprocessDocument 的数据源按优先级解析:
1) 有 contentComplete 标记 → 用库内全文(语义最忠实,JSON 的 fields/pointer
模式不受影响,也不依赖文件是否还在)
2) 历史遗留文档(无标记,content 是截断预览)→ 按 fileType 从原始文件重解析
(md → MarkdownDocumentLoader,json → JsonDocumentLoader,其余 → Tika),
成功后回填全文并打标记,此后不再依赖原始文件
3) 两者都不可用 → 明确报错拒绝,绝不静默按截断预览重建
无法完整还原时在标记 PROCESSING 之前抛错,事务回滚,文档状态与已有向量都不受影响
- contentTruncated 判据覆盖历史数据:长度恰好等于上限且无 contentComplete 标记时
同样判定为已截断,避免把截断预览误报成完整内容
- 同步 content 列注释(DatabaseInitConfig / init-database.sql / knowledge-base.sql),
该注释会在下次启动幂等刷新到库
验证(真实库 + 557 篇存量文档环境)
- 上传 4891 字符正文:库内存全文,详情接口返回 2000 字预览 + contentTruncated=true
- 重新处理:chunk_count 与首次完全一致(修复前会降为个位数),内容无丢失
- 模拟历史数据(截断 + 无标记):有文件时从文件重解析成功并回填全文;
文件也丢失时明确拒绝,且文档状态与 chunk_count 保持不变(向量未被破坏)
- 回填后再删文件重跑仍成功(已自愈);GET /document/list 不再含 content
- RAG 对话回归正常
已知降级
- JSON 的 basic/fields/pointer 解析模式上传时未持久化,历史 JSON 文档从文件重解析
只能按 basic 还原(不丢数据,仅抽取口径可能变化,日志有 WARN)
- 存量中疑似被截断的文档需重跑一次才能回填全文(均已保留原始文件,均可恢复)
|
2 months ago |
|
|
94c1a074e7 |
优化
|
2 months ago |
|
|
84eea7564f |
refactor: 自造轮子改用 Spring AI 标准组件、清理死代码并修复既有缺陷
组件替换(改用 1.1.x 标准组件)
- 文档提取:删除自写 TikaDocumentReader(内部直接 new org.apache.tika.Tika()),
改用官方 org.springframework.ai.reader.tika.TikaDocumentReader。
pom 早已引入 spring-ai-tika-document-reader 却从未使用其类,属典型「引了标准依赖却手写实现」
- 意图识别:IntentRouter 的手写正则解析(含去 BOM/零宽字符/全角空格等约 60 行防御逻辑)
改用标准 BeanOutputConverter<IntentResult>,保留原有降级语义(空输入/解析失败 → RAG)
- 推荐问题:SuggestionGenerator 的哨兵字符串 + 代码块剥离 + 按行降级解析改用
ChatClient.entity(ParameterizedTypeReference<List<String>>),整体删除 SuggestionResponseParser
- 分块:删除 MyTokenTextSplitter 薄壳,新增 OverlapTokenTextSplitter
分块器 overlap 缺陷修复(本次验证中发现并修复)
- 旧实现把 knowledge.chunk.overlap 传进了 TokenTextSplitter 第 2 个形参 minChunkSizeChars 位,
overlap 从未生效(Spring AI 的 TokenTextSplitter 根本没有 overlap 形参)
- 新实现继承标准 TextSplitter,复刻 TokenTextSplitter 全部切分语义,仅把前进步长改为
chunkSize - overlap,使重叠真正生效
- 验证中发现:标点截断会缩短本块消耗的 token 数,与 overlap 叠加后可把前进步长压到
1 个 token,分块数膨胀 10 倍(实测 349 块 vs 修复后 44 块)。故仅当截断后仍能前进
至少 (chunkSize - overlap) / 2 个 token 时才采用该截断,否则宁可切断句子
- overlap=0 时与标准 TokenTextSplitter 逐块一致(已对比验证)
- ⚠️ 存量文档需重新分块 + 重新向量化(POST /document/batch/reprocess),否则新旧向量口径混杂
其他既有缺陷修复
- RerankerService:原用 new RestTemplate() 且无任何超时,慢 provider 会把检索线程拖到
TCP 超时;改用 RestClient + JdkClientHttpRequestFactory 显式设置 connect/read 超时(各 3s)
- McpServerConfigController:MCP Server 增删改/启停/全量刷新后未清 AssistantApp 的
ChatClient 缓存,导致继续使用旧工具集;现补调 clearCache()
死代码清理(均已 grep 确认零引用)
- 删除 FileBasedChatMemory、ReReadingAdvisor(零装配且 before() 逻辑为 no-op)、
SseEventBuilder(零引用)、SuggestionResponseParser
- McpToolCallback 删除 EVENTS ThreadLocal 与 drainEvents()/resetEvents()(零调用),
保留在用的 MCP_EVENTS_KEY/MCP_ROUNDS_KEY ToolContext 机制
文档
- 同步更新 CLAUDE.md / README.md / DEPLOY.md 的版本号、组件说明与架构描述
- 修正 CLAUDE.md 中与代码不符的既有描述:主启动类并未排除 PgVectorStoreAutoConfiguration
(项目用的是非 starter 坐标,classpath 上本就没有该自动配置);分类过滤实现已完成,
原「Spring AI filter 支持有限」的 TODO 已不成立
|
2 months ago |
|
|
1731a3ca60 |
chore(deps): 升级 Spring Boot 3.5.8 / Spring AI 1.1.2 / Spring AI Alibaba 1.1.2.2
升级到官方兼容矩阵组合(SAA 1.1.2.2 的官方基线即 Boot 3.5.8 + SAI 1.1.2)。 编译适配 - SAA 1.1.2.2 将内部类 DashscopeChatOptionsBuilder 更名为 DashScopeChatOptionsBuilder - Spring AI 1.1.x 起 AssistantMessage/ToolResponseMessage 带 metadata 的构造器改为 protected,MessageConverter 改用标准 builder 依赖增删 - MCP SDK 继续排除传递依赖并锁定 0.18.3:1.1.2 传递的是 0.17.0,其 JsonMapper 包名 为 json.jackson(无 jackson2),与本项目使用的 json.jackson2 不兼容,不锁版本会编译失败; 已核对 spring-ai-mcp 1.1.2 仅引用 McpSyncClient/McpAsyncClient/McpClient/ McpTransportContext/McpSchema/Assert,这些类 0.18.3 均存在,覆盖安全 - 删除 spring-security-oauth2-client:原为 Spring AI 1.0.x ToolCallingAutoConfiguration 的 ClassNotFound workaround,1.1.2 已无该耦合(已核对字节码常量池零 oauth2 引用) - 删除显式 victools jsonschema-generator:改由 spring-ai-model 传递引入(4.38.0), 顺带消除原先 4.38.0/4.37.0 混用 - 删除 kryo:唯一使用者 FileBasedChatMemory 为死代码,本提交一并删除 - 新增 jtokkit 1.1.0(OverlapTokenTextSplitter 直接使用) - 升 Knife4j 4.5.0、MyBatis-Plus 3.5.14、hutool 5.8.41、jjwt 0.12.7; Lombok 版本交由 Boot 父 POM 托管(1.18.42) 刻意不采纳的升级建议 - 不导入 spring-ai-alibaba-bom:该 BOM 的 dependencyManagement 并不包含 spring-ai-alibaba-starter-dashscope,导入后仍需显式写版本号,无收益 - 保留 spring-ai-openai / spring-ai-pgvector-store 的非 starter 坐标:项目手动构建 ChatModel/EmbeddingModel/VectorStore,换 starter 会额外引入 model-openai、chat-client、 chat-memory 三个自动配置,可能生成与手写工厂、DatabaseChatMemory 冲突的 Bean |
2 months ago |
|
|
a1211e8c1b |
chore(frontend): 更新 SDK 测试面板构建产物引用
|
2 months ago |
|
|
f71e83bf7c |
refactor(frontend): 消息提示统一改用 TDesign 标准组件
- 删除 McpServerManager/ModelConfigManager 手写 fixed 浮层,连接测试结果改用 toast()(MessagePlugin) - 按钮加载态统一使用 :loading 替代 emoji/纯文字切换(刷新连接、测试连接、获取模型) - 登录错误、文档详情错误提示改 t-alert;目录树 / AI 执行链加载态改 t-loading - 导出图片等提示统一走 utils/toast,移除 PipelineFlow 对 MessagePlugin 的直接 import - 同步清理死代码与孤立样式(components.d.ts 补登记 TAlert) |
2 months ago |
|
|
90754d15b2 |
perf(rag): MULTI_QUERY 多路检索并行化并避免 FAQ 重复语义匹配
- 新增 ragRetrieveExecutor 有界线程池 + knowledge.rag.multiquery.parallel 开关: 多路检索默认并行执行(各路 embedding HTTP + PG 查询相互独立,可把 N 路串行降为 1 段最慢路), 按原查询顺序合并去重,结果与串行路径一致;单路异常降级为空不拖垮整次检索; 饱和时退回请求线程串行(CallerRuns),供高并发/embedding 侧限流时置 false 回退 - ChatPipeline/RagPipeline:FAQ 高置信已「干净跑完」完整三级匹配仍未命中时,进入 RAG 检索跳过 重复的 FAQ 匹配(FaqMatchOutcome 携带 completedCleanly 标记),避免同一请求两次 FAQ 语义 embedding; 异常降级的 miss 不跳过,防止误跳 - FaqMatchEngine:语义匹配前加 faq_embedding 门控,作用域内无向量候选时直接返回未命中, 省掉一次注定失败的语义 embedding HTTP |
2 months ago |
|
|
a7d0ad87c9 |
feat(doc): 文档列表目录树默认收起并移除标签筛选
- 目录树去除 expand-all,初始仅显示分类、子目录折叠,点击展开 - 移除工具栏「全部标签」下拉及其数据源链路:filterTag/tagOptions、 api 的 listDocuments tag 参数与 getTagList、store 的 tags/loadTags、 路由守卫预加载 - 后端删除 /document/list 的 tag 参数过滤与 /tag/list 聚合接口 (getTagList 实现一并移除) - 文档自带 tags 属性、行内/详情展示、向量 metadata 均保留不变 |
2 months ago |
|
|
6c35aa0ff6 |
style(frontend): 上传页让「选择文件夹上传」与「选择文件」按钮同行显示
将独立成行的文件夹上传按钮并入 t-upload file-flow 的 trigger 插槽, 与「选择文件」按钮并排一行,消除上传区顶部孤零按钮的松散观感。 - 文件夹上传:@click.stop.prevent 触发独立 webkitdirectory input - 选择文件:调用官方实例方法 uploadRef.triggerUpload() 打开多文件选择器 - 两个入口各自触发、互不误触;拖拽卡与底部上传按钮逻辑保持不变 |
2 months ago |
|
|
06edac2738 |
feat(frontend): 文档列表位置列显示目录路径并修复一屏滚动布局
- 文档列表分类列改为「位置」列:沿 folderMap 回溯 parentId 拼出目录路径(不含分类名),直接挂分类根显示 —,长路径省略号截断+悬停 title 看全 - 修复文档卡片一屏高度布局:补 .t-loading__parent 参与 flex 高度链,目录树/表格在面板内滚动不再撑高整页 |
2 months ago |
|
|
06fbbaa41d |
chore(frontend): 更新 SDK 测试面板构建产物引用
|
2 months ago |
|
|
d0cd06f9fc |
fix(document): 修复批量上传向量化只成功小文档问题并加固处理链路
- 移除逐 chunk 串行 AI 关键词提取(MyKeywordEnricher):产出 excerpt_keywords 全库无检索消费点,却将外部调用放大 chunkCount 倍且与在线对话共用模型抢限流 - 向量化改按批入库(默认 50 块/批, knowledge.vector.batch-size)逐批 try-catch 隔离:失败批只记录缺失区间继续,已入库块保留,error_message 聚合"已入库 x/y 块+缺失区间+原因",chunk_count 改为实际入库块数 - 新增文档级失败自动整体重试(≤2 次, 仅 timeout/429/5xx 等瞬时错误):重试前清残留向量防重复;处理期间删除文档即终止写入避免孤儿向量;外层兜底保证状态不悬挂 PROCESSING - Embedding 客户端(DashScope/OpenAI 兼容/豆包多模态)补显式 connect 10s/read 60s 超时,避免慢响应无限挂起占死 documentExecutor 线程 - 文档同步 CLAUDE.md/README.md,新增 knowledge.vector.batch-size 配置说明 |
2 months ago |
|
|
459cabb1bc |
fix(mcp-server): 修复启用开关触发 toggle/refresh 请求死循环
- 前端 toggleMcpServer 补发 { active } 请求体,与后端 @RequestBody 契约一致(此前 active 拼在 query、无 body,每次点开关必 400)
- 后端 toggleActive 兼容 query 兜底,避免旧调用方仍 400
- axios 拦截器加重试上限(_retry),刷新改用裸 axios 发起,切断 401→refresh→重试 自激递归
- MCP 开关失败回滚 _active 状态并加 _toggling 防连点
|
2 months ago |
|
|
9896dec82a |
feat(frontend): DocList 目录树支持拖拽宽度与更多操作菜单,接入 BaseTable
目录树宽度可拖拽调节(分隔条)并将节点操作收敛为 t-dropdown 更多菜单(新建子目录/重命名/删除);文档表格接入 BaseTable,改为固定表头 + 内容超高内部滚动(基于视口 calc 高度)。因与目录树样式在同一段 CSS 中交织,同文件一并提交 |
2 months ago |
|
|
97f9befc6d |
refactor(frontend): 统一列表表格为 BaseTable 组件
新增 BaseTable 封装 t-table,统一下发小尺寸/悬浮/占满宽度/固定表头,操作列自动右固定,替换 13 个列表页 t-table,补齐缺失 loading;同步 UI 开发准则与组件声明 |
2 months ago |
|
|
1ea991e5b8 |
fix(frontend): 全局重置 body margin 消除页面多余滚动
浏览器默认 body margin:8px 未清零,导致 .layout-container(100vh) 与登录页 min-height:100vh 超出视口出现滚动条及四周白边。在 shared.css 顶部新增全局基础重置(margin/padding 清零 + box-sizing:border-box)。 |
2 months ago |
|
|
2668b8e73b |
feat(document): 文档列表支持移动到目录(统一分类根/目录)
- 后端 moveDocuments 统一移动:目标为目录时校验存在并强制采用目录所属分类,分类变更时同步 vector_store 的 categoryId metadata - DocumentController 批量移动接口新增 folderId 参数(extractLong 兼容雪花 ID 字符串) - 前端 DocList 新增单行/批量「移动」弹窗,复用目录树 t-tree 选目标,替换原「移动分类」 |
2 months ago |
|
|
f06422d5e9 |
docs: 补充开发规范与踩坑记录
- 列表排序字段白名单、vector_store 物理删除、数据库初始化幂等自愈、Java 文本块拼 SQL 等规则 |
2 months ago |
|
|
1ffd960377 |
feat(document): 目录树独立滚动与文件夹上传完善
- 文档列表目录树独立滚动 - 上传页目录选择器改为先选分类再选目录 - 文件夹上传标题剥离目录前缀(原样文件名) - 部分文件失败时弹窗按原因分组展示失败明细 |
2 months ago |
|
|
562370ece5 |
fix(backend): 数据库初始化幂等自愈与 vector_store 物理删除
- DatabaseInitConfig 改为全幂等自愈(ADD COLUMN IF NOT EXISTS + 回填 + 索引/触发器幂等) - HybridSearchService 移除对 vector_store 的 is_delete 过滤(该表无此列,物理删除) |
2 months ago |
|
|
dc8aa7a9d1 |
refactor(backend): 列表排序字段白名单统一走 SortUtils 防 SQL 注入
- 新增 SortUtils.resolveColumn 统一排序字段白名单映射,未知字段回退默认列 - 各 Controller/Service 列表接口新增 sortField/sortOrder 参数并接入白名单 - 前端列表页列加 sorter 并透传服务端排序参数 |
2 months ago |
|
|
367ec79401 |
fix(document): 修复文件夹上传连接池耗尽与目录树独立滚动
- 移除 DocumentProcessingService 异步方法上的跨方法事务,AI 关键词提取/向量化期间不再占用数据库连接,避免文件夹批量上传时连接池被占满超时 - 新增有界 documentExecutor 线程池,替代默认无界 SimpleAsyncTaskExecutor,防线程爆炸与大模型限流 - 文档列表页目录树与列表改为左右独立滚动 |
2 months ago |
|
|
780917561b |
feat(document): 新增知识库文档目录(文件夹)功能
- 新增 KnowledgeFolder 实体/Mapper、FolderService、FolderController,支持目录树增删改查 - 文档新增 folderId 字段,列表支持按目录筛选与服务端排序 - 新增文件夹批量上传(按相对路径自动创建子目录) - 文档列表页新增左侧目录树面板,上传页新增目录选择与文件夹上传 |
2 months ago |
|
|
8256e9ee85 |
fix(upload): 文档上传去重改为仅同分类内查重
去重范围从全局收窄为按分类:checkContentDuplicate 增加 category_id 条件,不同分类允许内容重复,未分类按 0 归并;修正误导文案并改用 selectList 规避多结果异常 |
2 months ago |
|
|
9d5fe80fb3 |
fix(upload): 修复文档上传多文件失效与 401 跳登录
- 上传改走 axios 复用 401 自动刷新,显式 multipart/form-data 修复 MultipartException - DocUpload 文件 Tab 加 multiple,上传 handler 按 TDesign requestMethod 契约重写 - AuthController 的 refresh cookie Secure 属性按请求 scheme 动态判断,HTTP 自动降级 - application.yml / application-dev.yml 统一 refresh-cookie-secure 配置 |
2 months ago |
|
|
1986500321 |
style(frontend): API Key 角色绑定弹窗简化角色名展示
角色绑定弹窗 checkbox 仅展示角色 name,移除冗余的 roleKey 显示 |
2 months ago |
|
|
cbbb588c69 |
chore: 清理角色表误加的 sort_order 孤儿列
customer_service_role 无排序权重需求,按 id 排序即可;sort_order 列系误加且无代码引用,启动时 DROP COLUMN IF EXISTS 幂等清理 |
2 months ago |
|
|
1e768a7409 |
fix(service): 修复角色创建主键获取异常
创建角色改用 RETURNING id 显式取回主键,避免 PostgreSQL JDBC 对 RETURN_GENERATED_KEYS 追加 RETURNING * 导致 GeneratedKeyHolder.getKey() 抛 multiple keys 异常 |
2 months ago |
|
|
a2dff86923 |
feat(rag): FAQ 匹配支持分类隔离并在闲聊意图前精准匹配
FaqMatchEngine 三级匹配(精确/关键词/语义)新增 categoryIds 分类过滤,授权分类与未分类 FAQ 可见;ChatPipeline 闲聊(CHITCHAT)意图前先尝试 FAQ 精准匹配,命中则短路返回标准答案;同步更新 PipelineFlow 架构图 |
2 months ago |
|
|
3038cd0387 |
fix(security): CORS 拆分为 SDK 开放与管理白名单双轨
修复
|
2 months ago |
|
|
9fe3ee6828 |
fix(sdk): 修复历史会话不显示图片附件
- MessageConverter 落库时持久化多模态 Media 到 metadata._media - ConversationService 历史消息接口提取 _media 为顶层 images 字段 - SDK 前端历史映射补齐 images,复用 renderUserBubble 附件渲染 |
2 months ago |
|
|
081758128c |
fix(app): 修复切换会话后用户提问被查询重写结果覆盖
RAG 查询重写(REWRITE 等策略)的润色产物被当作用户消息落库到 chat_message 会话记忆表,导致 ChatSDK 切换角色/对话后前端拉取 历史显示的是润色版而非原始提问。 ChatPipeline 在 RAG 分支将 finalMessage 由 rag.rewrittenQuery() 改为 ctx.message(),使查询重写仅服务于检索,用户消息(传给模型、 落库、前端展示)统一为原始提问。 |
2 months ago |
|
|
377aad48db |
fix(app): 修复切换推理模型后对话流式超时
chatStreamOpenAi 流开始时立即发出首片 chunk(role=assistant),确保 SSE 响应头及时 flush,避免推理模型思考阶段无输出导致前端等待首字节触发 60s 超时;AI_READ_TIMEOUT 由 120s 放宽到 300s,给推理模型思考+生成留足总时长 |
2 months ago |
|
|
0aa11a5ec6 |
fix(app): 修复推荐问题显示哨兵标记异常
SuggestionResponseParser 剥离旧版 ___SUGGESTIONS___ 哨兵标记,避免按行降级解析成垃圾条目;同步更新 suggestion_prompt 种子默认值,移除与新架构"只输出 JSON 数组"要求矛盾的哨兵文案 |
2 months ago |
|
|
d5d392aca4 |
feat(sdk): 支持图片上传与多模态对话
SDK 前端新增图片附件上传与多模态对话能力,输入区新增附件按钮与图片预览,新增 allowImageUpload 配置项;/attachment/upload 鉴权改由 SdkAuthFilter 承担(移除方法级 @PreAuthorize),JwtAuthFilter/SecurityConfig 同步放行 |
2 months ago |
|
|
5856237746 |
fix(frontend): 移除分类管理列表中的文档数量显示
|
2 months ago |
|
|
9338fcbd5f |
feat(security): JWT token_version 服务端撤销与登录安全加固
- 新增 token_version 服务端撤销机制:改密/禁用/角色变更/登出时自增,使已签发 token 立即失效 - refresh token 从 localStorage 迁移到 httpOnly Cookie,access token 有效期缩短至 15 分钟 - 新增登录失败锁定(LoginAttemptService)与密码强度校验(PasswordValidator) - SDK JWT 密钥支持环境变量注入,使用默认密钥时告警 修复: - assignRoles 角色未变化仍自增 token_version,导致用户新建用户时 token 被误失效而跳转登录 - 角色 ID 解析兼容前端字符串传输,避免雪花 ID 精度丢失导致 ClassCastException - updateFill 改用 setFieldValByName,修复 update_time 未自动填充的问题 |
2 months ago |
135 changed files with 5283 additions and 1907 deletions
-
116CLAUDE.md
-
2DEPLOY.md
-
40README.md
-
70client/src/api.ts
-
132client/src/chat.ts
-
1client/src/config.ts
-
28client/src/dom.ts
-
2client/src/i18n.ts
-
1client/src/index.ts
-
20client/src/types.ts
-
10frontend/UI-DEV-GUIDE.md
-
5frontend/components.d.ts
-
39frontend/src/App.vue
-
7frontend/src/api/api-key.ts
-
16frontend/src/api/auth.ts
-
4frontend/src/api/conversation.ts
-
19frontend/src/api/document.ts
-
4frontend/src/api/faq.ts
-
17frontend/src/api/folder.ts
-
4frontend/src/api/llm-trace.ts
-
9frontend/src/api/mcp-server.ts
-
4frontend/src/api/model-config.ts
-
76frontend/src/api/request.ts
-
8frontend/src/api/sensitive-word.ts
-
39frontend/src/api/upload.ts
-
4frontend/src/api/user.ts
-
7frontend/src/api/webhook.ts
-
40frontend/src/components/BaseTable.vue
-
4frontend/src/layouts/MainLayout.vue
-
91frontend/src/layouts/Topbar.vue
-
1frontend/src/router/index.ts
-
18frontend/src/stores/auth.ts
-
14frontend/src/stores/document.ts
-
7frontend/src/styles/shared.css
-
21frontend/src/types/models.ts
-
4frontend/src/utils/sse.ts
-
21frontend/src/utils/token.ts
-
11frontend/src/views/AccountManager.vue
-
14frontend/src/views/ApiKeyManager.vue
-
12frontend/src/views/AuditLogManager.vue
-
25frontend/src/views/CategoryManager.vue
-
16frontend/src/views/ConversationManager.vue
-
6frontend/src/views/DashboardPanel.vue
-
11frontend/src/views/DocDetail.vue
-
588frontend/src/views/DocList.vue
-
262frontend/src/views/DocUpload.vue
-
18frontend/src/views/FaqManager.vue
-
18frontend/src/views/FeedbackOps.vue
-
8frontend/src/views/LoginPage.vue
-
44frontend/src/views/McpServerManager.vue
-
70frontend/src/views/ModelConfigManager.vue
-
42frontend/src/views/PipelineFlow.vue
-
52frontend/src/views/PromptTracePanel.vue
-
20frontend/src/views/SensitiveWordManager.vue
-
19frontend/src/views/UserManager.vue
-
12frontend/src/views/WebhookManager.vue
-
65pom.xml
-
5src/main/java/com/wok/supportbot/SupportBotApplication.java
-
43src/main/java/com/wok/supportbot/advisor/ReReadingAdvisor.java
-
13src/main/java/com/wok/supportbot/app/AssistantApp.java
-
32src/main/java/com/wok/supportbot/app/ChatPipeline.java
-
2src/main/java/com/wok/supportbot/app/ChatRequest.java
-
5src/main/java/com/wok/supportbot/app/ChatResult.java
-
31src/main/java/com/wok/supportbot/app/SuggestionGenerator.java
-
285src/main/java/com/wok/supportbot/app/SuggestionResponseParser.java
-
219src/main/java/com/wok/supportbot/auth/AuthController.java
-
85src/main/java/com/wok/supportbot/chatmemory/FileBasedChatMemory.java
-
40src/main/java/com/wok/supportbot/common/SortUtils.java
-
36src/main/java/com/wok/supportbot/config/AsyncExecutorConfig.java
-
8src/main/java/com/wok/supportbot/config/ChatModelFactory.java
-
31src/main/java/com/wok/supportbot/config/CorsConfig.java
-
158src/main/java/com/wok/supportbot/config/DatabaseInitConfig.java
-
30src/main/java/com/wok/supportbot/config/EmbeddingModelFactory.java
-
16src/main/java/com/wok/supportbot/config/VolcengineMultimodalEmbeddingModel.java
-
6src/main/java/com/wok/supportbot/controller/AiModelConfigController.java
-
6src/main/java/com/wok/supportbot/controller/ApiKeyController.java
-
6src/main/java/com/wok/supportbot/controller/AttachmentController.java
-
12src/main/java/com/wok/supportbot/controller/AuthController.java
-
6src/main/java/com/wok/supportbot/controller/ConversationController.java
-
143src/main/java/com/wok/supportbot/controller/DocumentController.java
-
6src/main/java/com/wok/supportbot/controller/FaqController.java
-
136src/main/java/com/wok/supportbot/controller/FolderController.java
-
6src/main/java/com/wok/supportbot/controller/LlmCallTraceController.java
-
33src/main/java/com/wok/supportbot/controller/McpServerConfigController.java
-
6src/main/java/com/wok/supportbot/controller/MessageFeedbackController.java
-
22src/main/java/com/wok/supportbot/controller/SensitiveWordController.java
-
38src/main/java/com/wok/supportbot/controller/SysUserController.java
-
6src/main/java/com/wok/supportbot/controller/WebhookController.java
-
83src/main/java/com/wok/supportbot/converter/MessageConverter.java
-
12src/main/java/com/wok/supportbot/dao/KnowledgeFolderMapper.java
-
73src/main/java/com/wok/supportbot/document/extract/TikaDocumentReader.java
-
30src/main/java/com/wok/supportbot/document/transform/MyKeywordEnricher.java
-
67src/main/java/com/wok/supportbot/document/transform/MyTokenTextSplitter.java
-
252src/main/java/com/wok/supportbot/document/transform/OverlapTokenTextSplitter.java
-
63src/main/java/com/wok/supportbot/entity/FolderNode.java
-
15src/main/java/com/wok/supportbot/entity/KnowledgeDocument.java
-
80src/main/java/com/wok/supportbot/entity/KnowledgeFolder.java
-
4src/main/java/com/wok/supportbot/entity/SysUser.java
-
5src/main/java/com/wok/supportbot/handler/MyMetaObjectHandler.java
-
36src/main/java/com/wok/supportbot/mcp/McpToolCallback.java
@ -0,0 +1,17 @@ |
|||
import request from './request' |
|||
import type { ApiResponse } from '@/types/api' |
|||
|
|||
/** 获取目录树(可按分类过滤) */ |
|||
export function getFolderTree(categoryId?: string): Promise<ApiResponse> { return request.get(`/folder/tree${categoryId ? `?categoryId=${categoryId}` : ''}`).then(r => r.data) } |
|||
|
|||
/** 获取目录列表(可按分类过滤) */ |
|||
export function getFolderList(categoryId?: string): Promise<ApiResponse> { return request.get(`/folder/list${categoryId ? `?categoryId=${categoryId}` : ''}`).then(r => r.data) } |
|||
|
|||
/** 创建目录 */ |
|||
export function createFolder(data: any): Promise<ApiResponse> { return request.post('/folder', data).then(r => r.data) } |
|||
|
|||
/** 更新目录 */ |
|||
export function updateFolder(id: string, data: any): Promise<ApiResponse> { return request.put(`/folder/${id}`, data).then(r => r.data) } |
|||
|
|||
/** 删除目录 */ |
|||
export function deleteFolder(id: string): Promise<ApiResponse> { return request.delete(`/folder/${id}`).then(r => r.data) } |
|||
@ -1,12 +1,17 @@ |
|||
import request from './request' |
|||
import type { ApiResponse } from '@/types/api' |
|||
|
|||
export function listMcpServers(page = 1, size = 10): Promise<ApiResponse> { return request.get(`/mcp-server/list?page=${page}&size=${size}`).then(r => r.data) } |
|||
export function listMcpServers(page = 1, size = 10, sortField?: string, sortOrder?: string): Promise<ApiResponse> { |
|||
let path = `/mcp-server/list?page=${page}&size=${size}` |
|||
if (sortField) path += `&sortField=${sortField}` |
|||
if (sortOrder) path += `&sortOrder=${sortOrder}` |
|||
return request.get(path).then(r => r.data) |
|||
} |
|||
export function getMcpServerDetail(id: string): Promise<ApiResponse> { return request.get(`/mcp-server/${id}`).then(r => r.data) } |
|||
export function createMcpServer(data: any): Promise<ApiResponse> { return request.post('/mcp-server', data).then(r => r.data) } |
|||
export function updateMcpServer(id: string, data: any): Promise<ApiResponse> { return request.put(`/mcp-server/${id}`, data).then(r => r.data) } |
|||
export function deleteMcpServer(id: string): Promise<ApiResponse> { return request.delete(`/mcp-server/${id}`).then(r => r.data) } |
|||
export function toggleMcpServer(id: string, active: boolean): Promise<ApiResponse> { return request.put(`/mcp-server/${id}/toggle?active=${active}`).then(r => r.data) } |
|||
export function toggleMcpServer(id: string, active: boolean): Promise<ApiResponse> { return request.put(`/mcp-server/${id}/toggle`, { active }).then(r => r.data) } |
|||
export function testMcpServer(id: string): Promise<ApiResponse> { return request.post(`/mcp-server/${id}/test`, {}).then(r => r.data) } |
|||
export function refreshMcpServers(): Promise<ApiResponse> { return request.post('/mcp-server/refresh', {}).then(r => r.data) } |
|||
export function listAvailableMcpTools(): Promise<ApiResponse> { return request.get('/mcp-server/tools').then(r => r.data) } |
|||
@ -1,7 +1,12 @@ |
|||
import request from './request' |
|||
import type { ApiResponse } from '@/types/api' |
|||
|
|||
export function listWebhooks(page = 1, size = 20): Promise<ApiResponse> { return request.get(`/webhook/list?page=${page}&size=${size}`).then(r => r.data) } |
|||
export function listWebhooks(page = 1, size = 20, sortField?: string, sortOrder?: string): Promise<ApiResponse> { |
|||
let path = `/webhook/list?page=${page}&size=${size}` |
|||
if (sortField) path += `&sortField=${sortField}` |
|||
if (sortOrder) path += `&sortOrder=${sortOrder}` |
|||
return request.get(path).then(r => r.data) |
|||
} |
|||
export function createWebhook(data: any): Promise<ApiResponse> { return request.post('/webhook', data).then(r => r.data) } |
|||
export function updateWebhook(id: string, data: any): Promise<ApiResponse> { return request.put(`/webhook/${id}`, data).then(r => r.data) } |
|||
export function deleteWebhook(id: string): Promise<ApiResponse> { return request.delete(`/webhook/${id}`).then(r => r.data) } |
|||
|
|||
@ -0,0 +1,40 @@ |
|||
<script setup lang="ts"> |
|||
import { computed } from 'vue' |
|||
|
|||
defineOptions({ inheritAttrs: false }) |
|||
|
|||
const props = withDefaults( |
|||
defineProps<{ |
|||
columns?: any[] |
|||
/** 统一最大高度,传 '' 或 0 关闭固定表头 */ |
|||
maxHeight?: string | number |
|||
}>(), |
|||
{ |
|||
columns: () => [], |
|||
maxHeight: 'calc(100vh - 240px)', |
|||
}, |
|||
) |
|||
|
|||
// 操作列自动右固定:colKey === 'op' 且未显式 fixed 时补 fixed:'right' |
|||
const resolvedColumns = computed(() => |
|||
props.columns.map((col) => |
|||
col && col.colKey === 'op' && !col.fixed ? { ...col, fixed: 'right' } : col, |
|||
), |
|||
) |
|||
</script> |
|||
|
|||
<template> |
|||
<t-table |
|||
v-bind="$attrs" |
|||
size="small" |
|||
hover |
|||
table-layout="fixed" |
|||
:max-height="maxHeight" |
|||
:columns="resolvedColumns" |
|||
> |
|||
<!-- 动态透传所有具名插槽(op / title / fileType / ... 及作用域插槽) --> |
|||
<template v-for="(_, name) in $slots" #[name]="slotProps"> |
|||
<slot :name="name" v-bind="slotProps || {}" /> |
|||
</template> |
|||
</t-table> |
|||
</template> |
|||
@ -1,43 +0,0 @@ |
|||
package com.wok.supportbot.advisor; |
|||
|
|||
import org.springframework.ai.chat.client.ChatClientRequest; |
|||
import org.springframework.ai.chat.client.ChatClientResponse; |
|||
import org.springframework.ai.chat.client.advisor.api.AdvisorChain; |
|||
import org.springframework.ai.chat.client.advisor.api.BaseAdvisor; |
|||
|
|||
import java.util.HashMap; |
|||
import java.util.Map; |
|||
|
|||
/** |
|||
* 自定义 Re2 Advisor(适配 Spring AI 1.0.1 新 Advisor API) |
|||
* 可提高大型语言模型的推理能力 |
|||
*/ |
|||
public class ReReadingAdvisor implements BaseAdvisor { |
|||
|
|||
@Override |
|||
public String getName() { |
|||
return this.getClass().getSimpleName(); |
|||
} |
|||
|
|||
@Override |
|||
public int getOrder() { |
|||
return 0; |
|||
} |
|||
|
|||
@Override |
|||
public ChatClientRequest before(ChatClientRequest request, AdvisorChain chain) { |
|||
// Re2 策略:将用户问题重复一次以增强推理 |
|||
// 通过 context 传递原始查询,在 prompt 中追加重复指令 |
|||
Map<String, Object> newContext = new HashMap<>(request.context()); |
|||
newContext.put("re2_enabled", true); |
|||
return ChatClientRequest.builder() |
|||
.prompt(request.prompt()) |
|||
.context(newContext) |
|||
.build(); |
|||
} |
|||
|
|||
@Override |
|||
public ChatClientResponse after(ChatClientResponse response, AdvisorChain chain) { |
|||
return response; |
|||
} |
|||
} |
|||
@ -1,285 +0,0 @@ |
|||
package com.wok.supportbot.app; |
|||
|
|||
import com.fasterxml.jackson.core.type.TypeReference; |
|||
import com.fasterxml.jackson.databind.ObjectMapper; |
|||
import com.wok.supportbot.cache.SuggestionCache; |
|||
import lombok.extern.slf4j.Slf4j; |
|||
import reactor.core.publisher.Flux; |
|||
|
|||
import java.util.ArrayList; |
|||
import java.util.Collections; |
|||
import java.util.List; |
|||
import java.util.stream.Collectors; |
|||
|
|||
/** |
|||
* 建议问题解析器:从 LLM 原始输出中分离 answer 和 suggestions。 |
|||
* <p> |
|||
* 哨兵标记为 <code>___SUGGESTIONS___</code>,之后为 JSON 字符串数组。 |
|||
* 支持同步路径(直接分割完整文本)和流式路径(滑动窗口检测哨兵)。 |
|||
*/ |
|||
@Slf4j |
|||
public final class SuggestionResponseParser { |
|||
|
|||
/** 主哨兵:严格匹配提示词要求的下划线分隔标记 */ |
|||
private static final String SENTINEL = "___SUGGESTIONS___"; |
|||
|
|||
/** 兼容哨兵:模型偶尔不按指令输出时的大写变形 */ |
|||
private static final String LOOSE_SENTINEL = "SUGGESTIONS"; |
|||
|
|||
private static final ObjectMapper objectMapper = new ObjectMapper(); |
|||
|
|||
private SuggestionResponseParser() { |
|||
} |
|||
|
|||
/** |
|||
* 查找最佳哨兵位置:优先严格哨兵,其次兼容哨兵。 |
|||
* <p> |
|||
* 兼容哨兵必须后面紧跟 JSON 数组特征('[' 或空白 + '['), |
|||
* 避免正文中出现 "SUGGESTIONS" 普通单词时误触发。 |
|||
* |
|||
* @param text 要搜索的文本 |
|||
* @return 哨兵起始位置,未找到返回 -1 |
|||
*/ |
|||
private static int findSentinelIndex(String text) { |
|||
int strictIdx = text.lastIndexOf(SENTINEL); |
|||
if (strictIdx >= 0) { |
|||
return strictIdx; |
|||
} |
|||
|
|||
// 兼容模式:从后往前找大写 SUGGESTIONS,且后续需连接 JSON 数组 |
|||
int idx = text.lastIndexOf(LOOSE_SENTINEL); |
|||
while (idx >= 0) { |
|||
int after = idx + LOOSE_SENTINEL.length(); |
|||
if (after < text.length()) { |
|||
char c = text.charAt(after); |
|||
// 允许 SUGGESTIONS[...]、SUGGESTIONS [...]、SUGGESTIONS:\n[...] |
|||
if (c == '[' || Character.isWhitespace(c) || c == ':' || c == '-') { |
|||
return idx; |
|||
} |
|||
} |
|||
// 继续向前查找更早的兼容哨兵 |
|||
idx = text.lastIndexOf(LOOSE_SENTINEL, idx - 1); |
|||
} |
|||
return -1; |
|||
} |
|||
|
|||
/** |
|||
* 同步路径:按最后一次出现哨兵的位置分割完整文本。 |
|||
* |
|||
* @param rawText LLM 原始输出 |
|||
* @return 解析结果(answer + suggestions) |
|||
*/ |
|||
public static ParsedResponse parse(String rawText) { |
|||
if (rawText == null || rawText.isEmpty()) { |
|||
return new ParsedResponse(rawText != null ? rawText : "", Collections.emptyList()); |
|||
} |
|||
|
|||
int lastIdx = findSentinelIndex(rawText); |
|||
if (lastIdx < 0) { |
|||
// 无哨兵标记,整段作为 answer |
|||
return new ParsedResponse(rawText, Collections.emptyList()); |
|||
} |
|||
|
|||
String answer = rawText.substring(0, lastIdx).trim(); |
|||
// 按实际匹配的哨兵长度截取后续内容(严格哨兵或兼容哨兵长度不同) |
|||
int matchedSentinelLength = rawText.startsWith(SENTINEL, lastIdx) ? SENTINEL.length() : LOOSE_SENTINEL.length(); |
|||
String suggestionsPart = rawText.substring(lastIdx + matchedSentinelLength).trim(); |
|||
|
|||
List<String> suggestions = parseSuggestionsJson(suggestionsPart); |
|||
return new ParsedResponse(answer, suggestions); |
|||
} |
|||
|
|||
/** |
|||
* 流式路径:从原始 Flux 中分离 answer 和 suggestions。 |
|||
* <p> |
|||
* 采用「安全缓冲区」策略防止哨兵泄漏到 UI: |
|||
* <ol> |
|||
* <li>每次 chunk 追加到累积缓冲区</li> |
|||
* <li>安全区域 = 缓冲区去掉末尾 SENTINEL.length() 字符(预留哨兵跨 chunk 截断空间)</li> |
|||
* <li>在安全区域内检查哨兵:找到则发出哨兵前的剩余内容,后续全进 suggestionsBuffer</li> |
|||
* <li>未找到哨兵则发出安全区域内容</li> |
|||
* <li>doOnComplete 时:哨兵已发现则解析 suggestions;未发现则发出安全缓冲区内剩余内容</li> |
|||
* </ol> |
|||
* |
|||
* @param rawStream LLM 原始输出流 |
|||
* @param cache 建议缓存 |
|||
* @param chatId 会话 ID |
|||
* @return 纯 answer 的 Flux(不含哨兵及之后的 suggestions) |
|||
*/ |
|||
/** |
|||
* 可变状态容器(供 lambda 内部修改)。 |
|||
*/ |
|||
private static class StreamState { |
|||
final StringBuilder buf = new StringBuilder(); |
|||
int emitted = 0; |
|||
boolean sentinelFound = false; |
|||
} |
|||
|
|||
public static Flux<String> parseFromStream(Flux<String> rawStream, SuggestionCache cache, String chatId) { |
|||
StreamState state = new StreamState(); |
|||
|
|||
return rawStream |
|||
.concatMap(chunk -> Flux.<String>create(sink -> { |
|||
if (state.sentinelFound) { |
|||
sink.complete(); |
|||
return; |
|||
} |
|||
state.buf.append(chunk); |
|||
|
|||
int maxSentinelLen = Math.max(SENTINEL.length(), LOOSE_SENTINEL.length()); |
|||
int safeEnd = Math.max(0, state.buf.length() - maxSentinelLen); |
|||
if (safeEnd <= state.emitted) { |
|||
sink.complete(); |
|||
return; |
|||
} |
|||
|
|||
String safeRegion = state.buf.substring(state.emitted, safeEnd); |
|||
int sentinelIdx = findSentinelIndex(safeRegion); |
|||
|
|||
if (sentinelIdx >= 0) { |
|||
state.sentinelFound = true; |
|||
String beforeSentinel = safeRegion.substring(0, sentinelIdx); |
|||
if (!beforeSentinel.isEmpty()) { |
|||
sink.next(beforeSentinel); |
|||
} |
|||
// 确定实际匹配到的哨兵长度 |
|||
int matchedLen = safeRegion.startsWith(SENTINEL, sentinelIdx) |
|||
? SENTINEL.length() |
|||
: LOOSE_SENTINEL.length(); |
|||
// 哨兵在安全区域内的位置 sentinelIdx,相对全缓冲区即 emitted + sentinelIdx |
|||
// emitted 推进到哨兵结束后,后续内容进 suggestions 解析 |
|||
state.emitted = state.emitted + sentinelIdx + matchedLen; |
|||
log.debug("流式路径检测到哨兵: chatId={}, pos={}, matchedLen={}", chatId, state.emitted, matchedLen); |
|||
} else { |
|||
if (!safeRegion.isEmpty()) { |
|||
sink.next(safeRegion); |
|||
} |
|||
state.emitted = safeEnd; |
|||
} |
|||
sink.complete(); |
|||
})) |
|||
// 流结束后发出安全缓冲区内未发出的残留内容(无哨兵场景) |
|||
.concatWith(Flux.defer(() -> { |
|||
if (!state.sentinelFound && state.emitted < state.buf.length()) { |
|||
String residual = state.buf.substring(state.emitted); |
|||
if (!residual.isEmpty()) { |
|||
return Flux.just(residual); |
|||
} |
|||
} |
|||
return Flux.empty(); |
|||
})) |
|||
.doOnComplete(() -> { |
|||
if (state.sentinelFound) { |
|||
String suggestionsPart = state.buf.length() > state.emitted |
|||
? state.buf.substring(state.emitted) : ""; |
|||
List<String> suggestions = parseSuggestionsJson(suggestionsPart.strip()); |
|||
if (!suggestions.isEmpty()) { |
|||
cache.put(chatId, suggestions); |
|||
log.info("流式 suggestions 解析成功: chatId={}, count={}", chatId, suggestions.size()); |
|||
} else { |
|||
log.debug("流式 suggestions 解析为空: chatId={}", chatId); |
|||
} |
|||
} |
|||
}); |
|||
} |
|||
|
|||
/** |
|||
* 容错解析 suggestions JSON 数组。 |
|||
* 先尝试标准 JSON 解析,失败则按行切分取前 3 条非空行。 |
|||
*/ |
|||
static List<String> parseSuggestionsJson(String jsonPart) { |
|||
if (jsonPart == null || jsonPart.isBlank()) { |
|||
return Collections.emptyList(); |
|||
} |
|||
|
|||
// 1. 尝试标准 JSON 解析 |
|||
String trimmed = jsonPart.strip(); |
|||
// 去掉可能的 markdown 代码块包裹 |
|||
trimmed = trimCodeBlock(trimmed); |
|||
|
|||
try { |
|||
List<String> parsed = objectMapper.readValue(trimmed, new TypeReference<List<String>>() {}); |
|||
List<String> result = new ArrayList<>(); |
|||
for (String s : parsed) { |
|||
if (s != null && !s.isBlank()) { |
|||
result.add(s.strip()); |
|||
} |
|||
} |
|||
if (!result.isEmpty()) { |
|||
return result.size() <= 3 ? result : result.subList(0, 3); |
|||
} |
|||
} catch (Exception e) { |
|||
log.debug("标准 JSON 解析 suggestions 失败,尝试按行切分: {}", e.getMessage()); |
|||
} |
|||
|
|||
// 2. 降级:按行切分,取前 3 条 |
|||
return parseByLines(trimmed); |
|||
} |
|||
|
|||
/** |
|||
* 降级解析:按行切分,去掉序号前缀后取前 3 条非空行。 |
|||
*/ |
|||
private static List<String> parseByLines(String text) { |
|||
List<String> lines = text.lines() |
|||
.map(String::strip) |
|||
.map(SuggestionResponseParser::stripNumberPrefix) |
|||
.map(SuggestionResponseParser::stripQuotes) |
|||
.filter(s -> !s.isBlank()) |
|||
.collect(Collectors.toList()); |
|||
|
|||
if (lines.isEmpty()) { |
|||
return Collections.emptyList(); |
|||
} |
|||
return lines.size() <= 3 ? lines : lines.subList(0, 3); |
|||
} |
|||
|
|||
/** 去掉可能的 markdown 代码块包裹(支持 3 个及以上反引号) */ |
|||
private static String trimCodeBlock(String s) { |
|||
if (s.startsWith("```") && s.endsWith("```")) { |
|||
// 计算开头的反引号数量 |
|||
int openCount = 0; |
|||
while (openCount < s.length() && s.charAt(openCount) == '`') openCount++; |
|||
if (openCount >= 3 && s.endsWith("`".repeat(openCount))) { |
|||
String inner = s.substring(openCount, s.length() - openCount).strip(); |
|||
if (inner.startsWith("json")) { |
|||
inner = inner.substring(4).strip(); |
|||
} else if (inner.startsWith("JSON")) { |
|||
inner = inner.substring(4).strip(); |
|||
} |
|||
return inner; |
|||
} |
|||
} |
|||
return s; |
|||
} |
|||
|
|||
/** 去掉行首序号,如 "1." "2." "3." "1、" "2、" "3、" */ |
|||
private static String stripNumberPrefix(String s) { |
|||
if (s.length() >= 2 && Character.isDigit(s.charAt(0)) && |
|||
(s.charAt(1) == '.' || s.charAt(1) == '、' || s.charAt(1) == ')')) { |
|||
return s.substring(2).strip(); |
|||
} |
|||
return s; |
|||
} |
|||
|
|||
/** 去掉首尾引号 */ |
|||
private static String stripQuotes(String s) { |
|||
if (s.length() >= 2) { |
|||
char first = s.charAt(0); |
|||
char last = s.charAt(s.length() - 1); |
|||
if ((first == '"' && last == '"') || (first == '\'' && last == '\'') || |
|||
(first == '“' && last == '”')) { // 中文引号 " " |
|||
return s.substring(1, s.length() - 1).strip(); |
|||
} |
|||
} |
|||
return s; |
|||
} |
|||
|
|||
/** |
|||
* 解析结果值对象。 |
|||
* |
|||
* @param answer LLM 回答正文(不含哨兵及之后的 suggestions) |
|||
* @param suggestions 建议问题列表(0~3 条) |
|||
*/ |
|||
public record ParsedResponse(String answer, List<String> suggestions) { |
|||
} |
|||
} |
|||
@ -1,85 +0,0 @@ |
|||
package com.wok.supportbot.chatmemory; |
|||
|
|||
import com.esotericsoftware.kryo.Kryo; |
|||
import com.esotericsoftware.kryo.io.Input; |
|||
import com.esotericsoftware.kryo.io.Output; |
|||
import org.objenesis.strategy.StdInstantiatorStrategy; |
|||
import org.springframework.ai.chat.memory.ChatMemory; |
|||
import org.springframework.ai.chat.messages.Message; |
|||
|
|||
import java.io.File; |
|||
import java.io.FileInputStream; |
|||
import java.io.FileOutputStream; |
|||
import java.io.IOException; |
|||
import java.util.ArrayList; |
|||
import java.util.List; |
|||
|
|||
/** |
|||
* 基于文件持久化的对话记忆 |
|||
*/ |
|||
public class FileBasedChatMemory implements ChatMemory { |
|||
|
|||
private final String BASE_DIR; |
|||
private static final Kryo kryo = new Kryo(); |
|||
|
|||
static { |
|||
kryo.setRegistrationRequired(false); |
|||
// 设置实例化策略 |
|||
kryo.setInstantiatorStrategy(new StdInstantiatorStrategy()); |
|||
} |
|||
|
|||
// 构造对象时,指定文件保存目录 |
|||
public FileBasedChatMemory(String dir) { |
|||
this.BASE_DIR = dir; |
|||
File baseDir = new File(dir); |
|||
if (!baseDir.exists()) { |
|||
baseDir.mkdirs(); |
|||
} |
|||
} |
|||
|
|||
@Override |
|||
public void add(String conversationId, List<Message> messages) { |
|||
List<Message> conversationMessages = getOrCreateConversation(conversationId); |
|||
conversationMessages.addAll(messages); |
|||
saveConversation(conversationId, conversationMessages); |
|||
} |
|||
|
|||
@Override |
|||
public List<Message> get(String conversationId) { |
|||
return getOrCreateConversation(conversationId); |
|||
} |
|||
|
|||
@Override |
|||
public void clear(String conversationId) { |
|||
File file = getConversationFile(conversationId); |
|||
if (file.exists()) { |
|||
file.delete(); |
|||
} |
|||
} |
|||
|
|||
private List<Message> getOrCreateConversation(String conversationId) { |
|||
File file = getConversationFile(conversationId); |
|||
List<Message> messages = new ArrayList<>(); |
|||
if (file.exists()) { |
|||
try (Input input = new Input(new FileInputStream(file))) { |
|||
messages = kryo.readObject(input, ArrayList.class); |
|||
} catch (IOException e) { |
|||
e.printStackTrace(); |
|||
} |
|||
} |
|||
return messages; |
|||
} |
|||
|
|||
private void saveConversation(String conversationId, List<Message> messages) { |
|||
File file = getConversationFile(conversationId); |
|||
try (Output output = new Output(new FileOutputStream(file))) { |
|||
kryo.writeObject(output, messages); |
|||
} catch (IOException e) { |
|||
e.printStackTrace(); |
|||
} |
|||
} |
|||
|
|||
private File getConversationFile(String conversationId) { |
|||
return new File(BASE_DIR, conversationId + ".kryo"); |
|||
} |
|||
} |
|||
@ -0,0 +1,40 @@ |
|||
package com.wok.supportbot.common; |
|||
|
|||
import java.util.Map; |
|||
|
|||
/** |
|||
* 列表排序工具类。 |
|||
* <p> |
|||
* 用于统一处理后台列表接口的排序参数:白名单映射(防 SQL 注入)、sortField 判空回退、sortOrder 二值化。 |
|||
* 避免在每个 Service 里重复书写「白名单 + getOrDefault(null) 判空 + asc/desc」逻辑。 |
|||
*/ |
|||
public final class SortUtils { |
|||
|
|||
private SortUtils() { |
|||
} |
|||
|
|||
/** |
|||
* 解析排序字段对应的数据库列名。 |
|||
* <p> |
|||
* sortField 为 null 或不在白名单内时,回退到 defaultColumn,杜绝将用户输入直接拼入 ORDER BY。 |
|||
* |
|||
* @param sortField 前端传入的 colKey(可为 null) |
|||
* @param whitelist 前端 colKey → 数据库列名(可含表别名)的白名单映射 |
|||
* @param defaultColumn 回退列名(通常为 create_time) |
|||
* @return 安全列名 |
|||
*/ |
|||
public static String resolveColumn(String sortField, Map<String, String> whitelist, String defaultColumn) { |
|||
if (sortField == null) { |
|||
return defaultColumn; |
|||
} |
|||
String column = whitelist.get(sortField); |
|||
return column != null ? column : defaultColumn; |
|||
} |
|||
|
|||
/** |
|||
* 排序方向二值化:仅 asc 视为升序,其余(含 null、desc、其他值)一律降序。 |
|||
*/ |
|||
public static boolean isAscending(String sortOrder) { |
|||
return "asc".equalsIgnoreCase(sortOrder); |
|||
} |
|||
} |
|||
@ -0,0 +1,136 @@ |
|||
package com.wok.supportbot.controller; |
|||
|
|||
import com.wok.supportbot.entity.FolderNode; |
|||
import com.wok.supportbot.entity.KnowledgeFolder; |
|||
import com.wok.supportbot.service.FolderService; |
|||
import org.springframework.beans.factory.annotation.Autowired; |
|||
import org.springframework.http.ResponseEntity; |
|||
import org.springframework.security.access.prepost.PreAuthorize; |
|||
import org.springframework.web.bind.annotation.*; |
|||
|
|||
import java.util.List; |
|||
import java.util.Map; |
|||
|
|||
/** |
|||
* 知识库文档目录控制器 |
|||
* 提供目录的树形查询、扁平列表、创建、重命名、删除等功能 |
|||
*/ |
|||
@RestController |
|||
public class FolderController { |
|||
|
|||
@Autowired |
|||
private FolderService folderService; |
|||
|
|||
/** |
|||
* 获取目录树 |
|||
*/ |
|||
@GetMapping("/folder/tree") |
|||
@PreAuthorize("hasAnyRole('admin','kb_operator')") |
|||
public ResponseEntity<Map<String, Object>> getFolderTree(@RequestParam(required = false) Long categoryId) { |
|||
try { |
|||
List<FolderNode> tree = folderService.getFolderTree(categoryId); |
|||
return ResponseEntity.ok(Map.of( |
|||
"success", true, |
|||
"data", tree |
|||
)); |
|||
} catch (Exception e) { |
|||
return ResponseEntity.status(500).body(Map.of( |
|||
"success", false, |
|||
"message", "获取目录树失败:" + e.getMessage() |
|||
)); |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* 获取目录扁平列表 |
|||
*/ |
|||
@GetMapping("/folder/list") |
|||
@PreAuthorize("hasAnyRole('admin','kb_operator')") |
|||
public ResponseEntity<Map<String, Object>> listFolders(@RequestParam(required = false) Long categoryId) { |
|||
try { |
|||
List<KnowledgeFolder> list = folderService.listFolders(categoryId); |
|||
return ResponseEntity.ok(Map.of( |
|||
"success", true, |
|||
"data", list |
|||
)); |
|||
} catch (Exception e) { |
|||
return ResponseEntity.status(500).body(Map.of( |
|||
"success", false, |
|||
"message", "获取目录列表失败:" + e.getMessage() |
|||
)); |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* 创建目录 |
|||
*/ |
|||
@PostMapping("/folder") |
|||
@PreAuthorize("hasAnyRole('admin','kb_operator')") |
|||
public ResponseEntity<Map<String, Object>> createFolder(@RequestBody Map<String, Object> body) { |
|||
try { |
|||
String name = (String) body.get("name"); |
|||
Long categoryId = body.get("categoryId") != null ? Long.valueOf(body.get("categoryId").toString()) : null; |
|||
Long parentId = body.get("parentId") != null ? Long.valueOf(body.get("parentId").toString()) : null; |
|||
Integer sortOrder = body.get("sortOrder") != null ? Integer.valueOf(body.get("sortOrder").toString()) : null; |
|||
|
|||
KnowledgeFolder folder = folderService.createFolder(name, categoryId, parentId, sortOrder); |
|||
return ResponseEntity.ok(Map.of( |
|||
"success", true, |
|||
"message", "目录创建成功", |
|||
"data", folder |
|||
)); |
|||
} catch (Exception e) { |
|||
return ResponseEntity.status(500).body(Map.of( |
|||
"success", false, |
|||
"message", "创建目录失败:" + e.getMessage() |
|||
)); |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* 重命名目录 / 调整排序 |
|||
*/ |
|||
@PutMapping("/folder/{id}") |
|||
@PreAuthorize("hasAnyRole('admin','kb_operator')") |
|||
public ResponseEntity<Map<String, Object>> renameFolder( |
|||
@PathVariable Long id, |
|||
@RequestBody Map<String, Object> body) { |
|||
try { |
|||
String name = (String) body.get("name"); |
|||
Integer sortOrder = body.get("sortOrder") != null ? Integer.valueOf(body.get("sortOrder").toString()) : null; |
|||
|
|||
KnowledgeFolder folder = folderService.renameFolder(id, name, sortOrder); |
|||
return ResponseEntity.ok(Map.of( |
|||
"success", true, |
|||
"message", "目录更新成功", |
|||
"data", folder |
|||
)); |
|||
} catch (Exception e) { |
|||
return ResponseEntity.status(500).body(Map.of( |
|||
"success", false, |
|||
"message", "更新目录失败:" + e.getMessage() |
|||
)); |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* 删除目录(级联删除子孙目录,并将其下文档移到分类根) |
|||
*/ |
|||
@DeleteMapping("/folder/{id}") |
|||
@PreAuthorize("hasAnyRole('admin','kb_operator')") |
|||
public ResponseEntity<Map<String, Object>> deleteFolder(@PathVariable Long id) { |
|||
try { |
|||
Map<String, Object> result = folderService.deleteFolder(id); |
|||
return ResponseEntity.ok(Map.of( |
|||
"success", true, |
|||
"message", "目录删除成功", |
|||
"data", result |
|||
)); |
|||
} catch (Exception e) { |
|||
return ResponseEntity.status(500).body(Map.of( |
|||
"success", false, |
|||
"message", "删除目录失败:" + e.getMessage() |
|||
)); |
|||
} |
|||
} |
|||
} |
|||
@ -0,0 +1,12 @@ |
|||
package com.wok.supportbot.dao; |
|||
|
|||
import com.baomidou.mybatisplus.core.mapper.BaseMapper; |
|||
import com.wok.supportbot.entity.KnowledgeFolder; |
|||
import org.apache.ibatis.annotations.Mapper; |
|||
|
|||
/** |
|||
* 知识库文档目录 Mapper |
|||
*/ |
|||
@Mapper |
|||
public interface KnowledgeFolderMapper extends BaseMapper<KnowledgeFolder> { |
|||
} |
|||
@ -1,73 +0,0 @@ |
|||
package com.wok.supportbot.document.extract; |
|||
|
|||
import lombok.extern.slf4j.Slf4j; |
|||
import org.apache.tika.Tika; |
|||
import org.apache.tika.exception.TikaException; |
|||
import org.springframework.ai.document.Document; |
|||
import org.springframework.core.io.Resource; |
|||
import org.springframework.core.io.FileSystemResource; |
|||
import org.springframework.stereotype.Component; |
|||
import org.springframework.web.multipart.MultipartFile; |
|||
|
|||
import java.io.File; |
|||
import java.io.IOException; |
|||
import java.util.Collections; |
|||
import java.util.List; |
|||
import java.util.UUID; |
|||
|
|||
@Component |
|||
@Slf4j |
|||
public class TikaDocumentReader { |
|||
|
|||
/** |
|||
* 从已保存的文件解析内容 |
|||
*/ |
|||
public List<Document> readFromFile(File file) { |
|||
try { |
|||
Tika tika = new Tika(); |
|||
String text = tika.parseToString(new java.io.FileInputStream(file)); |
|||
|
|||
Document doc = Document.builder() |
|||
.id(UUID.randomUUID().toString()) |
|||
.text(text) |
|||
.build(); |
|||
|
|||
return Collections.singletonList(doc); |
|||
|
|||
} catch (IOException | TikaException e) { |
|||
log.error("Tika 文件解析失败", e); |
|||
throw new RuntimeException("Tika 文件解析失败", e); |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* 从 MultipartFile 解析内容(兼容旧接口) |
|||
*/ |
|||
public List<Document> read(MultipartFile file) { |
|||
try { |
|||
Tika tika = new Tika(); |
|||
String text = tika.parseToString(file.getInputStream()); |
|||
|
|||
Document doc = Document.builder() |
|||
.id(UUID.randomUUID().toString()) |
|||
.text(text) |
|||
.build(); |
|||
|
|||
return Collections.singletonList(doc); |
|||
|
|||
} catch (IOException | TikaException e) { |
|||
log.error("Tika 文件解析失败", e); |
|||
throw new RuntimeException("Tika 文件解析失败", e); |
|||
} |
|||
} |
|||
|
|||
/** |
|||
* 获取文件扩展名(含点号),如 ".pptx"。无扩展名时返回 ".tmp" |
|||
*/ |
|||
private String getExtension(String filename) { |
|||
if (filename == null || !filename.contains(".")) { |
|||
return ".tmp"; |
|||
} |
|||
return filename.substring(filename.lastIndexOf(".")); |
|||
} |
|||
} |
|||
@ -1,30 +0,0 @@ |
|||
package com.wok.supportbot.document.transform; |
|||
|
|||
import com.wok.supportbot.config.ChatModelFactory; |
|||
import jakarta.annotation.Resource; |
|||
import org.springframework.ai.document.Document; |
|||
import org.springframework.ai.model.transformer.KeywordMetadataEnricher; |
|||
import org.springframework.stereotype.Component; |
|||
|
|||
import java.util.List; |
|||
|
|||
/** |
|||
* 基于 AI 的文档元信息增强器(为文档补充元信息) |
|||
* 通过 ChatModelFactory 获取 ChatModel,支持多提供商动态切换 |
|||
*/ |
|||
@Component |
|||
public class MyKeywordEnricher { |
|||
|
|||
@Resource |
|||
private ChatModelFactory chatModelFactory; |
|||
|
|||
/** |
|||
* 使用 AI 提取关键词并添加到元数据 |
|||
*/ |
|||
public List<Document> enrichDocuments(List<Document> documents) { |
|||
KeywordMetadataEnricher enricher = new KeywordMetadataEnricher.Builder(chatModelFactory.getChatModel("CHAT")) |
|||
.keywordCount(5) |
|||
.build(); |
|||
return enricher.apply(documents); |
|||
} |
|||
} |
|||
@ -1,67 +0,0 @@ |
|||
package com.wok.supportbot.document.transform; |
|||
|
|||
import com.wok.supportbot.config.ChunkConfig; |
|||
import org.springframework.ai.document.Document; |
|||
import org.springframework.ai.transformer.splitter.TokenTextSplitter; |
|||
import org.springframework.beans.factory.annotation.Autowired; |
|||
import org.springframework.stereotype.Component; |
|||
|
|||
import java.util.List; |
|||
|
|||
/** |
|||
* 自定义基于 Token 的切词器 |
|||
* 支持通过 ChunkConfig 动态调整分块参数 |
|||
*/ |
|||
@Component |
|||
public class MyTokenTextSplitter { |
|||
|
|||
@Autowired |
|||
private ChunkConfig chunkConfig; |
|||
|
|||
/** |
|||
* 使用全局配置参数创建分割器 |
|||
*/ |
|||
public List<Document> splitDocuments(List<Document> documents) { |
|||
TokenTextSplitter splitter = new TokenTextSplitter( |
|||
chunkConfig.getChunkSize(), |
|||
chunkConfig.getOverlap(), |
|||
chunkConfig.getMinChunkSizeChars(), |
|||
chunkConfig.getMaxNumChunks(), |
|||
chunkConfig.isKeepSeparator() |
|||
); |
|||
return splitter.apply(documents); |
|||
} |
|||
|
|||
/** |
|||
* 使用自定义参数创建分割器(覆盖全局配置) |
|||
* |
|||
* @param documents 文档列表 |
|||
* @param chunkSize 分块大小 |
|||
* @param overlap 重叠大小 |
|||
*/ |
|||
public List<Document> splitDocuments(List<Document> documents, Integer chunkSize, Integer overlap) { |
|||
int cs = chunkSize != null ? chunkSize : chunkConfig.getChunkSize(); |
|||
int ol = overlap != null ? overlap : chunkConfig.getOverlap(); |
|||
TokenTextSplitter splitter = new TokenTextSplitter( |
|||
cs, ol, |
|||
chunkConfig.getMinChunkSizeChars(), |
|||
chunkConfig.getMaxNumChunks(), |
|||
chunkConfig.isKeepSeparator() |
|||
); |
|||
return splitter.apply(documents); |
|||
} |
|||
|
|||
/** |
|||
* 使用自定义参数创建分割器(全参数覆盖) |
|||
*/ |
|||
public List<Document> splitCustomized(List<Document> documents) { |
|||
TokenTextSplitter splitter = new TokenTextSplitter( |
|||
chunkConfig.getChunkSize(), |
|||
chunkConfig.getOverlap(), |
|||
chunkConfig.getMinChunkSizeChars(), |
|||
chunkConfig.getMaxNumChunks(), |
|||
chunkConfig.isKeepSeparator() |
|||
); |
|||
return splitter.apply(documents); |
|||
} |
|||
} |
|||
@ -0,0 +1,252 @@ |
|||
package com.wok.supportbot.document.transform; |
|||
|
|||
import com.knuddels.jtokkit.Encodings; |
|||
import com.knuddels.jtokkit.api.Encoding; |
|||
import com.knuddels.jtokkit.api.EncodingType; |
|||
import com.knuddels.jtokkit.api.IntArrayList; |
|||
import org.slf4j.Logger; |
|||
import org.slf4j.LoggerFactory; |
|||
import org.springframework.ai.transformer.splitter.TextSplitter; |
|||
import org.springframework.util.Assert; |
|||
|
|||
import java.util.ArrayList; |
|||
import java.util.List; |
|||
|
|||
/** |
|||
* 带重叠(overlap)的 Token 分块器。 |
|||
* |
|||
* <p>Spring AI 的 {@link org.springframework.ai.transformer.splitter.TokenTextSplitter} |
|||
* 不支持 overlap —— 它的 5 参构造器与 Builder 均无该形参(社区 PR #4054 已标记不向 1.x 回迁)。 |
|||
* 本类直接继承标准基类 {@link TextSplitter},复刻 TokenTextSplitter 的全部切分语义 |
|||
* (标点截断、最小分块长度、分块数上限、分隔符处理),仅把「前进步长」由 chunkSize |
|||
* 改为 {@code chunkSize - overlap},从而让重叠真正生效。 |
|||
* |
|||
* <p>元数据继承({@code parent_document_id} / {@code chunk_index} / {@code total_chunks}) |
|||
* 由 {@link TextSplitter} 基类统一处理,本类只负责文本切分。 |
|||
* |
|||
* <p>Token 编码与 TokenTextSplitter 保持一致:jtokkit + CL100K_BASE。 |
|||
*/ |
|||
public class OverlapTokenTextSplitter extends TextSplitter { |
|||
|
|||
private static final Logger log = LoggerFactory.getLogger(OverlapTokenTextSplitter.class); |
|||
|
|||
/** 默认分块大小(token 数) */ |
|||
private static final int DEFAULT_CHUNK_SIZE = 800; |
|||
|
|||
/** 默认重叠 token 数(0 表示不重叠,等价于 TokenTextSplitter 行为) */ |
|||
private static final int DEFAULT_OVERLAP = 0; |
|||
|
|||
/** 默认最小分块字符数 */ |
|||
private static final int MIN_CHUNK_SIZE_CHARS = 350; |
|||
|
|||
/** 默认丢弃阈值:长度不超过该值的分块不入库 */ |
|||
private static final int MIN_CHUNK_LENGTH_TO_EMBED = 5; |
|||
|
|||
/** 默认单篇文档最大分块数 */ |
|||
private static final int MAX_NUM_CHUNKS = 10000; |
|||
|
|||
/** 默认保留分隔符 */ |
|||
private static final boolean KEEP_SEPARATOR = true; |
|||
|
|||
private final Encoding encoding = Encodings.newLazyEncodingRegistry().getEncoding(EncodingType.CL100K_BASE); |
|||
|
|||
/** 目标分块大小(token 数,不含重叠部分) */ |
|||
private final int chunkSize; |
|||
|
|||
/** 相邻分块的重叠 token 数 */ |
|||
private final int overlap; |
|||
|
|||
/** 最小分块字符数:仅当剩余 token 数超过 chunkSize 时才按标点截断,且截断点须大于该值 */ |
|||
private final int minChunkSizeChars; |
|||
|
|||
/** 丢弃长度不超过该值的分块 */ |
|||
private final int minChunkLengthToEmbed; |
|||
|
|||
/** 单篇文档最大分块数 */ |
|||
private final int maxNumChunks; |
|||
|
|||
/** 是否保留分隔符(false 时把换行替换为空格) */ |
|||
private final boolean keepSeparator; |
|||
|
|||
private OverlapTokenTextSplitter(int chunkSize, int overlap, int minChunkSizeChars, |
|||
int minChunkLengthToEmbed, int maxNumChunks, boolean keepSeparator) { |
|||
Assert.isTrue(chunkSize > 0, "chunkSize 必须大于 0"); |
|||
this.chunkSize = chunkSize; |
|||
// 重叠必须严格小于分块大小,否则前进步长会 <= 0 导致死循环;此处收敛到合法区间而非直接报错 |
|||
int clamped = Math.max(0, Math.min(overlap, chunkSize - 1)); |
|||
if (clamped != overlap) { |
|||
log.warn("overlap({}) 超出合法区间 [0, chunkSize-1={}],已收敛为 {};" |
|||
+ "该配置会使前进步长退化为 1 个 token,产生大量分块,请检查 knowledge.chunk.overlap", |
|||
overlap, chunkSize - 1, clamped); |
|||
} |
|||
this.overlap = clamped; |
|||
this.minChunkSizeChars = minChunkSizeChars; |
|||
this.minChunkLengthToEmbed = minChunkLengthToEmbed; |
|||
this.maxNumChunks = maxNumChunks; |
|||
this.keepSeparator = keepSeparator; |
|||
} |
|||
|
|||
public static Builder builder() { |
|||
return new Builder(); |
|||
} |
|||
|
|||
/** |
|||
* 标点截断允许的最小前进步长(token 数)。 |
|||
* <p> |
|||
* 防止「标点截断」与「overlap」叠加后把前进步长压到 1 个 token 导致分块数爆炸: |
|||
* 截断后必须仍能前进这么多 token,否则放弃本次截断(宁可切断句子也要保证分块数量可控)。 |
|||
* <p> |
|||
* overlap=0 时返回 1,即允许任意截断,与标准 {@code TokenTextSplitter} 行为完全一致。 |
|||
*/ |
|||
private int minAdvance() { |
|||
if (this.overlap == 0) { |
|||
return 1; |
|||
} |
|||
return Math.max(1, (this.chunkSize - this.overlap) / 2); |
|||
} |
|||
|
|||
@Override |
|||
protected List<String> splitText(String text) { |
|||
if (text == null || text.trim().isEmpty()) { |
|||
return new ArrayList<>(); |
|||
} |
|||
|
|||
List<Integer> tokens = getEncodedTokens(text); |
|||
List<String> chunks = new ArrayList<>(); |
|||
int numChunks = 0; |
|||
|
|||
while (!tokens.isEmpty() && numChunks < this.maxNumChunks) { |
|||
int windowSize = Math.min(this.chunkSize, tokens.size()); |
|||
boolean lastWindow = windowSize >= tokens.size(); |
|||
String chunkText = decodeTokens(tokens.subList(0, windowSize)); |
|||
|
|||
// 空白块直接跳过(不产出,也不做重叠回退) |
|||
if (chunkText.trim().isEmpty()) { |
|||
tokens = tokens.subList(windowSize, tokens.size()); |
|||
continue; |
|||
} |
|||
|
|||
// 仅当剩余 token 数超过 chunkSize 时才做标点截断,避免小文本被无谓切分 |
|||
if (tokens.size() > this.chunkSize) { |
|||
int lastPunctuation = Math.max(chunkText.lastIndexOf('.'), |
|||
Math.max(chunkText.lastIndexOf('?'), |
|||
Math.max(chunkText.lastIndexOf('!'), chunkText.lastIndexOf('\n')))); |
|||
if (lastPunctuation != -1 && lastPunctuation > this.minChunkSizeChars) { |
|||
String candidate = chunkText.substring(0, lastPunctuation + 1); |
|||
// 前进步长 = 截断后消耗的 token 数 - overlap。截断会缩短消耗量, |
|||
// 若不加约束,截断点靠前时步长会被压到 1 个 token,分块数成倍膨胀 |
|||
// (实测 chunkSize=60/overlap=30 时可达 10 倍)。此处要求截断后仍能前进至少 minAdvance()。 |
|||
if (getEncodedTokens(candidate).size() - this.overlap >= minAdvance()) { |
|||
chunkText = candidate; |
|||
} |
|||
} |
|||
} |
|||
|
|||
String chunkTextToAppend = this.keepSeparator |
|||
? chunkText.trim() |
|||
: chunkText.replace(System.lineSeparator(), " ").trim(); |
|||
if (chunkTextToAppend.length() > this.minChunkLengthToEmbed) { |
|||
chunks.add(chunkTextToAppend); |
|||
} |
|||
numChunks++; |
|||
|
|||
// 本窗口已覆盖全部剩余 token,无需再产出重叠块 |
|||
if (lastWindow) { |
|||
tokens = new ArrayList<>(); |
|||
break; |
|||
} |
|||
|
|||
// 本块实际消耗的 token 数(可能因标点截断而少于窗口大小) |
|||
int consumed = getEncodedTokens(chunkText).size(); |
|||
// 前进步长 = 消耗量 - 重叠量;至少前进 1 个 token,避免死循环 |
|||
int step = Math.max(1, consumed - this.overlap); |
|||
tokens = tokens.subList(step, tokens.size()); |
|||
} |
|||
|
|||
// 处理剩余 token(与 TokenTextSplitter 保持一致,用于达到 maxNumChunks 上限的场景) |
|||
if (!tokens.isEmpty()) { |
|||
String remainingText = decodeTokens(tokens).replace(System.lineSeparator(), " ").trim(); |
|||
if (remainingText.length() > this.minChunkLengthToEmbed) { |
|||
chunks.add(remainingText); |
|||
} |
|||
} |
|||
|
|||
return chunks; |
|||
} |
|||
|
|||
private List<Integer> getEncodedTokens(String text) { |
|||
Assert.notNull(text, "Text must not be null"); |
|||
return this.encoding.encode(text).boxed(); |
|||
} |
|||
|
|||
private String decodeTokens(List<Integer> tokens) { |
|||
Assert.notNull(tokens, "Tokens must not be null"); |
|||
IntArrayList tokensIntArray = new IntArrayList(tokens.size()); |
|||
tokens.forEach(tokensIntArray::add); |
|||
return this.encoding.decode(tokensIntArray); |
|||
} |
|||
|
|||
/** |
|||
* 分块器构建器 |
|||
*/ |
|||
public static final class Builder { |
|||
|
|||
private int chunkSize = DEFAULT_CHUNK_SIZE; |
|||
|
|||
private int overlap = DEFAULT_OVERLAP; |
|||
|
|||
private int minChunkSizeChars = MIN_CHUNK_SIZE_CHARS; |
|||
|
|||
private int minChunkLengthToEmbed = MIN_CHUNK_LENGTH_TO_EMBED; |
|||
|
|||
private int maxNumChunks = MAX_NUM_CHUNKS; |
|||
|
|||
private boolean keepSeparator = KEEP_SEPARATOR; |
|||
|
|||
private Builder() { |
|||
} |
|||
|
|||
/** 目标分块大小(token 数) */ |
|||
public Builder withChunkSize(int chunkSize) { |
|||
this.chunkSize = chunkSize; |
|||
return this; |
|||
} |
|||
|
|||
/** 相邻分块的重叠 token 数 */ |
|||
public Builder withOverlap(int overlap) { |
|||
this.overlap = overlap; |
|||
return this; |
|||
} |
|||
|
|||
/** 最小分块字符数(标点截断的下限) */ |
|||
public Builder withMinChunkSizeChars(int minChunkSizeChars) { |
|||
this.minChunkSizeChars = minChunkSizeChars; |
|||
return this; |
|||
} |
|||
|
|||
/** 丢弃长度不超过该值的分块 */ |
|||
public Builder withMinChunkLengthToEmbed(int minChunkLengthToEmbed) { |
|||
this.minChunkLengthToEmbed = minChunkLengthToEmbed; |
|||
return this; |
|||
} |
|||
|
|||
/** 单篇文档最大分块数 */ |
|||
public Builder withMaxNumChunks(int maxNumChunks) { |
|||
this.maxNumChunks = maxNumChunks; |
|||
return this; |
|||
} |
|||
|
|||
/** 是否保留分隔符 */ |
|||
public Builder withKeepSeparator(boolean keepSeparator) { |
|||
this.keepSeparator = keepSeparator; |
|||
return this; |
|||
} |
|||
|
|||
public OverlapTokenTextSplitter build() { |
|||
return new OverlapTokenTextSplitter(this.chunkSize, this.overlap, this.minChunkSizeChars, |
|||
this.minChunkLengthToEmbed, this.maxNumChunks, this.keepSeparator); |
|||
} |
|||
|
|||
} |
|||
|
|||
} |
|||
@ -0,0 +1,63 @@ |
|||
package com.wok.supportbot.entity; |
|||
|
|||
import com.fasterxml.jackson.databind.annotation.JsonSerialize; |
|||
import com.fasterxml.jackson.databind.ser.std.ToStringSerializer; |
|||
import lombok.AllArgsConstructor; |
|||
import lombok.Builder; |
|||
import lombok.Data; |
|||
import lombok.NoArgsConstructor; |
|||
|
|||
import java.io.Serial; |
|||
import java.io.Serializable; |
|||
import java.util.List; |
|||
|
|||
/** |
|||
* 目录树节点 - 用于返回树形结构 |
|||
*/ |
|||
@Data |
|||
@Builder |
|||
@AllArgsConstructor |
|||
@NoArgsConstructor |
|||
public class FolderNode implements Serializable { |
|||
|
|||
@Serial |
|||
private static final long serialVersionUID = 1L; |
|||
|
|||
/** |
|||
* 目录ID |
|||
*/ |
|||
@JsonSerialize(using = ToStringSerializer.class) |
|||
private Long id; |
|||
|
|||
/** |
|||
* 目录名称 |
|||
*/ |
|||
private String name; |
|||
|
|||
/** |
|||
* 所属分类ID |
|||
*/ |
|||
@JsonSerialize(using = ToStringSerializer.class) |
|||
private Long categoryId; |
|||
|
|||
/** |
|||
* 父目录ID |
|||
*/ |
|||
@JsonSerialize(using = ToStringSerializer.class) |
|||
private Long parentId; |
|||
|
|||
/** |
|||
* 排序权重 |
|||
*/ |
|||
private Integer sortOrder; |
|||
|
|||
/** |
|||
* 目录内文档数量 |
|||
*/ |
|||
private Integer documentCount; |
|||
|
|||
/** |
|||
* 子目录列表 |
|||
*/ |
|||
private List<FolderNode> children; |
|||
} |
|||
@ -0,0 +1,80 @@ |
|||
package com.wok.supportbot.entity; |
|||
|
|||
import com.baomidou.mybatisplus.annotation.*; |
|||
import com.fasterxml.jackson.databind.annotation.JsonSerialize; |
|||
import com.fasterxml.jackson.databind.ser.std.ToStringSerializer; |
|||
import lombok.AllArgsConstructor; |
|||
import lombok.Builder; |
|||
import lombok.Data; |
|||
import lombok.NoArgsConstructor; |
|||
|
|||
import java.io.Serial; |
|||
import java.io.Serializable; |
|||
import java.util.Date; |
|||
|
|||
/** |
|||
* 知识库文档目录表 - 支持分类下的目录树形结构 |
|||
*/ |
|||
@Data |
|||
@Builder |
|||
@AllArgsConstructor |
|||
@NoArgsConstructor |
|||
@TableName("knowledge_folder") |
|||
public class KnowledgeFolder implements Serializable { |
|||
|
|||
@Serial |
|||
@TableField(exist = false) |
|||
private static final long serialVersionUID = 1L; |
|||
|
|||
/** |
|||
* 目录ID |
|||
*/ |
|||
@TableId(value = "id", type = IdType.ASSIGN_ID) |
|||
@JsonSerialize(using = ToStringSerializer.class) |
|||
private Long id; |
|||
|
|||
/** |
|||
* 所属分类ID |
|||
*/ |
|||
@TableField("category_id") |
|||
@JsonSerialize(using = ToStringSerializer.class) |
|||
private Long categoryId; |
|||
|
|||
/** |
|||
* 父目录ID,0=该分类下的根目录 |
|||
*/ |
|||
@TableField("parent_id") |
|||
@JsonSerialize(using = ToStringSerializer.class) |
|||
private Long parentId; |
|||
|
|||
/** |
|||
* 目录名称 |
|||
*/ |
|||
@TableField("name") |
|||
private String name; |
|||
|
|||
/** |
|||
* 排序权重(越大越靠前) |
|||
*/ |
|||
@TableField("sort_order") |
|||
private Integer sortOrder; |
|||
|
|||
/** |
|||
* 创建时间 |
|||
*/ |
|||
@TableField(value = "create_time", fill = FieldFill.INSERT) |
|||
private Date createTime; |
|||
|
|||
/** |
|||
* 更新时间 |
|||
*/ |
|||
@TableField(value = "update_time", fill = FieldFill.INSERT_UPDATE) |
|||
private Date updateTime; |
|||
|
|||
/** |
|||
* 逻辑删除标志 - false:未删除, true:已删除(逻辑删除) |
|||
*/ |
|||
@TableField("is_delete") |
|||
@TableLogic |
|||
private boolean isDelete; |
|||
} |
|||
Some files were not shown because too many files changed in this diff
Write
Preview
Loading…
Cancel
Save
Reference in new issue