Compare commits
47
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
121e155248 | ||
|
|
d513c5f892 | ||
|
|
4cb97318b5 | ||
|
|
31b2a4d6ca | ||
|
|
db54921066 | ||
|
|
9fee9a1478 | ||
|
|
d3acccd146 | ||
|
|
d1f780e381 | ||
|
|
acb6ddbb4c | ||
|
|
cccc946660 | ||
|
|
aa15841d73 | ||
|
|
0246943165 | ||
|
|
d53eba77f0 | ||
|
|
3d7f5ca3cb | ||
|
|
d29bb5504a | ||
|
|
a583f44d5d | ||
|
|
5f3ca9a246 | ||
|
|
77262f8c3a | ||
|
|
84a3f6dd5e | ||
|
|
0e9efe4895 | ||
|
|
4480bce684 | ||
|
|
58725c8a12 | ||
|
|
90f5544673 | ||
|
|
5bcf22f9a7 | ||
|
|
6ef5d11d9d | ||
|
|
eb88a77238 | ||
|
|
a833cd0d5f | ||
|
|
207137c0dd | ||
|
|
b2ecb8e12b | ||
|
|
910ab8aa38 | ||
|
|
8de7f6696c | ||
|
|
1348660cd8 | ||
|
|
177a9ac9a6 | ||
|
|
ce142fe32f | ||
|
|
d03dbc7f91 | ||
|
|
a5af4bb404 | ||
|
|
9e0d39c912 | ||
|
|
ce3fb7fc74 | ||
|
|
2b133f1531 | ||
|
|
40523f6f8a | ||
|
|
8a5162c912 | ||
|
|
116d6b63bd | ||
|
|
0fa32cb5e7 | ||
|
|
63c29680ce | ||
|
|
76afa37c6b | ||
|
|
293d71e86b | ||
|
|
d3a672a2a8 |
No files matched your search
@@ -78,7 +78,7 @@
|
|||||||
| [aibox](aibox/) | Python | 8765 | 独立 AI 智能体系统(DeepSeek + WebSocket),自带技能系统/记忆/任务调度,由 core 拉起但与主干微服务无调用关系 |
|
| [aibox](aibox/) | Python | 8765 | 独立 AI 智能体系统(DeepSeek + WebSocket),自带技能系统/记忆/任务调度,由 core 拉起但与主干微服务无调用关系 |
|
||||||
| [BotOps](BotOps/) | Python | — | aibox 智能体 TUI 前端(Textual),WebSocket 连接 aibox,支持 Markdown 增量流式渲染、思考模式展示、任务独立侧栏、自动重连 |
|
| [BotOps](BotOps/) | Python | — | aibox 智能体 TUI 前端(Textual),WebSocket 连接 aibox,支持 Markdown 增量流式渲染、思考模式展示、任务独立侧栏、自动重连 |
|
||||||
| [bot4cj](bot4cj/) | 仓颉(Cangjie) | — | 机器人硬件脚本解析器,解析 .mt 矩阵脚本通过 I2C 驱动 PCA9685 舵机 / MPU6050 陀螺仪,面向树莓派 Dashun 蜘蛛机器人平台(独立硬件项目) |
|
| [bot4cj](bot4cj/) | 仓颉(Cangjie) | — | 机器人硬件脚本解析器,解析 .mt 矩阵脚本通过 I2C 驱动 PCA9685 舵机 / MPU6050 陀螺仪,面向树莓派 Dashun 蜘蛛机器人平台(独立硬件项目) |
|
||||||
| [pybot](pybot/) | Python | — | bot4cj 的 Python 迁移版:.mt 动作脚本解析器(零第三方依赖,与仓颉版逐语义对齐);支持执行 / `--dry-run` / `--record` 录制 actpred episode(含 imu.npy)/ `--loop` 实时预测闭环 |
|
| [pybot](pybot/) | Python | — | bot4cj 的 Python 迁移版:.mt 动作脚本解析器(零第三方依赖,与仓颉版逐语义对齐);支持执行 / `--dry-run` / `--record` 录制 actpred episode(含 imu.npy)/ `--loop` 实时预测闭环(HEF 后端自动异步,推理与执行并行) |
|
||||||
| [mtc](mtc/) | Python | — | 命令式动作脚本(Python DSL)→ .mt 编译器:非阻塞时间轴 + 10ms 栅格展开,输出可由 pybot/bot4cj 直接执行;零第三方依赖 |
|
| [mtc](mtc/) | Python | — | 命令式动作脚本(Python DSL)→ .mt 编译器:非阻塞时间轴 + 10ms 栅格展开,输出可由 pybot/bot4cj 直接执行;零第三方依赖 |
|
||||||
| [actpred](actpred/) | Python(PyTorch) | — | 动作预测模型(图像 + 动作历史 → 下一动作),bot4cj/pybot 的「大脑」配套;两阶段训练(.mt 预训练 → episode 微调),ONNX/Hailo HEF 部署(实验) |
|
| [actpred](actpred/) | Python(PyTorch) | — | 动作预测模型(图像 + 动作历史 → 下一动作),bot4cj/pybot 的「大脑」配套;两阶段训练(.mt 预训练 → episode 微调),ONNX/Hailo HEF 部署(实验) |
|
||||||
| [lci](lci/) | C++ | — | 嵌入式帧缓冲图形界面(LCI),基于 `/dev/fb0` 直接操作显存,无需 X11/Wayland,矢量字体、图片渲染、实时数据可视化,面向树莓派等嵌入式环境 |
|
| [lci](lci/) | C++ | — | 嵌入式帧缓冲图形界面(LCI),基于 `/dev/fb0` 直接操作显存,无需 X11/Wayland,矢量字体、图片渲染、实时数据可视化,面向树莓派等嵌入式环境 |
|
||||||
@@ -235,7 +235,8 @@
|
|||||||
|
|
||||||
- 一套相对独立的单智能体产品,基于 DeepSeek + WebSocket(:8765),由 `core` 拉起但与 waf / gateway / dnfs2 等主干微服务无调用关系。
|
- 一套相对独立的单智能体产品,基于 DeepSeek + WebSocket(:8765),由 `core` 拉起但与 waf / gateway / dnfs2 等主干微服务无调用关系。
|
||||||
- 自带技能系统(`skills/`)、任务调度器、记忆管理、前端页面与语音客户端(`voice_client.py`)。
|
- 自带技能系统(`skills/`)、任务调度器、记忆管理、前端页面与语音客户端(`voice_client.py`)。
|
||||||
- 默认模型 `deepseek-v4-flash`,配置在 `config.json`;详见 [aibox/README.md](aibox/README.md)。
|
- 默认模型 `deepseek-v4-flash`,配置在 `config.json`;`models` 配置节可挂多个 OpenAI 兼容端点按档案名切换(档案级 reasoning/温度/展示名/`fallback` 失败转移),`--once` 支持无 WebSocket 的 CLI 一次性对话;详见 [aibox/README.md](aibox/README.md)。
|
||||||
|
- **四轮安全审计修复(2026-08,Batch E/F/G/H/I 共 60+ 项)**:命令执行保护链(1MB/超时/中断)、凭据脱敏(代理密码不落盘、日志打码、Bearer 脱敏)、取消传播(子代理/情绪子代理/调度器超时)、DB 事务与 FTS5 转义、POSIX 进程组终止等;审计报告见 [docs/reports/aibox_audit_20260817.md](docs/reports/aibox_audit_20260817.md)。
|
||||||
|
|
||||||
### BotOps(aibox 智能体 TUI 前端)
|
### BotOps(aibox 智能体 TUI 前端)
|
||||||
|
|
||||||
@@ -256,7 +257,7 @@
|
|||||||
### pybot(机器人动作脚本解析器 · Python 迁移版)
|
### pybot(机器人动作脚本解析器 · Python 迁移版)
|
||||||
|
|
||||||
- [bot4cj](#bot4cj机器人硬件脚本解析器) 的 Python 迁移版,零第三方依赖(纯标准库),与仓颉版逐语义对齐(含占位符 `-` 递归继承、微步分解、块序依赖)。
|
- [bot4cj](#bot4cj机器人硬件脚本解析器) 的 Python 迁移版,零第三方依赖(纯标准库),与仓颉版逐语义对齐(含占位符 `-` 递归继承、微步分解、块序依赖)。
|
||||||
- 三种运行模式:脚本执行 `run()`、`--record` 录制 actpred episode(images + actions.npy + imu.npy,≤4 个 MPU6050 按块行序拼接)、`--loop` 实时预测闭环(daemon 线程队列非阻塞取帧,overrun 告警)。
|
- 三种运行模式:脚本执行 `run()`、`--record` 录制 actpred episode(images + actions.npy + imu.npy,≤4 个 MPU6050 按块行序拼接)、`--loop` 实时预测闭环(daemon 线程队列非阻塞取帧,overrun 告警;HEF 后端自动走异步推理——执行上一步动作与 NPU 推理并行,滞后一拍)。
|
||||||
- 扬声器静音语义:`frequency <= 0` 不出声仅保持节拍(对齐仓颉版 sleep 占位语义)。
|
- 扬声器静音语义:`frequency <= 0` 不出声仅保持节拍(对齐仓颉版 sleep 占位语义)。
|
||||||
- Windows 开发机可跑测试(fcntl mock + 符号链接 fallback);详见 [pybot/README.md](pybot/README.md)。
|
- Windows 开发机可跑测试(fcntl mock + 符号链接 fallback);详见 [pybot/README.md](pybot/README.md)。
|
||||||
|
|
||||||
|
|||||||
@@ -311,3 +311,17 @@ python3 dfc_compile.py # 产物 checkpoints/actpred_pretrain.hef
|
|||||||
`--pred-token` 喂入学习值、`--stats` 喂入归一化统计、`--imu` 喂入姿态。
|
`--pred-token` 喂入学习值、`--stats` 喂入归一化统计、`--imu` 喂入姿态。
|
||||||
环境调研与 API 踩坑记录见
|
环境调研与 API 踩坑记录见
|
||||||
[docs/reports/actpred_hailo_hat2_inference_research_20260813.md](../docs/reports/actpred_hailo_hat2_inference_research_20260813.md)。
|
[docs/reports/actpred_hailo_hat2_inference_research_20260813.md](../docs/reports/actpred_hailo_hat2_inference_research_20260813.md)。
|
||||||
|
|
||||||
|
### 异步推理(`predict_async`,run_async 流水线)
|
||||||
|
|
||||||
|
`HailoPredictor.predict_async(images, action_history, pred_token=None, imu=None)`
|
||||||
|
提交异步推理并立即返回 `_AsyncJob`,需要结果时 `job.result(timeout)` 阻塞获取:
|
||||||
|
|
||||||
|
- 与同步 `predict` 语义/输出完全一致(公共 `_prepare_inputs/_fill_bindings/_postprocess`)
|
||||||
|
- 内部维护 4 个绑定缓冲池 + 完成回调;在途任务超过缓冲数时提交会阻塞等待
|
||||||
|
空闲缓冲,`job.result` 超时**不会**释放缓冲(缓冲由完成回调释放)
|
||||||
|
- 连续提交(≥2 个在途)时 NPU 保持满负荷:实测连续吞吐 6.0ms vs 同步
|
||||||
|
15.5ms(约 2.6×);扩展测试覆盖乱序取结果/混合调用/缓冲压力/300 次
|
||||||
|
稳定性/4 线程并发,输出与同步 0 差异
|
||||||
|
- pybot `--loop` 检测到 `predict_async` 即自动走异步闭环(推理期间并行
|
||||||
|
执行上一步动作),见 [pybot/README.md](../pybot/README.md)
|
||||||
+38
-23
@@ -4,7 +4,7 @@
|
|||||||
|
|
||||||
4 角测试(真实/全零帧 × 真实/全零历史)+ 与 PyTorch 参考对比。
|
4 角测试(真实/全零帧 × 真实/全零历史)+ 与 PyTorch 参考对比。
|
||||||
HEF 尺寸/布局从 vstream info 自动推导,checkpoint 用
|
HEF 尺寸/布局从 vstream info 自动推导,checkpoint 用
|
||||||
checkpoints/pretrain_best_inference.pt。
|
checkpoints/best_inference.pt(v4 onehot,fixed_range)。
|
||||||
"""
|
"""
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
@@ -20,8 +20,8 @@ from config import parse_checkpoint_config
|
|||||||
from model import ActionPredictor
|
from model import ActionPredictor
|
||||||
from hailo_infer import HailoPredictor
|
from hailo_infer import HailoPredictor
|
||||||
|
|
||||||
HEF_PATH = "checkpoints/actpred_pretrain.hef"
|
HEF_PATH = "checkpoints/actpred_v4.hef"
|
||||||
CKPT = "checkpoints/pretrain_best_inference.pt"
|
CKPT = "checkpoints/best_inference.pt"
|
||||||
IMG_DIR = "data/episode_0000/images"
|
IMG_DIR = "data/episode_0000/images"
|
||||||
|
|
||||||
hef = HEF(HEF_PATH)
|
hef = HEF(HEF_PATH)
|
||||||
@@ -40,11 +40,11 @@ imgs_real = [np.array(Image.open(f"{IMG_DIR}/000{i}.jpg").convert("RGB"))
|
|||||||
imgs_zero = [np.zeros_like(imgs_real[0]) for _ in range(3)]
|
imgs_zero = [np.zeros_like(imgs_real[0]) for _ in range(3)]
|
||||||
|
|
||||||
hist_real = np.array([
|
hist_real = np.array([
|
||||||
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 10, 200, 0, 0, 0, 0, 0, 0],
|
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 10, 0, 0, 0, 0, 0, 0],
|
||||||
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 20, 0, 0, 0, 0, 0, 0],
|
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 10, 0, 0, 0, 0, 0, 0],
|
||||||
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 0, 0, 0, 0, 0, 0, 0],
|
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 10, 0, 0, 0, 0, 0, 0],
|
||||||
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 0, 0, 0, 0, 0, 0, 0],
|
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 10, 0, 0, 0, 0, 0, 0],
|
||||||
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 0, 0, 0, 0, 0, 0, 0],
|
[45, 0, 45, 0, 45, 0, 45, 0, 90, 90, 0, 10, 0, 0, 0, 0, 0, 0],
|
||||||
], dtype=np.float32)
|
], dtype=np.float32)
|
||||||
hist_zero = np.zeros((5, 18), dtype=np.float32)
|
hist_zero = np.zeros((5, 18), dtype=np.float32)
|
||||||
|
|
||||||
@@ -99,12 +99,24 @@ def torch_pred(imgs, hist, use_learned_pred=True, imu=None):
|
|||||||
ii = torch.from_numpy(normalize_imu(imu, imu_dim)).unsqueeze(0)
|
ii = torch.from_numpy(normalize_imu(imu, imu_dim)).unsqueeze(0)
|
||||||
if use_learned_pred:
|
if use_learned_pred:
|
||||||
with torch.no_grad():
|
with torch.no_grad():
|
||||||
return model(x, h, ii).squeeze(0).numpy()
|
out = model(x, h, ii).squeeze(0).numpy()
|
||||||
saved = model.pred_token.data.clone()
|
else:
|
||||||
model.pred_token.data.zero_()
|
saved = model.pred_token.data.clone()
|
||||||
with torch.no_grad():
|
model.pred_token.data.zero_()
|
||||||
out = model(x, h, ii).squeeze(0).numpy()
|
with torch.no_grad():
|
||||||
model.pred_token.data.copy_(saved)
|
out = model(x, h, ii).squeeze(0).numpy()
|
||||||
|
model.pred_token.data.copy_(saved)
|
||||||
|
if cfg.data.onehot:
|
||||||
|
# 多头 logits → 期望值解码 → 原始量纲(与 predict.py 同语义,
|
||||||
|
# 含权重 0 列固定常量),对比前无需反归一化
|
||||||
|
from bins import decode as bin_decode, fixed_zero_columns
|
||||||
|
from action_norm import default_column_weights
|
||||||
|
out = bin_decode(out, cfg.data.action_dim).astype(np.float32)
|
||||||
|
w = (np.asarray(cfg.data.column_weights, dtype=np.float32)
|
||||||
|
if cfg.data.column_weights is not None
|
||||||
|
else default_column_weights(cfg.data.action_dim))
|
||||||
|
fixed = fixed_zero_columns(w, cfg.data.action_dim)
|
||||||
|
out[w <= 0] = fixed[w <= 0]
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
@@ -141,15 +153,18 @@ with HailoPredictor(HEF_PATH, stats_path=stats_path,
|
|||||||
ref = torch_pred(imgs_real, hist_real, True, imu_real)
|
ref = torch_pred(imgs_real, hist_real, True, imu_real)
|
||||||
ref_zero_pred = torch_pred(imgs_real, hist_real, False, imu_real)
|
ref_zero_pred = torch_pred(imgs_real, hist_real, False, imu_real)
|
||||||
# torch 参考是模型原始归一化空间输出;HailoPredictor 已反归一化
|
# torch 参考是模型原始归一化空间输出;HailoPredictor 已反归一化
|
||||||
# 到原始动作空间,比较前须同空间对齐,否则归一化模型下 max_diff 无意义
|
# 到原始动作空间,比较前须同空间对齐,否则归一化模型下 max_diff
|
||||||
if norm_mode == "fixed_range":
|
# 无意义(onehot 模型 torch_pred 已 decode 为原始量纲,无需再处理)
|
||||||
from action_norm import denormalize_action
|
if not cfg.data.onehot:
|
||||||
ref = denormalize_action(ref, cfg.data.action_dim)
|
if norm_mode == "fixed_range":
|
||||||
ref_zero_pred = denormalize_action(ref_zero_pred, cfg.data.action_dim)
|
from action_norm import denormalize_action
|
||||||
elif stats_path is not None:
|
ref = denormalize_action(ref, cfg.data.action_dim)
|
||||||
s = np.load(stats_path)
|
ref_zero_pred = denormalize_action(ref_zero_pred,
|
||||||
ref = ref * s[1] + s[0]
|
cfg.data.action_dim)
|
||||||
ref_zero_pred = ref_zero_pred * s[1] + s[0]
|
elif stats_path is not None:
|
||||||
|
s = np.load(stats_path)
|
||||||
|
ref = ref * s[1] + s[0]
|
||||||
|
ref_zero_pred = ref_zero_pred * s[1] + s[0]
|
||||||
print(f"torch learned_pred : {np.round(ref[:6], 3)}...")
|
print(f"torch learned_pred : {np.round(ref[:6], 3)}...")
|
||||||
print(f"torch zero_pred : {np.round(ref_zero_pred[:6], 3)}...")
|
print(f"torch zero_pred : {np.round(ref_zero_pred[:6], 3)}...")
|
||||||
print(f"HEF vs torch(learned) max_diff={np.abs(a - ref).max():.4f}"
|
print(f"HEF vs torch(learned) max_diff={np.abs(a - ref).max():.4f}"
|
||||||
|
|||||||
@@ -86,6 +86,7 @@ def build_image_samples(n_samples, frame_len, height, width, rng,
|
|||||||
|
|
||||||
def build_deploy_set(args):
|
def build_deploy_set(args):
|
||||||
import torch
|
import torch
|
||||||
|
from action_norm import is_range_mode, normalize_action
|
||||||
from model import ActionPredictor
|
from model import ActionPredictor
|
||||||
|
|
||||||
ckpt = torch.load(args.model, map_location="cpu", weights_only=True)
|
ckpt = torch.load(args.model, map_location="cpu", weights_only=True)
|
||||||
|
|||||||
+229
-37
@@ -31,18 +31,52 @@ import contextlib
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
from PIL import Image
|
from PIL import Image
|
||||||
|
|
||||||
|
try:
|
||||||
|
import cv2 # 可选:帧归一化+量化合并查表(加速预处理)
|
||||||
|
HAS_CV2 = True
|
||||||
|
except ImportError:
|
||||||
|
cv2 = None
|
||||||
|
HAS_CV2 = False
|
||||||
|
|
||||||
from imu_norm import normalize_imu
|
from imu_norm import normalize_imu
|
||||||
from action_norm import (normalize_action, denormalize_action,
|
from action_norm import (normalize_action, denormalize_action,
|
||||||
is_range_mode)
|
is_range_mode)
|
||||||
from utils import parse_action_history
|
from utils import parse_action_history
|
||||||
|
from bins import TOTAL_BINS, decode as bin_decode, fixed_zero_columns
|
||||||
|
|
||||||
IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
||||||
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
||||||
|
|
||||||
|
# 异步推理缓冲数:≥4 时流水线吞吐趋于饱和(实测 4 缓冲 ~2.7×、6 缓冲 ~2.9×)
|
||||||
|
_ASYNC_BUFFERS = 4
|
||||||
|
|
||||||
|
|
||||||
|
class _AsyncJob:
|
||||||
|
"""predict_async 的推理句柄:result() 阻塞返回 (action_dim,) 动作向量"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._done = threading.Event()
|
||||||
|
self._result = None
|
||||||
|
self._error = None
|
||||||
|
|
||||||
|
def result(self, timeout: float = None) -> np.ndarray:
|
||||||
|
if not self._done.wait(timeout):
|
||||||
|
raise TimeoutError("async HEF inference timeout")
|
||||||
|
if self._error is not None:
|
||||||
|
raise self._error
|
||||||
|
return self._result
|
||||||
|
|
||||||
|
def _complete(self, result=None, error=None):
|
||||||
|
self._result = result
|
||||||
|
self._error = error
|
||||||
|
self._done.set()
|
||||||
|
|
||||||
|
|
||||||
def _input_layer_index(name: str) -> int:
|
def _input_layer_index(name: str) -> int:
|
||||||
m = re.search(r"input_layer(\d+)$", name)
|
m = re.search(r"input_layer(\d+)$", name)
|
||||||
@@ -62,7 +96,7 @@ class HailoPredictor:
|
|||||||
|
|
||||||
def __init__(self, hef_path: str, stats_path: str = None, vdevice=None,
|
def __init__(self, hef_path: str, stats_path: str = None, vdevice=None,
|
||||||
predict_delta: bool = False, pred_token=None,
|
predict_delta: bool = False, pred_token=None,
|
||||||
normalize_mode: str = "zscore"):
|
normalize_mode: str = "zscore", column_weights=None):
|
||||||
"""HEF 推理器。
|
"""HEF 推理器。
|
||||||
|
|
||||||
predict_delta: 模型用 --predict-delta 训练时须置 True,推理输出
|
predict_delta: 模型用 --predict-delta 训练时须置 True,推理输出
|
||||||
@@ -72,12 +106,19 @@ class HailoPredictor:
|
|||||||
normalize_mode: 与训练时 DataConfig.normalize_mode 一致。
|
normalize_mode: 与训练时 DataConfig.normalize_mode 一致。
|
||||||
"zscore"(默认)需传 stats_path;"fixed_range" / "log_range"
|
"zscore"(默认)需传 stats_path;"fixed_range" / "log_range"
|
||||||
免 stats 文件,按固定物理范围归一化(action_norm.py)。
|
免 stats 文件,按固定物理范围归一化(action_norm.py)。
|
||||||
|
column_weights: 逐列损失权重(onehot HEF 的权重 0 列按
|
||||||
|
bins.FIXED_OUTPUT 固定,防闭环垃圾回写);None 时用
|
||||||
|
action_norm.default_column_weights。
|
||||||
"""
|
"""
|
||||||
from hailo_platform import HEF, VDevice
|
from hailo_platform import HEF, VDevice
|
||||||
|
|
||||||
self._predict_delta = predict_delta
|
self._predict_delta = predict_delta
|
||||||
self._range_mode = (normalize_mode
|
self._range_mode = (normalize_mode
|
||||||
if is_range_mode(normalize_mode) else None)
|
if is_range_mode(normalize_mode) else None)
|
||||||
|
self._column_weights = None
|
||||||
|
if column_weights is not None:
|
||||||
|
self._column_weights = np.asarray(column_weights,
|
||||||
|
dtype=np.float32)
|
||||||
self._default_pred_token = None
|
self._default_pred_token = None
|
||||||
if pred_token is not None:
|
if pred_token is not None:
|
||||||
if isinstance(pred_token, str):
|
if isinstance(pred_token, str):
|
||||||
@@ -132,7 +173,6 @@ class HailoPredictor:
|
|||||||
imu 为 R2 方案一新增输入(1,1,1,imu_dim),旧 HEF 无此输入。
|
imu 为 R2 方案一新增输入(1,1,1,imu_dim),旧 HEF 无此输入。
|
||||||
"""
|
"""
|
||||||
ins = list(self._in_infos)
|
ins = list(self._in_infos)
|
||||||
out_dim = tuple(self._out_info.shape)[-1]
|
|
||||||
# 新版:每帧独立 (H,W,3) 输入
|
# 新版:每帧独立 (H,W,3) 输入
|
||||||
frames = [i for i in ins if len(i.shape) == 3
|
frames = [i for i in ins if len(i.shape) == 3
|
||||||
and i.shape[2] == 3 and i.shape[0] > 3]
|
and i.shape[2] == 3 and i.shape[0] > 3]
|
||||||
@@ -145,10 +185,10 @@ class HailoPredictor:
|
|||||||
history, pred, imu = None, None, None
|
history, pred, imu = None, None, None
|
||||||
for i in rest:
|
for i in rest:
|
||||||
s = tuple(i.shape)
|
s = tuple(i.shape)
|
||||||
if len(s) == 3 and s[-1] == out_dim:
|
if len(s) == 3 and s[1] == 1:
|
||||||
history = i
|
pred = i # pred_token (1,1,d)
|
||||||
elif len(s) == 3 and s[1] == 1:
|
elif len(s) == 3:
|
||||||
pred = i
|
history = i # 动作历史 (1,H_len,D)
|
||||||
elif len(s) == 4 and s[0] == 1 and s[1] == 1 and s[2] == 1:
|
elif len(s) == 4 and s[0] == 1 and s[1] == 1 and s[2] == 1:
|
||||||
imu = i # R2:imu (1,1,1,imu_dim)
|
imu = i # R2:imu (1,1,1,imu_dim)
|
||||||
else:
|
else:
|
||||||
@@ -159,7 +199,8 @@ class HailoPredictor:
|
|||||||
raise ValueError("HEF 输入中未找到图像输入")
|
raise ValueError("HEF 输入中未找到图像输入")
|
||||||
if history is None:
|
if history is None:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"HEF 输入中未找到动作历史输入(输出维度 {out_dim}),"
|
f"HEF 输入中未找到动作历史输入(输出维度 "
|
||||||
|
f"{tuple(self._out_info.shape)}),"
|
||||||
f"实际: {[(i.name, tuple(i.shape)) for i in ins]}")
|
f"实际: {[(i.name, tuple(i.shape)) for i in ins]}")
|
||||||
return frames, history, pred, imu
|
return frames, history, pred, imu
|
||||||
|
|
||||||
@@ -199,6 +240,20 @@ class HailoPredictor:
|
|||||||
self._normalize_frames = \
|
self._normalize_frames = \
|
||||||
(0 - self._frame_quant.qp_zp) * self._frame_quant.qp_scale < -0.5
|
(0 - self._frame_quant.qp_zp) * self._frame_quant.qp_scale < -0.5
|
||||||
|
|
||||||
|
# cv2.LUT 查表(归一化+量化合并一步):仅逐帧独立输入布局且
|
||||||
|
# 帧边缘为 uint8 时可用;无 cv2 或旧布局回退 float 路径
|
||||||
|
from hailo_platform import FormatType
|
||||||
|
self._frame_luts = None
|
||||||
|
if HAS_CV2 and self._image_layout == "nhwc_frames" and \
|
||||||
|
all(f.format.type == FormatType.UINT8
|
||||||
|
for f in self._frame_infos):
|
||||||
|
self._frame_luts = self._build_frame_luts()
|
||||||
|
|
||||||
|
# 异步推理缓冲池(predict_async 复用)
|
||||||
|
self._async_buffers = [self._configured.create_bindings()
|
||||||
|
for _ in range(_ASYNC_BUFFERS)]
|
||||||
|
self._async_flags = [[True] for _ in range(_ASYNC_BUFFERS)]
|
||||||
|
|
||||||
h_shape = tuple(self._history_info.shape)
|
h_shape = tuple(self._history_info.shape)
|
||||||
self.history_len, self.action_dim = h_shape[1], h_shape[2]
|
self.history_len, self.action_dim = h_shape[1], h_shape[2]
|
||||||
|
|
||||||
@@ -209,9 +264,16 @@ class HailoPredictor:
|
|||||||
|
|
||||||
out_shape = tuple(self._out_info.shape)
|
out_shape = tuple(self._out_info.shape)
|
||||||
out_dim = out_shape[-1]
|
out_dim = out_shape[-1]
|
||||||
if out_dim != self.action_dim:
|
if out_dim == TOTAL_BINS:
|
||||||
raise ValueError(
|
# onehot 模型:输出为多头 logits,predict 时经 bins.decode
|
||||||
f"输出动作维度 {out_dim} 与动作历史维度 {self.action_dim} 不一致")
|
# 解码为 action_dim 原始量纲(无需反归一化)
|
||||||
|
self._onehot = True
|
||||||
|
else:
|
||||||
|
self._onehot = False
|
||||||
|
if out_dim != self.action_dim:
|
||||||
|
raise ValueError(
|
||||||
|
f"输出动作维度 {out_dim} 与动作历史维度 "
|
||||||
|
f"{self.action_dim} 不一致")
|
||||||
|
|
||||||
# ---------------- 生命周期 ----------------
|
# ---------------- 生命周期 ----------------
|
||||||
|
|
||||||
@@ -227,16 +289,8 @@ class HailoPredictor:
|
|||||||
|
|
||||||
# ---------------- 推理 ----------------
|
# ---------------- 推理 ----------------
|
||||||
|
|
||||||
def predict(self, images, action_history: np.ndarray,
|
def _prepare_inputs(self, images, action_history):
|
||||||
pred_token: np.ndarray = None,
|
"""输入校验 + 动作历史归一化(predict/predict_async 共用)"""
|
||||||
imu: np.ndarray = None) -> np.ndarray:
|
|
||||||
"""images: 长度为 frame_len 的 RGB 图像列表(旧→新,末帧为当前帧)
|
|
||||||
imu: 可选原始姿态向量(roll,pitch,ax,ay,az,gx,gy,gz,HEF 含 imu
|
|
||||||
输入时生效,内部按物理量程归一化;None 等价零向量)
|
|
||||||
|
|
||||||
返回 (action_dim,) float32 动作向量。pred_token 仅新版 5 输入 HEF
|
|
||||||
需要;默认喂零,抖动校准的新 HEF 建议喂 checkpoint 学习值。
|
|
||||||
"""
|
|
||||||
if not isinstance(images, (list, tuple)):
|
if not isinstance(images, (list, tuple)):
|
||||||
images = [images]
|
images = [images]
|
||||||
if len(images) != self.frame_len:
|
if len(images) != self.frame_len:
|
||||||
@@ -254,12 +308,21 @@ class HailoPredictor:
|
|||||||
mode=self._range_mode)
|
mode=self._range_mode)
|
||||||
elif self._action_mean is not None:
|
elif self._action_mean is not None:
|
||||||
action_history = (action_history - self._action_mean) / self._action_std
|
action_history = (action_history - self._action_mean) / self._action_std
|
||||||
|
return images, action_history
|
||||||
|
|
||||||
|
def _fill_bindings(self, bindings, images, action_history,
|
||||||
|
pred_token=None, imu=None):
|
||||||
|
"""按 HEF 布局把(已归一化的)输入填充到 bindings 缓冲"""
|
||||||
from hailo_platform import FormatType
|
from hailo_platform import FormatType
|
||||||
bindings = self._configured.create_bindings()
|
|
||||||
|
|
||||||
frame_float = np.stack([self._frame_float(img) for img in images])
|
if self._frame_luts is not None:
|
||||||
self._bind_frames(bindings, frame_float)
|
# cv2.LUT 快路径:逐帧独立输入,直接绑定量化缓冲
|
||||||
|
for i, (info, img) in enumerate(zip(self._frame_infos, images)):
|
||||||
|
buf = self._frame_lut_quant(img, i)
|
||||||
|
bindings.input(info.name).set_buffer(buf)
|
||||||
|
else:
|
||||||
|
frame_float = np.stack([self._frame_float(img) for img in images])
|
||||||
|
self._bind_frames(bindings, frame_float)
|
||||||
|
|
||||||
h_shape = tuple(self._history_info.shape)
|
h_shape = tuple(self._history_info.shape)
|
||||||
if self._history_info.format.type == FormatType.UINT8:
|
if self._history_info.format.type == FormatType.UINT8:
|
||||||
@@ -301,28 +364,120 @@ class HailoPredictor:
|
|||||||
out_buf = np.zeros(out_shape, dtype=np.float32)
|
out_buf = np.zeros(out_shape, dtype=np.float32)
|
||||||
bindings.output(self._out_info.name).set_buffer(out_buf)
|
bindings.output(self._out_info.name).set_buffer(out_buf)
|
||||||
|
|
||||||
self._configured.run([bindings], timeout=10000)
|
def _postprocess(self, raw_result, history_tail=None) -> np.ndarray:
|
||||||
|
"""反量化 + 解码/反归一化 → (action_dim,)(predict/predict_async 共用)
|
||||||
|
|
||||||
result = np.asarray(
|
history_tail: 归一化空间的历史末步(predict_delta 残差模式用)。
|
||||||
bindings.output(self._out_info.name).get_buffer(),
|
"""
|
||||||
dtype=np.float32)
|
from hailo_platform import FormatType
|
||||||
if self._out_info.format.type == FormatType.UINT8:
|
if self._out_info.format.type == FormatType.UINT8:
|
||||||
q = self._out_info.quant_info
|
q = self._out_info.quant_info
|
||||||
result = (result - q.qp_zp) * q.qp_scale
|
raw_result = (raw_result - q.qp_zp) * q.qp_scale
|
||||||
result = result.reshape(-1)
|
result = raw_result.reshape(-1)
|
||||||
if self._predict_delta:
|
if self._onehot:
|
||||||
# 残差模式:Δ̂ + a_{t-1}(归一化空间),与 predict.py 对齐
|
# 多头 logits → 期望值解码 → 原始量纲(箱生活在原始空间,
|
||||||
result = result + action_history[-1]
|
# 无需反归一化);权重 0 列固定为常量(与 predict.py 对齐),
|
||||||
if self._range_mode:
|
# 防闭环垃圾回写
|
||||||
result = denormalize_action(result, self.action_dim,
|
result = bin_decode(result, self.action_dim).reshape(-1)
|
||||||
mode=self._range_mode)
|
if self._column_weights is None:
|
||||||
elif self._action_mean is not None:
|
from action_norm import default_column_weights
|
||||||
result = result * self._action_std + self._action_mean
|
self._column_weights = default_column_weights(
|
||||||
|
self.action_dim)
|
||||||
|
w = np.asarray(self._column_weights, dtype=np.float32)
|
||||||
|
fixed = fixed_zero_columns(w, self.action_dim)
|
||||||
|
result[w[:self.action_dim] <= 0.0] = (
|
||||||
|
fixed[w[:self.action_dim] <= 0.0])
|
||||||
|
else:
|
||||||
|
if self._predict_delta:
|
||||||
|
# 残差模式:Δ̂ + a_{t-1}(归一化空间),与 predict.py 对齐
|
||||||
|
result = result + history_tail
|
||||||
|
if self._range_mode:
|
||||||
|
result = denormalize_action(result, self.action_dim,
|
||||||
|
mode=self._range_mode)
|
||||||
|
elif self._action_mean is not None:
|
||||||
|
result = result * self._action_std + self._action_mean
|
||||||
if not np.all(np.isfinite(result)):
|
if not np.all(np.isfinite(result)):
|
||||||
print("WARNING: HEF output contains NaN/Inf, replacing with zeros")
|
print("WARNING: HEF output contains NaN/Inf, replacing with zeros")
|
||||||
result = np.nan_to_num(result, nan=0.0, posinf=0.0, neginf=0.0)
|
result = np.nan_to_num(result, nan=0.0, posinf=0.0, neginf=0.0)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
def predict(self, images, action_history: np.ndarray,
|
||||||
|
pred_token: np.ndarray = None,
|
||||||
|
imu: np.ndarray = None) -> np.ndarray:
|
||||||
|
"""images: 长度为 frame_len 的 RGB 图像列表(旧→新,末帧为当前帧)
|
||||||
|
imu: 可选原始姿态向量(roll,pitch,ax,ay,az,gx,gy,gz,HEF 含 imu
|
||||||
|
输入时生效,内部按物理量程归一化;None 等价零向量)
|
||||||
|
|
||||||
|
返回 (action_dim,) float32 动作向量。pred_token 仅新版 5 输入 HEF
|
||||||
|
需要;默认喂零,抖动校准的新 HEF 建议喂 checkpoint 学习值。
|
||||||
|
"""
|
||||||
|
images, action_history = self._prepare_inputs(images, action_history)
|
||||||
|
bindings = self._configured.create_bindings()
|
||||||
|
self._fill_bindings(bindings, images, action_history,
|
||||||
|
pred_token=pred_token, imu=imu)
|
||||||
|
self._configured.run([bindings], timeout=10000)
|
||||||
|
raw = np.asarray(
|
||||||
|
bindings.output(self._out_info.name).get_buffer(),
|
||||||
|
dtype=np.float32)
|
||||||
|
return self._postprocess(raw, action_history[-1])
|
||||||
|
|
||||||
|
# ---------------- 异步推理 ----------------
|
||||||
|
|
||||||
|
def predict_async(self, images, action_history: np.ndarray,
|
||||||
|
pred_token: np.ndarray = None,
|
||||||
|
imu: np.ndarray = None) -> "_AsyncJob":
|
||||||
|
"""提交异步推理(run_async 流水线,多缓冲复用)
|
||||||
|
|
||||||
|
与 predict() 语义/输出完全一致,但立即返回 job;调用方可在
|
||||||
|
NPU 推理期间并行做其他工作(如抓帧/执行上一步动作),需要结果
|
||||||
|
时 job.result() 阻塞获取。连续提交时(≥2 个在途)NPU 保持满负荷,
|
||||||
|
实测吞吐可达同步 predict 的 ~2.9×。
|
||||||
|
|
||||||
|
注意:绑定缓冲有限(_ASYNC_BUFFERS 个),在途任务超过缓冲数时
|
||||||
|
本调用会阻塞等待空闲缓冲;job.result(timeout) 超时不会释放缓冲,
|
||||||
|
缓冲由完成回调释放。
|
||||||
|
"""
|
||||||
|
images, action_history = self._prepare_inputs(images, action_history)
|
||||||
|
history_tail = action_history[-1]
|
||||||
|
idx = self._acquire_async_buffer()
|
||||||
|
bindings = self._async_buffers[idx]
|
||||||
|
job = _AsyncJob()
|
||||||
|
try:
|
||||||
|
self._fill_bindings(bindings, images, action_history,
|
||||||
|
pred_token=pred_token, imu=imu)
|
||||||
|
except Exception:
|
||||||
|
self._release_async_buffer(idx)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def cb(completion_info):
|
||||||
|
try:
|
||||||
|
exc = getattr(completion_info, "exception", None)
|
||||||
|
if exc:
|
||||||
|
raise exc
|
||||||
|
raw = np.asarray(
|
||||||
|
bindings.output(self._out_info.name).get_buffer(),
|
||||||
|
dtype=np.float32)
|
||||||
|
job._complete(result=self._postprocess(raw, history_tail))
|
||||||
|
except Exception as e: # noqa: BLE001 —— 回调内兜底
|
||||||
|
job._complete(error=e)
|
||||||
|
finally:
|
||||||
|
self._release_async_buffer(idx)
|
||||||
|
|
||||||
|
self._configured.run_async([bindings], callback=cb)
|
||||||
|
return job
|
||||||
|
|
||||||
|
def _acquire_async_buffer(self) -> int:
|
||||||
|
"""取一个空闲绑定缓冲(在途任务超限时阻塞等待)"""
|
||||||
|
while True:
|
||||||
|
for i in range(len(self._async_buffers)):
|
||||||
|
if self._async_flags[i][0]:
|
||||||
|
self._async_flags[i][0] = False
|
||||||
|
return i
|
||||||
|
time.sleep(0.0001)
|
||||||
|
|
||||||
|
def _release_async_buffer(self, idx: int):
|
||||||
|
self._async_flags[idx][0] = True
|
||||||
|
|
||||||
def _frame_float(self, image) -> np.ndarray:
|
def _frame_float(self, image) -> np.ndarray:
|
||||||
"""Resize → float32 HWC;归一化 HEF(负零偏)再走 ImageNet 归一化"""
|
"""Resize → float32 HWC;归一化 HEF(负零偏)再走 ImageNet 归一化"""
|
||||||
img = Image.fromarray(image) if isinstance(image, np.ndarray) else image
|
img = Image.fromarray(image) if isinstance(image, np.ndarray) else image
|
||||||
@@ -333,6 +488,43 @@ class HailoPredictor:
|
|||||||
x = (x / 255.0 - IMAGENET_MEAN) / IMAGENET_STD
|
x = (x / 255.0 - IMAGENET_MEAN) / IMAGENET_STD
|
||||||
return x
|
return x
|
||||||
|
|
||||||
|
def _build_frame_luts(self) -> list:
|
||||||
|
"""按帧 quant_info 构建 256 项单通道查表(归一化+量化合并一步)
|
||||||
|
|
||||||
|
OpenCV 5.0 的 cv2.LUT 只接受单通道表,故每帧返回 [R,G,B] 三张表。
|
||||||
|
数学上与 float 路径(_frame_float + quantize)等价(实测 0 LSB)。
|
||||||
|
"""
|
||||||
|
luts = []
|
||||||
|
for info in self._frame_infos:
|
||||||
|
q = info.quant_info
|
||||||
|
c = np.arange(256, dtype=np.float32)[:, None]
|
||||||
|
if self._normalize_frames:
|
||||||
|
v = (c / 255.0 - IMAGENET_MEAN[None, :]) / IMAGENET_STD[None, :]
|
||||||
|
else:
|
||||||
|
v = np.broadcast_to(c, (256, 3))
|
||||||
|
v = v / q.qp_scale + q.qp_zp
|
||||||
|
lut = np.clip(np.round(v), 0, 255).astype(np.uint8) # (256,3)
|
||||||
|
luts.append([lut[:, ch] for ch in range(3)])
|
||||||
|
return luts
|
||||||
|
|
||||||
|
def _frame_lut_quant(self, image, frame_idx: int = 0) -> np.ndarray:
|
||||||
|
"""cv2.LUT 直通:原始 uint8 像素 → 量化 uint8 缓冲(免 float 中间态)
|
||||||
|
|
||||||
|
非 uint8 / 非 3 通道 / 尺寸不匹配输入走与 _frame_float 相同的 PIL
|
||||||
|
转换,保证两条路径语义一致(RGBA 经 convert 合成、灰度转 RGB)。
|
||||||
|
"""
|
||||||
|
x = np.asarray(image)
|
||||||
|
if x.dtype != np.uint8 or x.ndim != 3 or x.shape[2] != 3 or \
|
||||||
|
x.shape[:2] != (self.image_height, self.image_width):
|
||||||
|
img = Image.fromarray(x).convert("RGB")
|
||||||
|
if img.size != (self.image_width, self.image_height):
|
||||||
|
img = img.resize((self.image_width, self.image_height),
|
||||||
|
Image.BILINEAR)
|
||||||
|
x = np.asarray(img, dtype=np.uint8)
|
||||||
|
luts = self._frame_luts[frame_idx]
|
||||||
|
outs = [cv2.LUT(x[..., ch], lut) for ch, lut in enumerate(luts)]
|
||||||
|
return np.ascontiguousarray(cv2.merge(outs))
|
||||||
|
|
||||||
def _bind_frames(self, bindings, frame_float: np.ndarray):
|
def _bind_frames(self, bindings, frame_float: np.ndarray):
|
||||||
"""(F,H,W,3) float 帧 → HEF 帧输入缓冲区(按布局与量化)"""
|
"""(F,H,W,3) float 帧 → HEF 帧输入缓冲区(按布局与量化)"""
|
||||||
from hailo_platform import FormatType
|
from hailo_platform import FormatType
|
||||||
|
|||||||
@@ -4,5 +4,9 @@ torchvision>=0.15,<1.0
|
|||||||
numpy>=1.24,<3.0
|
numpy>=1.24,<3.0
|
||||||
Pillow>=9.0,<14.0 # torchvision 依赖,上限放宽避免与 torch 源安装冲突
|
Pillow>=9.0,<14.0 # torchvision 依赖,上限放宽避免与 torch 源安装冲突
|
||||||
|
|
||||||
|
# 推理(Hailo 部署):帧归一化+量化合并查表(hailo_infer.py cv2.LUT 快路径,
|
||||||
|
# 无 cv2 自动回退 float 路径——本依赖缺失仅损失 3.3× 端到端加速)
|
||||||
|
opencv-python-headless>=4.9,<6.0
|
||||||
|
|
||||||
# Training utilities
|
# Training utilities
|
||||||
tqdm>=4.60,<5.0
|
tqdm>=4.60,<5.0
|
||||||
@@ -1279,6 +1279,13 @@ class _FakeConfigured:
|
|||||||
for b in bindings_list:
|
for b in bindings_list:
|
||||||
self._transform(b)
|
self._transform(b)
|
||||||
|
|
||||||
|
def run_async(self, bindings_list, callback=None):
|
||||||
|
# 模拟异步:同步完成 transform 后触发回调(completion_info 无异常)
|
||||||
|
for b in bindings_list:
|
||||||
|
self._transform(b)
|
||||||
|
if callback is not None:
|
||||||
|
callback(completion_info=types.SimpleNamespace(exception=None))
|
||||||
|
|
||||||
|
|
||||||
class _FakeInferModel:
|
class _FakeInferModel:
|
||||||
def __init__(self, transform):
|
def __init__(self, transform):
|
||||||
@@ -1582,6 +1589,40 @@ class TestHailoPredictor(unittest.TestCase):
|
|||||||
result = p.predict(self._frames(2), history)
|
result = p.predict(self._frames(2), history)
|
||||||
np.testing.assert_allclose(result, [3, 3, 3], atol=1e-6)
|
np.testing.assert_allclose(result, [3, 3, 3], atol=1e-6)
|
||||||
|
|
||||||
|
def test_predict_async_matches_predict(self):
|
||||||
|
from hailo_infer import HailoPredictor
|
||||||
|
infos_in = [
|
||||||
|
_FakeVStreamInfo("net/images", (1, 2, 3, 4, 4), _FakeFormatType.FLOAT32),
|
||||||
|
_FakeVStreamInfo("net/history", (1, 2, 3), _FakeFormatType.FLOAT32),
|
||||||
|
]
|
||||||
|
infos_out = [_FakeVStreamInfo("net/action", (1, 3), _FakeFormatType.FLOAT32)]
|
||||||
|
_install_fake_hailo(self, infos_in, infos_out,
|
||||||
|
_first_row_transform("net/history", "net/action"))
|
||||||
|
history = np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32)
|
||||||
|
with HailoPredictor("fake.hef") as p:
|
||||||
|
sync = p.predict(self._frames(2), history)
|
||||||
|
job = p.predict_async(self._frames(2), history)
|
||||||
|
async_result = job.result(timeout=5.0)
|
||||||
|
np.testing.assert_allclose(async_result, sync, atol=1e-6)
|
||||||
|
|
||||||
|
def test_predict_async_buffer_pool_roundtrip(self):
|
||||||
|
"""连续提交超过缓冲池数量的任务,输出互不串扰"""
|
||||||
|
from hailo_infer import HailoPredictor
|
||||||
|
infos_in = [
|
||||||
|
_FakeVStreamInfo("net/images", (1, 2, 3, 4, 4), _FakeFormatType.FLOAT32),
|
||||||
|
_FakeVStreamInfo("net/history", (1, 2, 3), _FakeFormatType.FLOAT32),
|
||||||
|
]
|
||||||
|
infos_out = [_FakeVStreamInfo("net/action", (1, 3), _FakeFormatType.FLOAT32)]
|
||||||
|
_install_fake_hailo(self, infos_in, infos_out,
|
||||||
|
_first_row_transform("net/history", "net/action"))
|
||||||
|
with HailoPredictor("fake.hef") as p:
|
||||||
|
jobs = []
|
||||||
|
for _ in range(6): # 超过 _ASYNC_BUFFERS=4
|
||||||
|
jobs.append(p.predict_async(self._frames(2), [[1, 2, 3]] * 2))
|
||||||
|
for job in jobs:
|
||||||
|
np.testing.assert_allclose(job.result(timeout=5.0), [1, 2, 3],
|
||||||
|
atol=1e-6)
|
||||||
|
|
||||||
|
|
||||||
class TestDfcCompile(unittest.TestCase):
|
class TestDfcCompile(unittest.TestCase):
|
||||||
"""dfc_compile.py 的 HAR 输入层连通性检查(编译前拦截输入断连)"""
|
"""dfc_compile.py 的 HAR 输入层连通性检查(编译前拦截输入断连)"""
|
||||||
|
|||||||
+188
-15
@@ -17,12 +17,14 @@
|
|||||||
- 🧬 **智能体自我进化**:通过 `update_soul` / `update_identity` 工具动态更新角色定位与用户画像
|
- 🧬 **智能体自我进化**:通过 `update_soul` / `update_identity` 工具动态更新角色定位与用户画像
|
||||||
- 🤖 **子代理**:通过 `create_subagent` 工具派生出拥有独立上下文/角色设定的轻量子代理,主代理可分解长任务、隔离上下文,子代理可选用不同模型
|
- 🤖 **子代理**:通过 `create_subagent` 工具派生出拥有独立上下文/角色设定的轻量子代理,主代理可分解长任务、隔离上下文,子代理可选用不同模型
|
||||||
- 🔄 **技能热重载**:无需重启即可添加、修改或删除扩展技能
|
- 🔄 **技能热重载**:无需重启即可添加、修改或删除扩展技能
|
||||||
- 🔐 **SOCKS5 代理**:通过 `set_proxy` / `get_proxy` 工具为网页访问与 API 调用配置代理,可持久化
|
- 🔐 **SOCKS5 代理**:通过 `set_proxy` / `get_proxy` 工具为网页访问与 API 调用配置代理,可持久化;密码明文仅存内存不落盘,跨重启经 `password_env` 环境变量注入(对齐 `api_key_env` 惯例)
|
||||||
- 💾 **对话管理**:历史对话归档、开始新对话(重置 AI 记忆并重新加载提示词)
|
- 💾 **对话管理**:历史对话归档、开始新对话(重置 AI 记忆并重新加载提示词)
|
||||||
- 📦 **对话压缩**:消息数或 Token 数达到阈值时自动压缩,保持上下文质量
|
- 📦 **对话压缩**:消息数或 Token 数达到阈值时自动压缩,保持上下文质量
|
||||||
- 🛡️ **内存安全防护**:文件读取按需加载不全载、命令输出 1MB 限制、单行超长截断、工具结果多层防护防止 OOM
|
- 🛡️ **内存安全防护**:文件读取按需加载不全载、命令输出 1MB 限制、单行超长截断、工具结果多层防护防止 OOM
|
||||||
- 🔒 **错误自动恢复**:API 请求失败自动回滚消息,工具结果安全清洗(控制字符过滤、非字符串转换)
|
- 🔒 **错误自动恢复**:API 请求失败自动回滚消息,工具结果安全清洗(控制字符过滤、非字符串转换)
|
||||||
- 🛑 **可靠中断机制**:支持 Ctrl+C 实时中断,通过 `threading.Event` + HTTP 连接关闭 + 子进程 kill 三层取消,流式响应和长时间运行命令均可立即终止
|
- 🛑 **可靠中断机制**:支持 Ctrl+C 实时中断,通过 `threading.Event` + HTTP 连接关闭 + 子进程 kill 三层取消,流式响应和长时间运行命令均可立即终止
|
||||||
|
- 💻 **CLI 一次性对话**:`--once "问题"` 不启动 WebSocket 直接执行并退出,stdout/stderr 分流可管道组合,支持 `--quiet` / `--show-reasoning` / `--worker-only` / `--model` / `--temperature` / `--once-timeout`
|
||||||
|
- 🔀 **多模型档案**:`config.json` 的 `models` 节可配置多个 OpenAI Chat Completions 兼容端点(DeepSeek/Kimi/Qwen/本地 Ollama 等),按名称切换,档案级 api_url/model/api_key_env/reasoning_enabled/timeout/temperature/display_name;`--model` 支持 `model:`/`profile:` 前缀逃逸
|
||||||
|
|
||||||
## 🚀 快速开始
|
## 🚀 快速开始
|
||||||
|
|
||||||
@@ -55,13 +57,28 @@ python aibox.py
|
|||||||
|
|
||||||
启动后,WebSocket 服务器将运行在 `ws://localhost:8765`(地址与端口可在 `config.json` 的 `system.websocket_host` / `system.websocket_port` 中修改)。
|
启动后,WebSocket 服务器将运行在 `ws://localhost:8765`(地址与端口可在 `config.json` 的 `system.websocket_host` / `system.websocket_port` 中修改)。
|
||||||
|
|
||||||
> 注意:本模块入口文件为 `aibox.py`(当前约 3018 行,集成 WebSocket 服务器、AI 引擎、工具/技能系统、任务调度器于一体;自 v3.0 起工具类与子系统已抽离至 `tools/`、`subsystems/`、`common/`、`tests/` 包,aibox.py 仅保留核心类与编排逻辑)。`client.py` 为示例 WebSocket 客户端,`voice_client.py` 为语音客户端,`llm_client.py` 为多厂商大模型客户端抽象层,`_run.sh` 为被 `core` 拉起的启动脚本。
|
### 管理大模型接入点(不用手写配置)
|
||||||
|
|
||||||
|
不想手写 `config.json` 的 `models` 节?用交互式菜单工具:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python model_config.py # 交互式菜单:列出 / 添加 / 删除 / 连通性验证
|
||||||
|
python model_config.py --list # 非交互:列出当前接入点
|
||||||
|
```
|
||||||
|
|
||||||
|
- **添加接入点**:按提示填写档案名、API 地址(附常见厂商端点参考)、模型名;Key 两种方式——环境变量名(推荐,Key 不落盘)或免密本地端点(Ollama 等);可选显示名/温度/超时/思考模式/失败转移(从已有接入点多选)/设为默认
|
||||||
|
- **删除接入点**:编号选择 + 确认,自动清理其他档案的失败转移引用
|
||||||
|
- **连通性验证**:对选中接入点发一次最小请求,直接给出成败原因
|
||||||
|
- 添加后可立即使用:`python aibox.py --model <档案名>` 指定,或设为默认后无需 `--model`
|
||||||
|
|
||||||
|
> 注意:本模块入口文件为 `aibox.py`(当前约 3408 行,集成 WebSocket 服务器、AI 引擎、工具/技能系统、任务调度器于一体;自 v3.0 起工具类与子系统已抽离至 `tools/`、`subsystems/`、`common/`、`tests/` 包,aibox.py 仅保留核心类与编排逻辑)。`client.py` 为示例 WebSocket 客户端,`voice_client.py` 为语音客户端,`llm_client.py` 为多厂商大模型客户端抽象层,`_run.sh` 为被 `core` 拉起的启动脚本。
|
||||||
|
|
||||||
## 📁 目录结构
|
## 📁 目录结构
|
||||||
|
|
||||||
```
|
```
|
||||||
aibox/
|
aibox/
|
||||||
├── aibox.py # 服务主程序(WebSocket 服务器 + AI 引擎 + 任务调度器 + 核心类 DeepSeekChat/WebSocketHandler/EnhancedAIManager)
|
├── aibox.py # 服务主程序(WebSocket 服务器 + AI 引擎 + 任务调度器 + 核心类 DeepSeekChat/WebSocketHandler/EnhancedAIManager)
|
||||||
|
├── model_config.py # 大模型接入点交互式管理工具(列表/添加/删除/连通性验证,不用手写配置)
|
||||||
├── client.py # 示例 WebSocket 客户端
|
├── client.py # 示例 WebSocket 客户端
|
||||||
├── voice_client.py # 语音客户端
|
├── voice_client.py # 语音客户端
|
||||||
├── llm_client.py # 大模型客户端抽象层(多厂商切换,DeepSeekClient 兼容 OpenAI 协议)
|
├── llm_client.py # 大模型客户端抽象层(多厂商切换,DeepSeekClient 兼容 OpenAI 协议)
|
||||||
@@ -90,7 +107,7 @@ aibox/
|
|||||||
│ ├── prompt_tools.py # B 批:UpdateIdentity/UpdateSoul/SaveMemoryAndEndConversation
|
│ ├── prompt_tools.py # B 批:UpdateIdentity/UpdateSoul/SaveMemoryAndEndConversation
|
||||||
│ ├── proxy_tools.py # B 批:SetProxy/GetProxy
|
│ ├── proxy_tools.py # B 批:SetProxy/GetProxy
|
||||||
│ └── misc_tools.py # B 批:AskUserTool
|
│ └── misc_tools.py # B 批:AskUserTool
|
||||||
├── tests/ # 单元测试套件(232 项全过)
|
├── tests/ # 单元测试套件(441 项全过)
|
||||||
│ ├── conftest.py # sys.path 配置 + websockets 桩
|
│ ├── conftest.py # sys.path 配置 + websockets 桩
|
||||||
│ ├── test_basic_tools.py # A 批工具 43 项
|
│ ├── test_basic_tools.py # A 批工具 43 项
|
||||||
│ ├── test_b_tools.py # B 批工具 58 项
|
│ ├── test_b_tools.py # B 批工具 58 项
|
||||||
@@ -99,7 +116,15 @@ aibox/
|
|||||||
│ ├── test_db.py # 记忆/备忘录数据库(含中文检索/迁移)
|
│ ├── test_db.py # 记忆/备忘录数据库(含中文检索/迁移)
|
||||||
│ ├── test_conversation_history.py # 对话历史
|
│ ├── test_conversation_history.py # 对话历史
|
||||||
│ ├── test_conversation_compressor.py # 对话压缩器
|
│ ├── test_conversation_compressor.py # 对话压缩器
|
||||||
│ └── test_skill_manager.py # 技能管理器
|
│ ├── test_skill_manager.py # 技能管理器
|
||||||
|
│ ├── test_once_mode.py # CLI 一次性对话模式(Batch A/B/C 15 项)
|
||||||
|
│ ├── test_models_profile.py # 多模型档案(Batch D 17 项)
|
||||||
|
│ ├── test_config.py # 配置脱敏(F1 3 项)
|
||||||
|
│ ├── test_f2_fixes.py # 超时清理/线程收尾(F2 5 项)
|
||||||
|
│ ├── test_g_fixes.py # 第三轮审计 Batch G(20 项)
|
||||||
|
│ ├── test_h_fixes.py # 第四轮审计 Batch H(26 项)
|
||||||
|
│ ├── test_i_fixes.py # 遗留收尾 Batch I(16 项)
|
||||||
|
│ └── test_model_config.py # 模型接入点管理工具(17 项)
|
||||||
├── skills/ # 内置扩展技能(*.py 可执行脚本 / *.md Markdown 技能)
|
├── skills/ # 内置扩展技能(*.py 可执行脚本 / *.md Markdown 技能)
|
||||||
│ ├── hello-world.py
|
│ ├── hello-world.py
|
||||||
│ ├── aes-crypto.py
|
│ ├── aes-crypto.py
|
||||||
@@ -134,7 +159,7 @@ common ← subsystems ← tools ← aibox.py
|
|||||||
```bash
|
```bash
|
||||||
# 从仓库根目录运行
|
# 从仓库根目录运行
|
||||||
py -m pytest aibox/tests -q
|
py -m pytest aibox/tests -q
|
||||||
# 当前:232 项全过,约 11s
|
# 当前:264 项全过
|
||||||
```
|
```
|
||||||
|
|
||||||
运行期数据存放于 `~/.aibox/`:
|
运行期数据存放于 `~/.aibox/`:
|
||||||
@@ -186,7 +211,7 @@ py -m pytest aibox/tests -q
|
|||||||
| `memory` | 保存、搜索(支持分页)、删除记忆 |
|
| `memory` | 保存、搜索(支持分页)、删除记忆 |
|
||||||
| `memo` | 创建、查看、完成、搜索任务 |
|
| `memo` | 创建、查看、完成、搜索任务 |
|
||||||
| `express_emotion` | 表达情绪并记录当前情绪状态(anger/joy/sadness/surprise),可指定 `persona` 从 `~/.aibox/skills/` 加载人格技能(如 angry/coquettish/tsundere)派生携带该人格的情绪子代理,保证人格独立、不污染主对话上下文 |
|
| `express_emotion` | 表达情绪并记录当前情绪状态(anger/joy/sadness/surprise),可指定 `persona` 从 `~/.aibox/skills/` 加载人格技能(如 angry/coquettish/tsundere)派生携带该人格的情绪子代理,保证人格独立、不污染主对话上下文 |
|
||||||
| `create_subagent` | 派生子代理:独立上下文/角色设定,仅拥有执行面工具(无记忆/人格/派生/提问能力),可选独立模型,用于长任务分解、上下文隔离、独立角色专注处理 |
|
| `create_subagent` | 派生子代理:独立上下文/角色设定,仅拥有执行面工具(无记忆/人格/派生/提问能力),可选独立模型(工具描述动态列出 `models` 节可用档案名,防 LLM 编造名称),用于长任务分解、上下文隔离、独立角色专注处理 |
|
||||||
| `search_skills` | 搜索扩展技能(支持 Markdown 和可执行技能),查到后派生子代理执行 |
|
| `search_skills` | 搜索扩展技能(支持 Markdown 和可执行技能),查到后派生子代理执行 |
|
||||||
| `reload_skills` | 热重载扩展技能 |
|
| `reload_skills` | 热重载扩展技能 |
|
||||||
| `create_skill` | 创建新技能(AI 自主编写 Python 脚本,实际写入经子代理完成) |
|
| `create_skill` | 创建新技能(AI 自主编写 Python 脚本,实际写入经子代理完成) |
|
||||||
@@ -196,7 +221,7 @@ py -m pytest aibox/tests -q
|
|||||||
| `update_soul` | 更新 AI 自我定义(`99_SOUL.md`,智能体进化) |
|
| `update_soul` | 更新 AI 自我定义(`99_SOUL.md`,智能体进化) |
|
||||||
| `save_memory_and_end_conversation` | 保存记忆并结束对话 |
|
| `save_memory_and_end_conversation` | 保存记忆并结束对话 |
|
||||||
| `get_current_time` | 获取当前时间 |
|
| `get_current_time` | 获取当前时间 |
|
||||||
| `set_proxy` | 设置 SOCKS5 代理(host/port/可选账号密码,可持久化,即时生效) |
|
| `set_proxy` | 设置 SOCKS5 代理(host/port/可选账号密码,可持久化,即时生效);`password` 明文仅本次会话有效不落盘,跨重启用 `password_env` 指定存有密码的环境变量名 |
|
||||||
| `get_proxy` | 查看当前 SOCKS5 代理配置 |
|
| `get_proxy` | 查看当前 SOCKS5 代理配置 |
|
||||||
|
|
||||||
### 子代理(worker)工具集
|
### 子代理(worker)工具集
|
||||||
@@ -502,15 +527,69 @@ AI 可以通过 `update_soul` 工具动态更新 `99_SOUL.md`,通过 `update_i
|
|||||||
"broadcast_enabled": true
|
"broadcast_enabled": true
|
||||||
},
|
},
|
||||||
"api": {
|
"api": {
|
||||||
"provider": "deepseek",
|
"provider": "openai_compatible",
|
||||||
"deepseek_api_url": "https://api.deepseek.com/chat/completions",
|
"deepseek_api_url": "https://api.deepseek.com/chat/completions",
|
||||||
"deepseek_model": "deepseek-v4-flash",
|
"deepseek_model": "deepseek-v4-flash",
|
||||||
"reasoning_enabled": false,
|
"reasoning_enabled": false,
|
||||||
"api_key_env": "DEEPSEEK_API_KEY",
|
"api_key_env": "DEEPSEEK_API_KEY",
|
||||||
"timeout": 120,
|
"timeout": 120,
|
||||||
"temperature": 0.7
|
"temperature": 0.7,
|
||||||
|
"default_model": "deepseek"
|
||||||
},
|
},
|
||||||
"context": {
|
"models": {},
|
||||||
|
```
|
||||||
|
|
||||||
|
### 🔀 多模型档案(models 节)
|
||||||
|
|
||||||
|
`models` 可配置多个 OpenAI Chat Completions 协议兼容端点,按**档案名**切换(v3.2):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"models": {
|
||||||
|
"kimi": {
|
||||||
|
"api_url": "https://api.moonshot.cn/v1/chat/completions",
|
||||||
|
"model": "kimi-k2",
|
||||||
|
"api_key_env": "MOONSHOT_API_KEY",
|
||||||
|
"reasoning_enabled": false,
|
||||||
|
"timeout": 120
|
||||||
|
},
|
||||||
|
"local": {
|
||||||
|
"api_url": "http://localhost:11434/v1/chat/completions",
|
||||||
|
"model": "qwen3:32b",
|
||||||
|
"api_key_env": ""
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"api": { "default_model": "kimi" }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**档案字段**:
|
||||||
|
|
||||||
|
| 字段 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| `api_url` | **必填**。端点 URL(OpenAI Chat Completions 协议) |
|
||||||
|
| `model` | **必填**(除非调用时传裸模型名)。该档案使用的模型名 |
|
||||||
|
| `api_key_env` | API Key 环境变量名;**未配置回退 `api.api_key_env`;显式空串 `""` 为本地免密端点**(Ollama/vLLM)。档案端点环境变量取不到时**直接报错,绝不回退 `~/.cert/deepseek.key`**——该文件为 DeepSeek 专属凭据,跨厂商复用属于泄漏(key 文件回退仅对无档案的旧 `api` 扁平节生效) |
|
||||||
|
| `reasoning_enabled` | 档案级思考模式开关(缺省回退 `api.reasoning_enabled`) |
|
||||||
|
| `timeout` | 请求超时秒数 |
|
||||||
|
| `temperature` | 档案级默认采样温度(可选)。优先级:CLI/调用方显式传入 > 本字段 > 全局默认;reasoning 模型建议按厂商文档设置 |
|
||||||
|
| `display_name` | 展示名(可选,缺省用档案名)。`client.name` 用于对话上下文中的 assistant 名,不再恒显示 "DeepSeek" |
|
||||||
|
| `fallback` | 失败转移档案名列表(可选,如 `["kimi", "local"]`,一层不递归)。主档案瞬时故障(HTTP 429/5xx、连接失败/超时)时按序转移;认证/参数类 4xx **不转移**(配置问题响亮暴露);失败档案进入 60s 冷却期(429 响应带 Retry-After 数字秒时按其延长)。`chat_completion`(对话压缩/轻量子代理)无条件转移;`stream_chat`(主对话流式)仅**首事件前**转移(已输出内容/用户中断绝不转移) |
|
||||||
|
| `failover_timeout_budget` | 整条转移链的总时间预算秒数(可选,缺省不限)。耗尽即停止尝试剩余档案,防止长链故障时超时叠加(3 档案最坏 3×timeout);`FailoverClient.get_stats()` 暴露 requests/failovers/cooldown_skips/exhausted 计数,`on_failover(from, to, reason)` 回调供管理面/前端通知接入 |
|
||||||
|
|
||||||
|
**选择优先级**:`--model <档案名>`(命中 `models` 节)→ `api.default_model` 指向档案 → 回退旧 `api` 扁平节。`--model` 为**进程级覆盖**:WebSocket 服务模式(主对话、每个会话实例、定时任务实例)与 `--once` 一次性模式均生效。
|
||||||
|
|
||||||
|
`--model` 传**裸模型名**(未命中档案)时,沿用默认端点与 Key,仅覆盖模型名——即 "同端点换模型" 语义。
|
||||||
|
|
||||||
|
**前缀逃逸**(档案名与裸模型名歧义时):`--model model:NAME` 强制按裸模型名解析(档案恰好叫 NAME 时仍走裸名语义);`--model profile:NAME` 强制按档案名解析(不存在则明确报错,不再静默落回裸名语义)。
|
||||||
|
|
||||||
|
**兼容性**:`models` 留空 `{}` 时行为与旧版完全一致(走 `api` 节),已有配置无需任何改动。
|
||||||
|
|
||||||
|
其余配置节:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"context": {
|
||||||
"max_history_messages": 500,
|
"max_history_messages": 500,
|
||||||
"max_context_messages": 120,
|
"max_context_messages": 120,
|
||||||
"max_tokens": 8192,
|
"max_tokens": 8192,
|
||||||
@@ -562,11 +641,14 @@ AI 可以通过 `update_soul` 工具动态更新 `99_SOUL.md`,通过 `update_i
|
|||||||
"llm_max_tokens": 800
|
"llm_max_tokens": 800
|
||||||
},
|
},
|
||||||
"proxy": {
|
"proxy": {
|
||||||
"type": "",
|
"socks5": {
|
||||||
"host": "",
|
"enabled": false,
|
||||||
"port": 0,
|
"host": "127.0.0.1",
|
||||||
"username": "",
|
"port": 1080,
|
||||||
"password": ""
|
"username": "",
|
||||||
|
"password": null,
|
||||||
|
"password_env": "SOCKS5_PASSWORD"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
@@ -574,6 +656,8 @@ AI 可以通过 `update_soul` 工具动态更新 `99_SOUL.md`,通过 `update_i
|
|||||||
> 默认模型为 `deepseek-v4-flash`(通过 `api.deepseek_model` 修改)。设置 `api.reasoning_enabled: true` 可启用思考模式,此时会在请求中添加 `thinking: {type: "enabled"}` 参数,模型会先进行思考再回答,实时展示思考过程(灰色斜体区域),回答质量更高且**支持工具调用**(文件操作、命令执行、记忆搜索等工具均可在思考模式下正常使用)。
|
> 默认模型为 `deepseek-v4-flash`(通过 `api.deepseek_model` 修改)。设置 `api.reasoning_enabled: true` 可启用思考模式,此时会在请求中添加 `thinking: {type: "enabled"}` 参数,模型会先进行思考再回答,实时展示思考过程(灰色斜体区域),回答质量更高且**支持工具调用**(文件操作、命令执行、记忆搜索等工具均可在思考模式下正常使用)。
|
||||||
>
|
>
|
||||||
> API Key 从环境变量 `DEEPSEEK_API_KEY` 读取(亦支持 `~/.cert/deepseek.key` 文件)。
|
> API Key 从环境变量 `DEEPSEEK_API_KEY` 读取(亦支持 `~/.cert/deepseek.key` 文件)。
|
||||||
|
>
|
||||||
|
> **代理密码不落盘**:`proxy.socks5.password` 仅在运行期内存中有效,保存配置时强制置空;跨重启持久化请将密码存入环境变量(如 `SOCKS5_PASSWORD`),配置 `password_env` 指向该变量名。加载时检测到旧版明文密码会自动迁移(本会话保留可用,文件立即重写脱敏)。
|
||||||
>
|
>
|
||||||
> **多厂商切换**:大模型调用已通过 `aibox/llm_client.py` 抽象层解耦,业务层不再直接依赖具体厂商。配置 `api.provider`(默认 `deepseek`)即可切换厂商,当前内置 `DeepSeekClient`(兼容 OpenAI Chat Completions 协议,Kimi / Qwen / GLM / Doubao 等绝大多数厂商均可复用)。新增厂商只需在此文件新增一个 `LLMClient` 子类并在 `create_llm_client` 工厂中注册,无需改动业务代码,为后续子代理/多模型共发预留了扩展点。
|
> **多厂商切换**:大模型调用已通过 `aibox/llm_client.py` 抽象层解耦,业务层不再直接依赖具体厂商。配置 `api.provider`(默认 `deepseek`)即可切换厂商,当前内置 `DeepSeekClient`(兼容 OpenAI Chat Completions 协议,Kimi / Qwen / GLM / Doubao 等绝大多数厂商均可复用)。新增厂商只需在此文件新增一个 `LLMClient` 子类并在 `create_llm_client` 工厂中注册,无需改动业务代码,为后续子代理/多模型共发预留了扩展点。
|
||||||
|
|
||||||
@@ -595,6 +679,14 @@ python aibox.py [选项]
|
|||||||
--memo-add "标题|内容|时间" 添加任务
|
--memo-add "标题|内容|时间" 添加任务
|
||||||
--memo-list 列出任务
|
--memo-list 列出任务
|
||||||
--memo-complete ID 完成任务
|
--memo-complete ID 完成任务
|
||||||
|
--once QUERY 一次性对话:执行 QUERY 后退出(不启动 WebSocket)
|
||||||
|
--quiet 一次性模式静默输出(stdout 仅回答正文)
|
||||||
|
--show-reasoning 一次性模式输出思考流(默认静默,走 stderr)
|
||||||
|
--no-save-memory 一次性模式禁用记忆工具(干净跳过,不进请求 schema)
|
||||||
|
--worker-only 一次性模式纯执行面(worker profile,无人格/记忆工具)
|
||||||
|
--model NAME 模型档案名(models 节)或裸模型名(同端点换模型);歧义时 model:NAME 强制裸名 / profile:NAME 强制档案;服务模式与 --once 均生效(进程级覆盖)
|
||||||
|
--temperature T 覆盖配置的对话温度
|
||||||
|
--once-timeout SEC 一次性模式全局超时(默认 300 秒)
|
||||||
```
|
```
|
||||||
|
|
||||||
### 使用示例
|
### 使用示例
|
||||||
@@ -616,6 +708,30 @@ python aibox.py --memo-add "提醒开会|下午3点会议室A|+1h"
|
|||||||
python aibox.py --memo-list
|
python aibox.py --memo-list
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 💻 CLI 一次性对话(--once)
|
||||||
|
|
||||||
|
不启动 WebSocket,单次执行后退出。回答正文走 **stdout**,工具过程/思考流走 **stderr**,可直接管道组合:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 基础问答
|
||||||
|
python aibox.py --once "帮我看下当前日期"
|
||||||
|
|
||||||
|
# 管道组合(--quiet 静默模式)
|
||||||
|
python aibox.py --once --quiet "列出当前目录 py 文件" | grep ".py" | wc -l
|
||||||
|
|
||||||
|
# 按档案名选模型 / 同端点换模型
|
||||||
|
python aibox.py --once --model kimi "总结这段代码"
|
||||||
|
python aibox.py --once --model deepseek-v4-lite "1+1?"
|
||||||
|
|
||||||
|
# 纯执行面(无人格/记忆,单次收敛,适合脚本调用)
|
||||||
|
python aibox.py --once --worker-only "执行 echo hello 并返回输出"
|
||||||
|
|
||||||
|
# 显式开启思考流 / 禁用记忆写入
|
||||||
|
python aibox.py --once --show-reasoning --no-save-memory "分析这个问题"
|
||||||
|
```
|
||||||
|
|
||||||
|
**退出码**:`0` 成功 / `1` 工具执行失败 / `2` API 调用失败 / `3` 用户中断(Ctrl+C)/ `4` 全局超时。
|
||||||
|
|
||||||
## 🔄 架构流程
|
## 🔄 架构流程
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -795,6 +911,63 @@ elif command == "/mycommand":
|
|||||||
|
|
||||||
## 📄 更新日志
|
## 📄 更新日志
|
||||||
|
|
||||||
|
### v3.5
|
||||||
|
- ✅ **安全审计与修复(第三轮 Batch G + 第四轮 Batch H + 遗留收尾 Batch I)**(完整记录见 [docs/reports/aibox_audit_20260817.md](../docs/reports/aibox_audit_20260817.md)):
|
||||||
|
- **Batch G(llm 加固 + 线程生命周期 + 取消传播 + DB 事务)**:
|
||||||
|
- llm_client:非瞬时 HTTP 错误体截断 `[:200]`(密钥/错误页不进异常栈)、Retry-After 上限 cap 3600s、fallback 档案缺 model 跳过并告警、fallback 去重、删除 `LLMClient._cancelled` 死状态、FailoverClient 并发契约文档化
|
||||||
|
- 子代理取消传播:`run_full_subagent` 共享父 `_cancel_event` + 0.2s 观察者线程转发深度中断(强关在途 HTTP、kill 子进程);`SubAgentTool` 构造注入父事件
|
||||||
|
- drain 线程重构:批次级单线程 + stop 事件 + try/finally 统一回收(消除每次工具调用泄漏一个常驻线程);`/new` 的 `reset_conversation` 移入线程池(不再同步获取 `_msg_lock` 冻结事件循环);broadcast 收窄 except + unregister 改 discard;`_processing_tasks` 改按连接的任务集合登记(关闭时取消全部在途任务)
|
||||||
|
- DB:FTS5 关键词双引号翻倍转义(`"` → `""`)、两处 `LIMIT` f-string 参数化、`mark_as_triggered` 读写收进单连接 `BEGIN IMMEDIATE` 事务(消灭并发触发竞态);历史索引裸 except 收窄 + 归档文件名同秒加序号
|
||||||
|
- **Batch H(第四轮审计,3 路并行审读 + Linux PoC 6/6 实证)**:
|
||||||
|
- **H1(高危)**:修复 N5 回归——子代理共享父取消事件后首轮 think 无条件 `clear()` 会吞掉父级已置位的取消信号(用户中断 / --once 超时被绕过),共享事件时跳过 clear + 父已取消直接早退
|
||||||
|
- `/reload_prompts` 移入线程池(N3 同类遗漏);技能扫描(串行 `--help` 子进程)启动前预热,不再在首条消息/首个任务时冻结事件循环
|
||||||
|
- 调度器任务级超时(`scheduler.task_timeout`,默认 600s)——AI think 卡死不再停摆整个调度循环
|
||||||
|
- `get_recent_memories(limit=None)` 默认上限(不再全表加载);`memo delete` 改物理删除(此前只标记完成);`PromptManager.reload` 本地构建后原子替换(并发读者不再读到空/半截系统提示词)
|
||||||
|
- read_file 的 sed 参数强转 int(防引号注入);`set_proxy` 落盘失败如实报告;SSE 非 UTF-8 行容错;persona 路径 realpath 校验(软链越界拒绝);1MB 截断无半个字符;compressor 除零防护;InstallSkillTool 指引命令路径引用;ask_user 缺参明确报错
|
||||||
|
- **POSIX 进程组终止**:executor 的 `Popen` 加 `start_new_session` + 组信号 kill(组 ID 在 Popen 后立即捕获)——`(sleep 300 &)` 形态的孙进程持管道不再悬挂 reader 线程/泄漏 fd(实证修复前孙进程残留)
|
||||||
|
- web 工具 scheme 白名单(http/https,`file://` 等拒绝);registry 日志打码改包含匹配(apiKey/passwd 变体);FailoverClient 日志 Bearer 令牌正则脱敏(网关回显 Authorization 头不再进日志)
|
||||||
|
- **Batch I(遗留收尾)**:
|
||||||
|
- **N9**:prompt 写入内容长度上限(`MAX_CONTENT_LENGTH=10000`,identity/soul/status 超长拒绝)——约束持久化注入面
|
||||||
|
- **F5**:情绪子代理取消传播——`run_light_subagent` 改走 stream_chat + 观察者线程,用户中断可即时终止(此前阻塞至请求超时)
|
||||||
|
- 提醒时间格式异常任务按"到期"返回(此前被 SQL 层字符串比较静默丢弃,任务永不执行);`mark_as_triggered` 加 `busy_timeout=10000`(多进程同库写竞争等待而非立即报 locked)
|
||||||
|
- 对话文件原子写(tmp + os.replace,崩溃不留半截 JSON);`load_conversation` realpath 越界防护;WS 命令专用线程池(`/new`、`/reload_prompts` 不再占用默认池);--once 硬超时兜底(软超时后宽限一个周期强杀进程)
|
||||||
|
- PromptManager 单例加锁;`limit=0` 语义修正(None 才不加 LIMIT)
|
||||||
|
- ✅ **测试**:424 项全过(Linux 环境;Batch G 20 + Batch H 26 + Batch I 16 项新增;`test_execute_timeout`/`test_flat_api_still_reads_key_file` 为 Windows 向用例,Linux 预存失败与本轮无关)
|
||||||
|
|
||||||
|
### v3.4
|
||||||
|
- ✅ **多模型档案加固(H1/H2/H3/M1/M2/M3 六项修复)**:
|
||||||
|
- **H1(凭据隔离)**:档案端点环境变量取不到时直接报错,**绝不回退读 `~/.cert/deepseek.key`**(该文件为 DeepSeek 专属凭据,跨厂商复用属于泄漏);key 文件回退仅对无档案的旧 `api` 扁平节生效
|
||||||
|
- **H2(schema 校验前移)**:档案缺 `api_url`/`model` 或类型错误时,解析期即抛出指明档案名与缺失字段的 `ValueError`——不再把档案名当模型名发出、不再延迟到 `requests InvalidURL` 或 API 400
|
||||||
|
- **H3(--model 服务模式生效)**:`--model` 升级为进程级覆盖,WebSocket 服务模式(主对话/每个会话实例/定时任务实例)与 `--once` 均生效,此前服务模式静默忽略
|
||||||
|
- **M2(防模型名幻觉)**:`create_subagent` 工具描述与 JSON Schema 动态携带 `models` 节可用档案名列表(默认档案标注),LLM 无从编造名称;`models` 为空时描述保持原样(零破坏)
|
||||||
|
- **M1(换模型同步派生引用)**:新增 `DeepSeekChat.set_client()` 统一替换 LLM 客户端——同步刷新 `compressor.llm_client` 与 `api_key`/`name` 快照;`create_subagent` 子代理换模型经此入口,压缩链不再残留默认模型
|
||||||
|
- **M3(reasoning 契约显式化)**:档案级 `reasoning_enabled` 从"构造后挂实例属性"改为 `LLMClient` 构造参数(三态:`True`/`False` 档案级显式、`None` 未配置回退全局),外部误设属性无从静默覆盖全局开关
|
||||||
|
- ✅ **L 批体验与兼容修复(L1/L2/L3/L5 四项)**:
|
||||||
|
- **L1(展示名按档案)**:`client.name` = 档案 `display_name` > 档案名,用 Kimi/Qwen 时对话上下文不再误标 "DeepSeek";旧扁平节保持 "DeepSeek"(零破坏)
|
||||||
|
- **L2(前缀逃逸)**:`--model model:NAME` 强制裸模型名语义、`--model profile:NAME` 强制档案语义(不存在时明确报错并列出可用档案),档案名与模型名歧义可逃逸
|
||||||
|
- **L3(档案级温度)**:档案 `temperature` 字段作为默认采样温度,优先级:CLI/调用方显式传入 > 档案字段 > 全局默认(0.7/0.3),主对话/轻量子代理/compressor 全链路生效
|
||||||
|
- **L5(思考流字段别名)**:`_parse_delta` 识别 `reasoning_content`/`reasoning`/`thinking`/`thinking_content` 四种常见思考字段(首个命中生效),其他厂商思考流不再静默丢失
|
||||||
|
- ✅ **L4a(chat_completion 跨档案失败转移)**:档案新字段 `fallback`(档案名列表,一层不递归);`create_llm_client` 构造 `FailoverClient` 包装链(主档案 + fallback)。瞬时故障(HTTP 429/5xx 抛 `TransientLLMError`、连接失败/超时)按链转移,认证/参数类 4xx 不转移(配置问题响亮暴露);失败档案 60s 冷却;fallback 档案缺 Key 等环境问题跳过 + warning 不阻断主档案;配置结构错误(类型/不存在/自引用)启动期硬性拦截。当前覆盖对话压缩与轻量子代理链路,流式(`stream_chat`)转移属 L4b
|
||||||
|
- ✅ **L4b(stream_chat 首事件前转移 + 中断联动)**:流式仅在**未产出任何事件**(reasoning/content/tool_call)时转移——重放对用户不可见;已产出事件后失败直接上抛(防前端重复输出/工具增量错乱)。`LLMClient.cancel()` 置 `_cancelled` 标志,`FailoverClient.cancel()` 广播整条链并置包装器标志——用户中断引发的连接异常与真实故障不可区分,凭标志跳过转移(否则用户越中断、系统越换厂商重试);`chat_completion` 同受标志保护。链耗尽抛最后异常(既有回滚/报错处理接住),全链冷却抛 RuntimeError
|
||||||
|
- ✅ **L4c(转移可观测性 + 预算控制 + Retry-After)**:`FailoverClient.get_stats()` 暴露 requests/failovers/cooldown_skips/exhausted 聚合计数;`on_failover(from, to, reason)` 回调(to=None 表示链耗尽,回调异常不影响主流程)供管理面/前端通知接入;档案字段 `failover_timeout_budget` 限制整链尝试总秒数(缺省不限,<=0 视为未配置),耗尽停止尝试剩余档案防超时叠加;429 响应 Retry-After(数字秒)经 `TransientLLMError.retry_after` 延长该档案冷却期(非数字格式忽略)
|
||||||
|
- ✅ **测试**:355 项全过(H1 隔离 4 + H2 校验 6 + H3 传导 6 + M2 提示 5 + M1 替换 2 + M3 契约 4 + L 批 16 + L4a 转移 15 + L4b 流式 12 + L4c 可观测 13 项新增)
|
||||||
|
|
||||||
|
### v3.3
|
||||||
|
- ✅ **安全审计与修复**(两轮审计报告见 [docs/reports/aibox_audit_20260817.md](../docs/reports/aibox_audit_20260817.md)):
|
||||||
|
- **Batch E(命令执行保护链修复)**:`CommandLineTool.execute_streaming` 纯委托注入的 executor(删除绕过 1MB/超时/中断保护的 150 行重复 Popen 实现);`SubprocessCommandExecutor` 读取循环线程化(后台 reader 线程 + 0.1s 超时轮询,静默命令超时不再失效);两处 `Popen` 改 `stdin=DEVNULL`(等待输入的命令不再永久挂起工具线程);write_file 指引 heredoc 定界符唯一化;registry 流式转发循环加 3600s 总超时兜底。PoC 回归 5/5 + 全量单测通过
|
||||||
|
- **Batch F(凭据保护 + 线程收尾)**:
|
||||||
|
- **F1**:SOCKS5 代理密码不再明文落盘——`_save_config` 落盘前强制脱敏(纵深防御),新增 `password_env` 环境变量注入(对齐 `api_key_env` 惯例),旧版明文配置加载时自动迁移清除;`registry` debug 日志对 `password`/`api_key`/`token`/`secret`/`authorization` 参数打码
|
||||||
|
- **F2**:registry 流式转发超时路径统一清理 `tool.streaming_output`/`tool.output_queue`(消除僵尸 daemon 线程经实例属性把输出混入后续会话队列的串流路径),`stream_tool_timeout` 提为实例属性(默认 3600s,测试可调);`SubprocessCommandExecutor` reader 线程收尾——正常 EOF 后显式 close 释放管道 fd,仍阻塞时有限 join 后放弃(daemon 兜底);实证修正:CPython 流锁下阻塞中的 readline 无法被 `close()` 中断(首版实现致超时被拖满命令时长,已按 PoC 回归修正)
|
||||||
|
- ✅ **测试**:272 项全过(Batch E PoC 回归 + F1 配置脱敏 3 项 + F2 超时清理/线程收尾 5 项新增)
|
||||||
|
|
||||||
|
### v3.2
|
||||||
|
- ✅ **CLI 一次性对话模式(`--once`)**(分三批落地,开发文档见 [docs/reports/aibox_cli_once_mode_dev_doc.md](../docs/reports/aibox_cli_once_mode_dev_doc.md)):
|
||||||
|
- **Batch A**:`--once QUERY` / `--quiet` / `--once-timeout`;不启动 WebSocket,执行后退出;回答走 stdout、过程走 stderr,可管道组合;SIGINT 与超时复用三层取消机制
|
||||||
|
- **Batch B**:`--no-save-memory`(`enable_memory` 开关干净禁用 MemoryTool)、`--show-reasoning`(思考流默认静默);压缩提示改走 output_callback 不再污染 stdout;退出码 1(工具执行失败)
|
||||||
|
- **Batch C**:SQLite WAL(多 CLI 进程并发写锁缓解)、`--worker-only`(纯执行面 + 精简 system prompt 防人格污染)、`--model` / `--temperature` 覆盖、子代理工具失败标记透传(`_SUBAGENT_TOOL_FAILED_MARKER` → orchestrator stats)
|
||||||
|
- ✅ **多模型档案(`models` 配置节)**:配置多个 OpenAI Chat Completions 兼容端点(DeepSeek/Kimi/Qwen/本地 Ollama 等)按档案名切换;`--model` 支持档案名与裸模型名(同端点换模型)双语义;档案级 `api_key_env` 三级语义(回退 / 显式 / 空串免密);档案级 `reasoning_enabled` 优先于全局;`models: {}` 时零破坏回退旧 `api` 节;`llm_client.py` 增加 `openai_compatible` / `openai` provider 别名
|
||||||
|
- ✅ **测试**:264 项全过(once 模式 15 项 + 多模型档案 17 项新增)
|
||||||
|
|
||||||
### v3.1
|
### v3.1
|
||||||
- ✅ **修复 FTS5 中文搜索失效**([subsystems/db.py](subsystems/db.py)):
|
- ✅ **修复 FTS5 中文搜索失效**([subsystems/db.py](subsystems/db.py)):
|
||||||
- FTS5 虚拟表改用 `tokenize='trigram'` 分词器(需 SQLite >= 3.34,环境不支持时自动退回 unicode61)
|
- FTS5 虚拟表改用 `tokenize='trigram'` 分词器(需 SQLite >= 3.34,环境不支持时自动退回 unicode61)
|
||||||
|
|||||||
+575
-161
File diff suppressed because it is too large.
Load diff
+83
-18
@@ -8,6 +8,7 @@ common.config - aibox 配置管理器
|
|||||||
|
|
||||||
import os
|
import os
|
||||||
import json
|
import json
|
||||||
|
import copy
|
||||||
from typing import Dict, Optional
|
from typing import Dict, Optional
|
||||||
|
|
||||||
from .i18n import i18n
|
from .i18n import i18n
|
||||||
@@ -64,14 +65,36 @@ class ConfigManager:
|
|||||||
|
|
||||||
# 大模型调用接口
|
# 大模型调用接口
|
||||||
"api": {
|
"api": {
|
||||||
|
"provider": "openai_compatible", # 提供商:deepseek / openai_compatible(OpenAI Chat Completions 协议兼容)
|
||||||
"deepseek_api_url": "https://api.deepseek.com/chat/completions", # deepseek API URL
|
"deepseek_api_url": "https://api.deepseek.com/chat/completions", # deepseek API URL
|
||||||
"deepseek_model": "deepseek-v4-flash", # 模型名字
|
"deepseek_model": "deepseek-v4-flash", # 模型名字
|
||||||
"reasoning_enabled": False, # 开启思维链
|
"reasoning_enabled": False, # 开启思维链
|
||||||
"api_key_env": "DEEPSEEK_API_KEY", # API-KEY所在的环境变量
|
"api_key_env": "DEEPSEEK_API_KEY", # API-KEY所在的环境变量
|
||||||
"timeout": 120, # 调用大模型请求超时时间
|
"timeout": 120, # 调用大模型请求超时时间
|
||||||
"temperature": 0.7 # 大模型对话温度(类似思维发散程度)
|
"temperature": 0.7, # 大模型对话温度(类似思维发散程度)
|
||||||
|
"default_model": "deepseek" # 默认模型档案名(对应 models 节的 key)
|
||||||
},
|
},
|
||||||
|
|
||||||
|
# 多模型档案:可配置多个 OpenAI Chat Completions 兼容端点,按名称选择
|
||||||
|
# 示例(在 ~/.aibox/config.json 中添加):
|
||||||
|
# "models": {
|
||||||
|
# "kimi": {
|
||||||
|
# "api_url": "https://api.moonshot.cn/v1/chat/completions",
|
||||||
|
# "model": "kimi-k2",
|
||||||
|
# "api_key_env": "MOONSHOT_API_KEY", # 未配置回退 api.api_key_env;空串 "" 为免密端点
|
||||||
|
# "reasoning_enabled": false,
|
||||||
|
# "timeout": 120
|
||||||
|
# },
|
||||||
|
# "local": {
|
||||||
|
# "api_url": "http://localhost:11434/v1/chat/completions",
|
||||||
|
# "model": "qwen3:32b",
|
||||||
|
# "api_key_env": ""
|
||||||
|
# }
|
||||||
|
# }
|
||||||
|
# 选择优先级:--model <档案名> > api.default_model > 回退旧 api 扁平节。
|
||||||
|
# 默认留空:用户未配置档案时走旧 api 节(零破坏)。
|
||||||
|
"models": {},
|
||||||
|
|
||||||
# 大模型上下文
|
# 大模型上下文
|
||||||
"context": {
|
"context": {
|
||||||
"max_history_messages": 500, # 最大历史消息数量
|
"max_history_messages": 500, # 最大历史消息数量
|
||||||
@@ -165,6 +188,7 @@ class ConfigManager:
|
|||||||
print(i18n.get('config_not_found', config_path))
|
print(i18n.get('config_not_found', config_path))
|
||||||
self._config = self._get_default_config()
|
self._config = self._get_default_config()
|
||||||
self._save_default_config(config_path)
|
self._save_default_config(config_path)
|
||||||
|
self._migrate_legacy_proxy_password()
|
||||||
|
|
||||||
def _merge_configs(self, default: Dict, user: Dict) -> Dict:
|
def _merge_configs(self, default: Dict, user: Dict) -> Dict:
|
||||||
result = default.copy()
|
result = default.copy()
|
||||||
@@ -198,6 +222,28 @@ class ConfigManager:
|
|||||||
self._load_config()
|
self._load_config()
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_proxy_password(self, proxy_conf: Dict) -> Optional[str]:
|
||||||
|
"""解析代理密码:内存明文优先,否则按 password_env 查环境变量。
|
||||||
|
|
||||||
|
对齐 api_key_env 惯例:配置文件只持久化环境变量名,明文密码不落盘。
|
||||||
|
"""
|
||||||
|
password = proxy_conf.get("password")
|
||||||
|
if password:
|
||||||
|
return password
|
||||||
|
env_name = proxy_conf.get("password_env")
|
||||||
|
if env_name:
|
||||||
|
return os.environ.get(env_name) or None
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _migrate_legacy_proxy_password(self):
|
||||||
|
"""一次性迁移:旧版把代理密码明文落盘。加载后明文保留在内存供本会话
|
||||||
|
使用,同时立即重写脱敏配置文件,清除盘上存量明文。"""
|
||||||
|
socks = (self._config or {}).get("proxy", {}).get("socks5")
|
||||||
|
if socks and socks.get("password"):
|
||||||
|
print("⚠️ 检测到配置文件中的代理明文密码:已保留供本次会话使用,不再落盘。"
|
||||||
|
"跨重启请将密码存入环境变量,并把 proxy.socks5.password_env 指向该变量名")
|
||||||
|
self._save_config()
|
||||||
|
|
||||||
def get_proxy(self) -> Optional[Dict[str, str]]:
|
def get_proxy(self) -> Optional[Dict[str, str]]:
|
||||||
"""获取当前代理配置(返回 requests 可用的 proxies 字典)"""
|
"""获取当前代理配置(返回 requests 可用的 proxies 字典)"""
|
||||||
proxy_conf = self.get("proxy.socks5")
|
proxy_conf = self.get("proxy.socks5")
|
||||||
@@ -206,15 +252,20 @@ class ConfigManager:
|
|||||||
host = proxy_conf.get("host", "127.0.0.1")
|
host = proxy_conf.get("host", "127.0.0.1")
|
||||||
port = proxy_conf.get("port", 1080)
|
port = proxy_conf.get("port", 1080)
|
||||||
username = proxy_conf.get("username")
|
username = proxy_conf.get("username")
|
||||||
password = proxy_conf.get("password")
|
password = self._resolve_proxy_password(proxy_conf)
|
||||||
if username and password:
|
if username and password:
|
||||||
proxy_url = f"socks5://{username}:{password}@{host}:{port}"
|
proxy_url = f"socks5://{username}:{password}@{host}:{port}"
|
||||||
else:
|
else:
|
||||||
proxy_url = f"socks5://{host}:{port}"
|
proxy_url = f"socks5://{host}:{port}"
|
||||||
return {"http": proxy_url, "https": proxy_url}
|
return {"http": proxy_url, "https": proxy_url}
|
||||||
|
|
||||||
def set_proxy(self, host: str, port: int, username: str = None, password: str = None, enabled: bool = True, save: bool = True):
|
def set_proxy(self, host: str, port: int, username: str = None, password: str = None,
|
||||||
"""动态设置代理配置(内存中更新,并可选择写入配置文件)"""
|
password_env: str = None, enabled: bool = True, save: bool = True):
|
||||||
|
"""动态设置代理配置(内存中更新,并可选择写入配置文件)。
|
||||||
|
|
||||||
|
password 明文仅存内存(_save_config 落盘时强制脱敏);
|
||||||
|
跨重启持久化请用 password_env 指向存有密码的环境变量名。
|
||||||
|
"""
|
||||||
# 确保 proxy 节点存在
|
# 确保 proxy 节点存在
|
||||||
if "proxy" not in self._config:
|
if "proxy" not in self._config:
|
||||||
self._config["proxy"] = {}
|
self._config["proxy"] = {}
|
||||||
@@ -223,31 +274,45 @@ class ConfigManager:
|
|||||||
"host": host,
|
"host": host,
|
||||||
"port": port,
|
"port": port,
|
||||||
"username": username,
|
"username": username,
|
||||||
"password": password
|
"password": password,
|
||||||
|
"password_env": password_env
|
||||||
}
|
}
|
||||||
if save:
|
if save:
|
||||||
self._save_config()
|
return self._save_config()
|
||||||
return True
|
return True
|
||||||
|
|
||||||
def _save_config(self):
|
def _save_config(self) -> bool:
|
||||||
"""将当前配置保存到配置文件(覆盖)"""
|
"""将当前配置保存到配置文件(覆盖)。返回是否成功落盘。
|
||||||
|
|
||||||
|
落盘前对代理密码做纵深防御脱敏:无论调用路径如何,明文密码一律不写入文件。
|
||||||
|
L2(审计 2026-08-18):此前异常仅 print,set_proxy 无条件报"已保存",
|
||||||
|
磁盘只读等场景下用户以为已持久化,重启后配置丢失。
|
||||||
|
"""
|
||||||
config_path = self._config_path
|
config_path = self._config_path
|
||||||
if config_path:
|
if not config_path:
|
||||||
try:
|
return False
|
||||||
os.makedirs(os.path.dirname(config_path), exist_ok=True)
|
try:
|
||||||
with open(config_path, 'w', encoding='utf-8') as f:
|
safe_config = copy.deepcopy(self._config)
|
||||||
json.dump(self._config, f, ensure_ascii=False, indent=2)
|
socks = safe_config.get("proxy", {}).get("socks5")
|
||||||
print(f"✅ 配置已保存到 {config_path}")
|
if socks and socks.get("password"):
|
||||||
except Exception as e:
|
socks["password"] = None
|
||||||
print(f"⚠️ 保存配置失败: {e}")
|
os.makedirs(os.path.dirname(config_path), exist_ok=True)
|
||||||
|
with open(config_path, 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(safe_config, f, ensure_ascii=False, indent=2)
|
||||||
|
print(f"✅ 配置已保存到 {config_path}")
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
print(f"⚠️ 保存配置失败: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
def get_proxy_info(self) -> Dict:
|
def get_proxy_info(self) -> Dict:
|
||||||
"""获取代理配置的详细信息(用于显示)"""
|
"""获取代理配置的详细信息(用于显示,不回传任何密码值)"""
|
||||||
proxy_conf = self.get("proxy.socks5", {})
|
proxy_conf = self.get("proxy.socks5", {})
|
||||||
return {
|
return {
|
||||||
"enabled": proxy_conf.get("enabled", False),
|
"enabled": proxy_conf.get("enabled", False),
|
||||||
"host": proxy_conf.get("host", ""),
|
"host": proxy_conf.get("host", ""),
|
||||||
"port": proxy_conf.get("port", 0),
|
"port": proxy_conf.get("port", 0),
|
||||||
"username": proxy_conf.get("username"),
|
"username": proxy_conf.get("username"),
|
||||||
"password": proxy_conf.get("password") is not None # 只显示是否存在
|
"password": self._resolve_proxy_password(proxy_conf) is not None, # 只显示是否存在
|
||||||
|
"password_env": proxy_conf.get("password_env") or ""
|
||||||
}
|
}
|
||||||
+489
-34
@@ -22,27 +22,48 @@ from typing import Dict, Iterator, List, Optional
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
|
import time
|
||||||
import requests
|
import requests
|
||||||
|
|
||||||
|
# L12(审计 2026-08-18):瞬时错误体可能回显网关/WAF 错误页中的
|
||||||
|
# Authorization 头,抹掉 Bearer 令牌后再进日志,防密钥落盘
|
||||||
|
_AUTH_HEADER_RE = re.compile(r"Bearer\s+[A-Za-z0-9._~+/=-]+")
|
||||||
|
|
||||||
|
|
||||||
|
def _sanitize_log_msg(msg: str) -> str:
|
||||||
|
"""日志脱敏:替换错误消息中的 Bearer 令牌。"""
|
||||||
|
if not msg:
|
||||||
|
return msg
|
||||||
|
return _AUTH_HEADER_RE.sub("Bearer ***", msg)
|
||||||
|
|
||||||
|
|
||||||
class LLMClient(ABC):
|
class LLMClient(ABC):
|
||||||
"""大模型客户端抽象基类。子类需实现流式/非流式两种调用,并归一化为统一事件格式。"""
|
"""大模型客户端抽象基类。子类需实现流式/非流式两种调用,并归一化为统一事件格式。"""
|
||||||
|
|
||||||
def __init__(self, api_key: str, model: str, base_url: str,
|
def __init__(self, api_key: str, model: str, base_url: str,
|
||||||
timeout: int, config=None, logger=None):
|
timeout: int, config=None, logger=None,
|
||||||
|
reasoning_enabled: bool = None,
|
||||||
|
display_name: str = None, default_temperature: float = None):
|
||||||
self.api_key = api_key
|
self.api_key = api_key
|
||||||
self.model = model
|
self.model = model
|
||||||
self.base_url = base_url
|
self.base_url = base_url
|
||||||
self.timeout = timeout
|
self.timeout = timeout
|
||||||
self.config = config # 可选,用于读取代理等动态配置
|
self.config = config # 可选,用于读取代理等动态配置
|
||||||
self.logger = logger # 可选,用于日志输出
|
self.logger = logger # 可选,用于日志输出
|
||||||
|
# M3:档案级思考模式开关为构造契约;None = 未配置(消费方回退全局 api.reasoning_enabled)
|
||||||
|
self.reasoning_enabled = reasoning_enabled
|
||||||
|
# L1:展示名(默认取档案名),业务层上下文(如 history 的 assistant 名)不再误标 "DeepSeek"
|
||||||
|
self._display_name = display_name
|
||||||
|
# L3:档案级默认采样温度;None = 未配置(方法参数未显式传入时回退硬编码默认值)
|
||||||
|
self.default_temperature = default_temperature
|
||||||
# 当前活动的 HTTP 响应引用,供 cancel() 强制关闭底层连接以中断阻塞读取
|
# 当前活动的 HTTP 响应引用,供 cancel() 强制关闭底层连接以中断阻塞读取
|
||||||
self._current_response = None
|
self._current_response = None
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def name(self) -> str:
|
def name(self) -> str:
|
||||||
"""厂商/模型展示名,业务层可用于上下文(如 history 里的 assistant 名)。"""
|
"""厂商/模型展示名,业务层可用于上下文(如 history 里的 assistant 名)。"""
|
||||||
return "LLM"
|
return self._display_name or "LLM"
|
||||||
|
|
||||||
# ---------- 工具方法 ----------
|
# ---------- 工具方法 ----------
|
||||||
|
|
||||||
@@ -55,12 +76,16 @@ class LLMClient(ABC):
|
|||||||
def _log(self, level: str, msg: str):
|
def _log(self, level: str, msg: str):
|
||||||
if self.logger is not None and hasattr(self.logger, level):
|
if self.logger is not None and hasattr(self.logger, level):
|
||||||
try:
|
try:
|
||||||
getattr(self.logger, level)(msg)
|
getattr(self.logger, level)(_sanitize_log_msg(msg))
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
def cancel(self):
|
def cancel(self):
|
||||||
"""中断当前流式请求。线程安全,可从任意线程调用。"""
|
"""中断当前流式请求。线程安全,可从任意线程调用。
|
||||||
|
|
||||||
|
注:用户中断标志由 FailoverClient 维护(其 cancel() 广播整条链并置
|
||||||
|
自身 _cancelled,据此跳过转移)——底层 client 只负责强关连接。
|
||||||
|
"""
|
||||||
resp = self._current_response
|
resp = self._current_response
|
||||||
self._current_response = None
|
self._current_response = None
|
||||||
if resp is not None:
|
if resp is not None:
|
||||||
@@ -78,16 +103,18 @@ class LLMClient(ABC):
|
|||||||
# ---------- 抽象接口 ----------
|
# ---------- 抽象接口 ----------
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def chat_completion(self, messages: List[Dict], temperature: float = 0.3,
|
def chat_completion(self, messages: List[Dict], temperature: float = None,
|
||||||
max_tokens: int = 1024, **kwargs) -> str:
|
max_tokens: int = 1024, **kwargs) -> str:
|
||||||
"""非流式对话,返回完整文本。用于对话压缩等离线摘要场景。"""
|
"""非流式对话,返回完整文本。用于对话压缩等离线摘要场景。
|
||||||
|
temperature=None 时按 default_temperature(档案级)→ 0.3 回退。"""
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def stream_chat(self, messages: List[Dict], temperature: float = 0.7,
|
def stream_chat(self, messages: List[Dict], temperature: float = None,
|
||||||
max_tokens: int = 8192, tools: Optional[List[Dict]] = None,
|
max_tokens: int = 8192, tools: Optional[List[Dict]] = None,
|
||||||
tool_choice: Optional[str] = None,
|
tool_choice: Optional[str] = None,
|
||||||
reasoning_enabled: bool = False, **extra) -> Iterator[Dict]:
|
reasoning_enabled: bool = False, **extra) -> Iterator[Dict]:
|
||||||
"""流式对话,逐条 yield 统一事件格式的 dict。业务层据此驱动输出/工具调用。"""
|
"""流式对话,逐条 yield 统一事件格式的 dict。业务层据此驱动输出/工具调用。
|
||||||
|
temperature=None 时按 default_temperature(档案级)→ 0.7 回退。"""
|
||||||
|
|
||||||
|
|
||||||
class DeepSeekClient(LLMClient):
|
class DeepSeekClient(LLMClient):
|
||||||
@@ -99,12 +126,18 @@ class DeepSeekClient(LLMClient):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, api_key: str, model: str, base_url: str,
|
def __init__(self, api_key: str, model: str, base_url: str,
|
||||||
timeout: int = 30, config=None, logger=None):
|
timeout: int = 30, config=None, logger=None,
|
||||||
super().__init__(api_key, model, base_url, timeout, config, logger)
|
reasoning_enabled: bool = None,
|
||||||
|
display_name: str = None, default_temperature: float = None):
|
||||||
|
super().__init__(api_key, model, base_url, timeout, config, logger,
|
||||||
|
reasoning_enabled=reasoning_enabled,
|
||||||
|
display_name=display_name,
|
||||||
|
default_temperature=default_temperature)
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def name(self) -> str:
|
def name(self) -> str:
|
||||||
return "DeepSeek"
|
# L1:档案命中时展示档案名(或档案 display_name),旧扁平节保持 "DeepSeek"
|
||||||
|
return self._display_name or "DeepSeek"
|
||||||
|
|
||||||
def _headers(self) -> Dict[str, str]:
|
def _headers(self) -> Dict[str, str]:
|
||||||
return {
|
return {
|
||||||
@@ -112,8 +145,11 @@ class DeepSeekClient(LLMClient):
|
|||||||
"Content-Type": "application/json",
|
"Content-Type": "application/json",
|
||||||
}
|
}
|
||||||
|
|
||||||
def chat_completion(self, messages: List[Dict], temperature: float = 0.3,
|
def chat_completion(self, messages: List[Dict], temperature: float = None,
|
||||||
max_tokens: int = 1024, **kwargs) -> str:
|
max_tokens: int = 1024, **kwargs) -> str:
|
||||||
|
# L3:None → 档案级 default_temperature → 硬编码默认(与旧版行为一致)
|
||||||
|
if temperature is None:
|
||||||
|
temperature = self.default_temperature if self.default_temperature is not None else 0.3
|
||||||
payload = {
|
payload = {
|
||||||
"model": self.model,
|
"model": self.model,
|
||||||
"messages": messages,
|
"messages": messages,
|
||||||
@@ -132,16 +168,29 @@ class DeepSeekClient(LLMClient):
|
|||||||
if response.status_code == 200:
|
if response.status_code == 200:
|
||||||
result = response.json()
|
result = response.json()
|
||||||
return result["choices"][0]["message"]["content"].strip()
|
return result["choices"][0]["message"]["content"].strip()
|
||||||
|
if response.status_code == 429 or response.status_code >= 500:
|
||||||
|
# L4a:瞬时故障显式抛出——FailoverClient 据此转移下一档案;
|
||||||
|
# 无包装器时由调用方 catch-all 接住,下游仍走失败分支(语义等价)
|
||||||
|
raise TransientLLMError(
|
||||||
|
f"LLM 请求瞬时失败: HTTP {response.status_code}: {response.text[:200]}",
|
||||||
|
retry_after=_retry_after_of(response))
|
||||||
self._log("error", f"LLM 压缩请求失败: HTTP {response.status_code}")
|
self._log("error", f"LLM 压缩请求失败: HTTP {response.status_code}")
|
||||||
return None
|
return None
|
||||||
|
except TransientLLMError:
|
||||||
|
raise # L4a:瞬时故障上抛供 FailoverClient 转移,不吞
|
||||||
|
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||||
|
raise # L4a:连接类瞬时故障同样上抛(原被吞成 None,无法转移)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self._log("error", f"LLM 压缩请求异常: {e}")
|
self._log("error", f"LLM 压缩请求异常: {e}")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
def stream_chat(self, messages: List[Dict], temperature: float = 0.7,
|
def stream_chat(self, messages: List[Dict], temperature: float = None,
|
||||||
max_tokens: int = 8192, tools: Optional[List[Dict]] = None,
|
max_tokens: int = 8192, tools: Optional[List[Dict]] = None,
|
||||||
tool_choice: Optional[str] = None,
|
tool_choice: Optional[str] = None,
|
||||||
reasoning_enabled: bool = False, **extra) -> Iterator[Dict]:
|
reasoning_enabled: bool = False, **extra) -> Iterator[Dict]:
|
||||||
|
# L3:None → 档案级 default_temperature → 硬编码默认(与旧版行为一致)
|
||||||
|
if temperature is None:
|
||||||
|
temperature = self.default_temperature if self.default_temperature is not None else 0.7
|
||||||
payload = {
|
payload = {
|
||||||
"model": self.model,
|
"model": self.model,
|
||||||
"messages": messages,
|
"messages": messages,
|
||||||
@@ -170,14 +219,19 @@ class DeepSeekClient(LLMClient):
|
|||||||
self._current_response = response
|
self._current_response = response
|
||||||
|
|
||||||
if response.status_code != 200:
|
if response.status_code != 200:
|
||||||
|
if response.status_code == 429 or response.status_code >= 500:
|
||||||
|
# L4b:瞬时故障显式分类(此时尚未产出任何事件,可安全转移重放)
|
||||||
|
raise TransientLLMError(
|
||||||
|
f"LLM 流式请求瞬时失败: HTTP {response.status_code}: {response.text[:200]}",
|
||||||
|
retry_after=_retry_after_of(response))
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
f"HTTP {response.status_code}: {response.text}")
|
f"HTTP {response.status_code}: {response.text[:200]}")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
for line in response.iter_lines():
|
for line in response.iter_lines():
|
||||||
if not line:
|
if not line:
|
||||||
continue
|
continue
|
||||||
line_text = line.decode("utf-8")
|
line_text = line.decode("utf-8", errors="replace") # L3:非 UTF-8 行不中断流
|
||||||
if not line_text.startswith("data: "):
|
if not line_text.startswith("data: "):
|
||||||
continue
|
continue
|
||||||
data = line_text[6:]
|
data = line_text[6:]
|
||||||
@@ -198,11 +252,19 @@ class DeepSeekClient(LLMClient):
|
|||||||
finally:
|
finally:
|
||||||
self._current_response = None
|
self._current_response = None
|
||||||
|
|
||||||
|
# L5:思考内容的常见厂商字段别名(首个命中的非空字段生效)
|
||||||
|
REASONING_FIELDS = ("reasoning_content", "reasoning", "thinking", "thinking_content")
|
||||||
|
|
||||||
def _parse_delta(self, delta: Dict, reasoning_enabled: bool) -> Iterator[Dict]:
|
def _parse_delta(self, delta: Dict, reasoning_enabled: bool) -> Iterator[Dict]:
|
||||||
"""将单条 delta 归一化为统一事件。子类覆写以适配不同厂商字段。"""
|
"""将单条 delta 归一化为统一事件。子类覆写以适配不同厂商字段。"""
|
||||||
# 思考内容
|
# 思考内容:DeepSeek reasoning_content / OpenRouter reasoning /
|
||||||
if reasoning_enabled and delta.get("reasoning_content"):
|
# 部分 GLM thinking(_content) —— 此前仅识别 DeepSeek 字段,其他厂商思考流静默丢失
|
||||||
yield {"type": "reasoning", "text": delta["reasoning_content"]}
|
if reasoning_enabled:
|
||||||
|
for field in self.REASONING_FIELDS:
|
||||||
|
value = delta.get(field)
|
||||||
|
if value:
|
||||||
|
yield {"type": "reasoning", "text": value}
|
||||||
|
break
|
||||||
# 正文
|
# 正文
|
||||||
if delta.get("content"):
|
if delta.get("content"):
|
||||||
yield {"type": "content", "text": delta["content"]}
|
yield {"type": "content", "text": delta["content"]}
|
||||||
@@ -219,17 +281,251 @@ class DeepSeekClient(LLMClient):
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def resolve_api_key(api_key: Optional[str], config=None) -> str:
|
class TransientLLMError(RuntimeError):
|
||||||
"""解析 API Key:优先显式传入,其次环境变量,最后 ~/.cert/deepseek.key。"""
|
"""瞬时 LLM 故障(HTTP 429/5xx)——FailoverClient 据此转移下一档案(L4a)。
|
||||||
|
|
||||||
|
L4c:retry_after 携带服务端 Retry-After 响应头(数字秒)建议冷却时长,
|
||||||
|
FailoverClient 以其替代默认冷却期(429 限流时尊重服务端节奏)。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, message: str, retry_after: Optional[float] = None):
|
||||||
|
super().__init__(message)
|
||||||
|
self.retry_after = retry_after
|
||||||
|
|
||||||
|
|
||||||
|
def _retry_after_of(response) -> Optional[float]:
|
||||||
|
"""从 Retry-After 响应头解析冷却秒数(仅支持数字秒格式,其余忽略返回 None)。
|
||||||
|
|
||||||
|
P2:上限 cap 3600s——异常/恶意服务端返回超大值(如 999999)时,
|
||||||
|
避免档案被单条响应头冷却数天。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
v = response.headers.get("Retry-After")
|
||||||
|
if v is None:
|
||||||
|
return None
|
||||||
|
return min(max(1.0, float(v)), 3600.0)
|
||||||
|
except (TypeError, ValueError, AttributeError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class FailoverClient:
|
||||||
|
"""跨档案失败转移包装器(L4a)。
|
||||||
|
|
||||||
|
持有序 client 链(主档案 + fallback 档案),对外实现与 LLMClient 相同的
|
||||||
|
消费面(未显式定义的属性经 __getattr__ 委托主 client)。
|
||||||
|
|
||||||
|
L4a 范围——chat_completion(非流式幂等)按链转移:
|
||||||
|
- 转移:HTTP 429/5xx(TransientLLMError)、连接失败/超时(requests 连接类异常)
|
||||||
|
- 不转移:认证/参数类 4xx(底层返回 None,直接透传——配置问题应响亮暴露)
|
||||||
|
失败档案进入冷却期(cooldown_seconds,默认 60s),期间直接跳过,
|
||||||
|
避免每次请求都撞刚失败的端点。链耗尽返回 None(与 chat_completion 失败契约一致)。
|
||||||
|
|
||||||
|
L4b 范围——stream_chat 仅**首事件前**故障转移(连接失败/HTTP 429/5xx,
|
||||||
|
尚未产出任何 reasoning/content/tool_call 事件,重放对用户不可见);
|
||||||
|
已产出事件后失败直接上抛(重放会导致前端重复输出/工具增量错乱);
|
||||||
|
用户中断(cancel() 置位)引发的连接异常绝不转移。
|
||||||
|
|
||||||
|
L4c 范围——可观测性与预算控制:
|
||||||
|
- get_stats() 暴露 requests/failovers/cooldown_skips/exhausted 聚合计数
|
||||||
|
- on_failover(from_name, to_name, reason) 回调(to_name=None 表示链耗尽),
|
||||||
|
供管理面/前端通知接入,回调异常不影响主流程
|
||||||
|
- timeout_budget 限制整条链尝试的总秒数(缺省不限),耗尽即停止尝试剩余档案,
|
||||||
|
防止长链故障时超时叠加(3 档案最坏 3×timeout)
|
||||||
|
- 429 响应带 Retry-After(数字秒)时按其延长该档案冷却期
|
||||||
|
|
||||||
|
并发契约(P4):实例非线程安全(_cancelled/_call_started/_stats 无锁)。
|
||||||
|
当前调用面满足"单线程串行调用 + cancel() 可从其他线程"的模式(compressor
|
||||||
|
在 _msg_lock 内同步调用、子代理各持独立实例);如需多线程共享调用,
|
||||||
|
需由调用方加锁或为实例补锁。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, clients: List, logger=None, cooldown_seconds: float = 60.0,
|
||||||
|
timeout_budget: Optional[float] = None, on_failover=None):
|
||||||
|
self._clients = list(clients)
|
||||||
|
self.logger = logger
|
||||||
|
self.cooldown_seconds = cooldown_seconds
|
||||||
|
self._fail_at = {} # client.name → (失败时刻, 冷却秒数)
|
||||||
|
self._cancelled = False
|
||||||
|
self._stats = {"requests": 0, "failovers": 0, "cooldown_skips": 0, "exhausted": 0}
|
||||||
|
if timeout_budget is not None and timeout_budget <= 0:
|
||||||
|
self._log("warning", f"timeout_budget 非法({timeout_budget} <= 0),按未配置处理")
|
||||||
|
timeout_budget = None
|
||||||
|
self.timeout_budget = timeout_budget
|
||||||
|
self.on_failover = on_failover
|
||||||
|
self._call_started = None
|
||||||
|
|
||||||
|
def __getattr__(self, item):
|
||||||
|
# 未显式定义的属性/方法委托主 client(name/model/api_key/cancel 等);
|
||||||
|
# 下划线属性直接报错,防 __init__ 前访问 self._clients 的无限递归
|
||||||
|
if item.startswith("_"):
|
||||||
|
raise AttributeError(item)
|
||||||
|
return getattr(self._clients[0], item)
|
||||||
|
|
||||||
|
def _log(self, level: str, msg: str):
|
||||||
|
if self.logger is not None and hasattr(self.logger, level):
|
||||||
|
try:
|
||||||
|
getattr(self.logger, level)(_sanitize_log_msg(msg))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _cooling(self, client) -> bool:
|
||||||
|
entry = self._fail_at.get(client.name)
|
||||||
|
if entry is None:
|
||||||
|
return False
|
||||||
|
t, cd = entry
|
||||||
|
return (time.monotonic() - t) < cd
|
||||||
|
|
||||||
|
def _mark_fail(self, client, cooldown: Optional[float] = None):
|
||||||
|
# L4c:429 Retry-After(经 TransientLLMError.retry_after)覆盖默认冷却时长
|
||||||
|
self._fail_at[client.name] = (time.monotonic(),
|
||||||
|
cooldown if cooldown and cooldown > 0
|
||||||
|
else self.cooldown_seconds)
|
||||||
|
|
||||||
|
def _budget_exhausted(self) -> bool:
|
||||||
|
if self.timeout_budget is None or self._call_started is None:
|
||||||
|
return False
|
||||||
|
return (time.monotonic() - self._call_started) >= self.timeout_budget
|
||||||
|
|
||||||
|
def get_stats(self) -> Dict:
|
||||||
|
"""L4c:转移统计快照(requests/failovers/cooldown_skips/exhausted)。"""
|
||||||
|
return dict(self._stats)
|
||||||
|
|
||||||
|
def _notify_failover(self, from_name: str, to_name, reason: str):
|
||||||
|
self._stats["failovers"] += 1
|
||||||
|
try:
|
||||||
|
if self.on_failover is not None:
|
||||||
|
self.on_failover(from_name, to_name, _sanitize_log_msg(reason))
|
||||||
|
except Exception as e:
|
||||||
|
self._log("warning", f"on_failover 回调异常(忽略): {e}")
|
||||||
|
|
||||||
|
def cancel(self):
|
||||||
|
"""L4b:中断链上全部 client(任一可能正在流式中)并置自身中断标志。
|
||||||
|
|
||||||
|
显式定义以覆盖 __getattr__ 委托——包装器必须同时通知整条链。
|
||||||
|
"""
|
||||||
|
self._cancelled = True
|
||||||
|
for client in self._clients:
|
||||||
|
try:
|
||||||
|
client.cancel()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def chat_completion(self, messages: List[Dict], temperature: float = None,
|
||||||
|
max_tokens: int = 1024, **kwargs):
|
||||||
|
self._cancelled = False # 新一轮请求,清除上一轮中断标志
|
||||||
|
self._call_started = time.monotonic()
|
||||||
|
self._stats["requests"] += 1
|
||||||
|
last_exc = None
|
||||||
|
tried = 0
|
||||||
|
for idx, client in enumerate(self._clients):
|
||||||
|
if self._cooling(client):
|
||||||
|
self._stats["cooldown_skips"] += 1
|
||||||
|
self._log("info", f"模型档案 {client.name} 处于失败冷却期,跳过")
|
||||||
|
continue
|
||||||
|
if self._budget_exhausted():
|
||||||
|
self._log("warning",
|
||||||
|
f"失败转移时间预算({self.timeout_budget}s)已耗尽,停止尝试剩余档案")
|
||||||
|
break
|
||||||
|
tried += 1
|
||||||
|
try:
|
||||||
|
result = client.chat_completion(messages, temperature=temperature,
|
||||||
|
max_tokens=max_tokens, **kwargs)
|
||||||
|
if result is not None:
|
||||||
|
return result
|
||||||
|
# None:认证/参数类 4xx 或响应异常——不可转移,直接透传
|
||||||
|
return None
|
||||||
|
except (TransientLLMError, requests.exceptions.RequestException) as e:
|
||||||
|
if self._cancelled:
|
||||||
|
return None # 用户中断引发的异常:不转移
|
||||||
|
last_exc = e
|
||||||
|
self._mark_fail(client, cooldown=getattr(e, "retry_after", None))
|
||||||
|
to_name = (self._clients[idx + 1].name
|
||||||
|
if idx + 1 < len(self._clients) else None)
|
||||||
|
self._notify_failover(client.name, to_name, str(e))
|
||||||
|
self._log("warning", f"模型档案 {client.name} 瞬时故障({e}),转移下一档案")
|
||||||
|
self._stats["exhausted"] += 1 # 末尾统一计数(链耗尽/预算耗尽/全冷却)
|
||||||
|
self._log("error", f"全部 {len(self._clients)} 个模型档案均不可用(尝试 {tried} 个):{last_exc}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def stream_chat(self, messages: List[Dict], temperature: float = None,
|
||||||
|
max_tokens: int = 8192, tools: Optional[List[Dict]] = None,
|
||||||
|
tool_choice: Optional[str] = None,
|
||||||
|
reasoning_enabled: bool = False, **extra) -> Iterator[Dict]:
|
||||||
|
"""L4b:流式转移——仅首事件前故障可安全重放(对用户不可见)。"""
|
||||||
|
self._cancelled = False # 新一轮请求,清除上一轮中断标志
|
||||||
|
self._call_started = time.monotonic()
|
||||||
|
self._stats["requests"] += 1
|
||||||
|
last_exc = None
|
||||||
|
for idx, client in enumerate(self._clients):
|
||||||
|
if self._cooling(client):
|
||||||
|
self._stats["cooldown_skips"] += 1
|
||||||
|
self._log("info", f"模型档案 {client.name} 处于失败冷却期,跳过")
|
||||||
|
continue
|
||||||
|
if self._budget_exhausted():
|
||||||
|
self._log("warning",
|
||||||
|
f"失败转移时间预算({self.timeout_budget}s)已耗尽,停止尝试剩余档案")
|
||||||
|
break
|
||||||
|
produced_any = False
|
||||||
|
try:
|
||||||
|
for event in client.stream_chat(
|
||||||
|
messages, temperature=temperature, max_tokens=max_tokens,
|
||||||
|
tools=tools, tool_choice=tool_choice,
|
||||||
|
reasoning_enabled=reasoning_enabled, **extra):
|
||||||
|
produced_any = True
|
||||||
|
yield event
|
||||||
|
return # 自然结束(含 done 事件)视为成功
|
||||||
|
except (TransientLLMError, requests.exceptions.RequestException) as e:
|
||||||
|
if self._cancelled:
|
||||||
|
raise # 用户中断(cancel 强关连接):绝不转移
|
||||||
|
if produced_any:
|
||||||
|
raise # 已向消费方产出事件,重放会重复输出/工具增量错乱
|
||||||
|
last_exc = e
|
||||||
|
self._mark_fail(client, cooldown=getattr(e, "retry_after", None))
|
||||||
|
to_name = (self._clients[idx + 1].name
|
||||||
|
if idx + 1 < len(self._clients) else None)
|
||||||
|
self._notify_failover(client.name, to_name, str(e))
|
||||||
|
self._log("warning",
|
||||||
|
f"模型档案 {client.name} 流式瞬时故障(首事件前,{e}),转移下一档案")
|
||||||
|
self._stats["exhausted"] += 1
|
||||||
|
if last_exc is not None:
|
||||||
|
raise last_exc # 链耗尽:抛最后异常,由既有错误处理(回滚/报错)接住
|
||||||
|
raise RuntimeError(
|
||||||
|
f"全部 {len(self._clients)} 个模型档案不可用(冷却期或转移时间预算耗尽)")
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_api_key(api_key: Optional[str], config=None,
|
||||||
|
env_name: Optional[str] = None,
|
||||||
|
allow_missing: bool = False,
|
||||||
|
allow_key_file: bool = True) -> str:
|
||||||
|
"""解析 API Key:优先显式传入,其次环境变量,最后 ~/.cert/deepseek.key。
|
||||||
|
|
||||||
|
env_name: 档案级环境变量名(来自 models.<name>.api_key_env);
|
||||||
|
None 时回退 config 的 api.api_key_env。
|
||||||
|
allow_missing: 本地免密端点(models.<name>.api_key_env 为空串)时,
|
||||||
|
允许无 Key,返回空串(Bearer 空头,Ollama/vLLM 忽略)。
|
||||||
|
allow_key_file: 是否允许回退读取 ~/.cert/deepseek.key。该文件是 DeepSeek
|
||||||
|
专属凭据,模型档案端点(其他厂商)必须传 False——环境变量取不到
|
||||||
|
时直接报错,绝不把 A 厂商 Key 发给 B 厂商(跨厂商凭据泄漏)。
|
||||||
|
"""
|
||||||
if api_key:
|
if api_key:
|
||||||
return api_key
|
return api_key
|
||||||
env_name = None
|
if env_name is None:
|
||||||
if config is not None and hasattr(config, "get"):
|
if config is not None and hasattr(config, "get"):
|
||||||
env_name = config.get("api.api_key_env")
|
env_name = config.get("api.api_key_env")
|
||||||
env_name = env_name or "DEEPSEEK_API_KEY"
|
env_name = env_name or "DEEPSEEK_API_KEY"
|
||||||
|
if not env_name: # 档案显式置空 → 免密端点
|
||||||
|
if allow_missing:
|
||||||
|
return ""
|
||||||
|
raise RuntimeError("模型档案 api_key_env 配置为空且未启用免密")
|
||||||
env_key = os.environ.get(env_name)
|
env_key = os.environ.get(env_name)
|
||||||
if env_key:
|
if env_key:
|
||||||
return env_key
|
return env_key
|
||||||
|
if allow_missing:
|
||||||
|
return ""
|
||||||
|
if not allow_key_file:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"模型档案端点缺少 API Key:请设置环境变量 {env_name},"
|
||||||
|
f"或将该档案的 api_key_env 配置为空串以启用本地免密端点")
|
||||||
try:
|
try:
|
||||||
home = os.environ.get("HOME", os.environ["USERPROFILE"])
|
home = os.environ.get("HOME", os.environ["USERPROFILE"])
|
||||||
key_path = os.path.join(home, ".cert", "deepseek.key")
|
key_path = os.path.join(home, ".cert", "deepseek.key")
|
||||||
@@ -240,26 +536,184 @@ def resolve_api_key(api_key: Optional[str], config=None) -> str:
|
|||||||
f"请设置环境变量 {env_name} 或在配置文件中设置 API Key")
|
f"请设置环境变量 {env_name} 或在配置文件中设置 API Key")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_profile(name: str, profile) -> None:
|
||||||
|
"""H2:档案类型与 api_url 校验(在读取档案字段之前调用)。"""
|
||||||
|
if not isinstance(profile, dict):
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{name}' 配置无效:应为对象(字典),"
|
||||||
|
f"实际为 {type(profile).__name__},请检查 config.json 的 models.{name}")
|
||||||
|
if not str(profile.get("api_url") or "").strip():
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{name}' 缺少 api_url 字段,"
|
||||||
|
f"请在 config.json 的 models.{name} 中补齐(OpenAI Chat Completions 端点 URL)")
|
||||||
|
fb = profile.get("fallback")
|
||||||
|
if fb is not None and (not isinstance(fb, list) or not all(isinstance(x, str) for x in fb)):
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{name}' 的 fallback 字段无效:应为档案名字符串列表"
|
||||||
|
f"(如 [\"kimi\", \"local\"]),实际为 {type(fb).__name__},"
|
||||||
|
f"请检查 config.json 的 models.{name}")
|
||||||
|
|
||||||
|
|
||||||
|
def _require_model(name: str, effective_model) -> None:
|
||||||
|
"""H2:有效模型名校验——缺 model 会把档案名当模型名发给 API(或 "model": null → 400)。"""
|
||||||
|
if not str(effective_model or "").strip():
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{name}' 缺少 model 字段(且未传入裸模型名),"
|
||||||
|
f"请在 config.json 的 models.{name} 中补齐,或调用时用 --model 指定模型名")
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_model_profile(config, model: Optional[str]):
|
||||||
|
"""解析模型档案。返回 (profile_dict, effective_model_name, profile_name)。
|
||||||
|
|
||||||
|
优先级:model 命中 models.<name> 档案 > api.default_model 档案 > None(回退旧 api 节)。
|
||||||
|
model 未命中档案时视为裸模型名(同端点换模型,用默认档案的 url/key,模型名用裸名)。
|
||||||
|
|
||||||
|
L2 前缀逃逸(档案名与裸模型名歧义时使用):
|
||||||
|
- "model:NAME" 强制按裸模型名解析(档案恰好叫 NAME 时仍走裸名语义)
|
||||||
|
- "profile:NAME" 强制按档案名解析(不存在则报错,不再静默落回裸名语义)
|
||||||
|
"""
|
||||||
|
if config is None or not hasattr(config, "get"):
|
||||||
|
return None, model, None
|
||||||
|
models = config.get("models") or {}
|
||||||
|
if not isinstance(models, dict):
|
||||||
|
return None, model, None
|
||||||
|
|
||||||
|
force_bare = False
|
||||||
|
if isinstance(model, str) and ":" in model:
|
||||||
|
prefix, _, rest = model.partition(":")
|
||||||
|
if prefix == "model":
|
||||||
|
if not rest.strip():
|
||||||
|
raise ValueError("model: 前缀后为空,请提供裸模型名,如 model:deepseek-chat")
|
||||||
|
model, force_bare = rest, True
|
||||||
|
elif prefix == "profile":
|
||||||
|
if not rest.strip():
|
||||||
|
raise ValueError("profile: 前缀后为空,请提供档案名,如 profile:kimi")
|
||||||
|
prof = models.get(rest)
|
||||||
|
if prof is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{rest}' 不存在,可用档案:{', '.join(models) or '(无)'}")
|
||||||
|
_validate_profile(rest, prof)
|
||||||
|
effective = prof.get("model")
|
||||||
|
_require_model(rest, effective)
|
||||||
|
return prof, effective, rest
|
||||||
|
|
||||||
|
explicit = models.get(model) if (model and not force_bare) else None
|
||||||
|
default_name = config.get("api.default_model")
|
||||||
|
default_profile = models.get(default_name) if default_name else None
|
||||||
|
|
||||||
|
if explicit is not None:
|
||||||
|
# model 是档案名 → 完整采用该档案(url/key/模型名);
|
||||||
|
# 档案缺 model 属配置错误,直接报错(不再把档案名当模型名发出)
|
||||||
|
_validate_profile(model, explicit)
|
||||||
|
effective = explicit.get("model")
|
||||||
|
_require_model(model, effective)
|
||||||
|
return explicit, effective, model
|
||||||
|
if default_profile is not None:
|
||||||
|
# model 是裸名或未指定 → 默认档案的 url/key,模型名被裸名覆盖
|
||||||
|
_validate_profile(default_name, default_profile)
|
||||||
|
effective = model or default_profile.get("model")
|
||||||
|
_require_model(default_name, effective)
|
||||||
|
return default_profile, effective, default_name
|
||||||
|
return None, model, None
|
||||||
|
|
||||||
|
|
||||||
|
def _construct_profile_client(config, profile: Dict, profile_name: str,
|
||||||
|
effective_model: str, logger=None,
|
||||||
|
api_key: Optional[str] = None) -> LLMClient:
|
||||||
|
"""L4a:按档案构造单个 DeepSeekClient(主路径与 fallback 链复用,不处理 fallback 字段)。"""
|
||||||
|
env_name = profile.get("api_key_env")
|
||||||
|
allow_missing = (env_name == "") # 档案显式空串 → 本地免密端点
|
||||||
|
resolved_key = resolve_api_key(api_key, config, env_name=env_name,
|
||||||
|
allow_missing=allow_missing, allow_key_file=False)
|
||||||
|
return DeepSeekClient(
|
||||||
|
api_key=resolved_key,
|
||||||
|
model=effective_model,
|
||||||
|
base_url=profile.get("api_url"),
|
||||||
|
timeout=profile.get("timeout") or config.get("api.timeout", 30),
|
||||||
|
config=config,
|
||||||
|
logger=logger,
|
||||||
|
reasoning_enabled=profile.get("reasoning_enabled",
|
||||||
|
config.get("api.reasoning_enabled", False)),
|
||||||
|
display_name=profile.get("display_name") or profile_name,
|
||||||
|
default_temperature=profile.get("temperature"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def create_llm_client(config, api_key: Optional[str] = None,
|
def create_llm_client(config, api_key: Optional[str] = None,
|
||||||
logger=None, model: Optional[str] = None) -> LLMClient:
|
logger=None, model: Optional[str] = None) -> LLMClient:
|
||||||
"""根据配置创建对应厂商的 LLM 客户端。可通过 `api.provider` 切换厂商。
|
"""根据配置创建对应厂商的 LLM 客户端。
|
||||||
|
|
||||||
|
模型选择优先级:
|
||||||
|
1. model 命中 models 节档案名 → 采用该档案(api_url/model/api_key_env/timeout)
|
||||||
|
2. api.default_model 指向的档案 → model 为裸模型名时覆盖档案内模型名
|
||||||
|
3. 均未配置 → 回退旧 api 扁平节(deepseek_api_url/deepseek_model)
|
||||||
|
L2 前缀逃逸:"model:NAME" 强制裸名 / "profile:NAME" 强制档案(--model 同样支持)。
|
||||||
|
|
||||||
|
provider: deepseek / openai_compatible / openai 均映射到 DeepSeekClient
|
||||||
|
(OpenAI Chat Completions 协议兼容实现,厂商差异由子类覆写 _parse_delta)。
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
config: 配置对象(需提供 get 方法)。
|
config: 配置对象(需提供 get 方法)。
|
||||||
api_key: 显式传入的 API Key;为 None 时按 resolve_api_key 解析。
|
api_key: 显式传入的 API Key;为 None 时按 resolve_api_key 解析。
|
||||||
logger: 可选日志对象。
|
logger: 可选日志对象。
|
||||||
model: 可选模型名覆盖;为 None 时使用配置中的 `api.deepseek_model`。
|
model: 档案名或裸模型名(支持 model:/profile: 前缀逃逸)。
|
||||||
子代理等场景可通过此参数显式指定与主代理不同的模型。
|
|
||||||
"""
|
"""
|
||||||
provider = (config.get("api.provider", "deepseek") or "deepseek").lower()
|
profile, model, profile_name = _resolve_model_profile(config, model)
|
||||||
base_url = config.get("api.deepseek_api_url")
|
|
||||||
model = model or config.get("api.deepseek_model", "deepseek-v4-flash")
|
|
||||||
timeout = config.get("api.timeout", 30)
|
|
||||||
|
|
||||||
resolved_key = resolve_api_key(api_key, config)
|
if profile is None:
|
||||||
|
# 旧 api 扁平节(唯一允许回退 ~/.cert/deepseek.key 的路径,H1)
|
||||||
|
provider = (config.get("api.provider", "deepseek") or "deepseek").lower()
|
||||||
|
base_url = config.get("api.deepseek_api_url")
|
||||||
|
model = model or config.get("api.deepseek_model", "deepseek-v4-flash")
|
||||||
|
timeout = config.get("api.timeout", 30)
|
||||||
|
resolved_key = resolve_api_key(api_key, config, env_name=None,
|
||||||
|
allow_missing=False, allow_key_file=True)
|
||||||
|
else:
|
||||||
|
provider = "openai_compatible"
|
||||||
|
primary = _construct_profile_client(config, profile, profile_name,
|
||||||
|
model, logger=logger, api_key=api_key)
|
||||||
|
fallback_names = profile.get("fallback") or []
|
||||||
|
if not fallback_names:
|
||||||
|
return primary
|
||||||
|
# L4a:构造 fallback 链(一层,不递归——fallback 档案自身的 fallback 字段被忽略)
|
||||||
|
models = config.get("models") or {}
|
||||||
|
chain = [primary]
|
||||||
|
seen = {profile_name} # P6:重复项去重(冷却键同名冲突/重复撞同端点)
|
||||||
|
for fb_name in fallback_names:
|
||||||
|
if fb_name == profile_name:
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{profile_name}' 的 fallback 不能包含自身,"
|
||||||
|
f"请检查 config.json 的 models.{profile_name}.fallback")
|
||||||
|
if fb_name in seen:
|
||||||
|
if logger is not None and hasattr(logger, "warning"):
|
||||||
|
logger.warning(f"fallback 列表重复项 '{fb_name}',已忽略")
|
||||||
|
continue
|
||||||
|
seen.add(fb_name)
|
||||||
|
fb_profile = models.get(fb_name)
|
||||||
|
if fb_profile is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"模型档案 '{profile_name}' 的 fallback 指向不存在的档案 '{fb_name}',"
|
||||||
|
f"可用档案:{', '.join(models) or '(无)'}")
|
||||||
|
try:
|
||||||
|
# P3:fallback 档案与主档案同等校验(缺 api_url/model 时跳过并告警,
|
||||||
|
# 而非构造出 model=None 的 client 在请求期表现为神秘失败)
|
||||||
|
_validate_profile(fb_name, fb_profile)
|
||||||
|
fb_model = fb_profile.get("model")
|
||||||
|
_require_model(fb_name, fb_model)
|
||||||
|
chain.append(_construct_profile_client(
|
||||||
|
config, fb_profile, fb_name, fb_model,
|
||||||
|
logger=logger))
|
||||||
|
except (RuntimeError, ValueError) as e:
|
||||||
|
# 环境问题(如 fallback 档案缺 API Key)或配置缺陷(缺 model):
|
||||||
|
# 跳过该 fallback 并告警,不阻断主档案可用性
|
||||||
|
# (配置结构错误——fallback 指向不存在的档案——在上方已硬性拦截)
|
||||||
|
if logger is not None and hasattr(logger, "warning"):
|
||||||
|
logger.warning(f"fallback 档案 '{fb_name}' 构造失败,已忽略:{e}")
|
||||||
|
return FailoverClient(chain, logger=logger,
|
||||||
|
timeout_budget=profile.get("failover_timeout_budget"))
|
||||||
|
|
||||||
if provider in ("deepseek", "deepseek-chat"):
|
if provider in ("deepseek", "deepseek-chat", "openai_compatible", "openai"):
|
||||||
return DeepSeekClient(
|
client = DeepSeekClient(
|
||||||
api_key=resolved_key,
|
api_key=resolved_key,
|
||||||
model=model,
|
model=model,
|
||||||
base_url=base_url,
|
base_url=base_url,
|
||||||
@@ -267,6 +721,7 @@ def create_llm_client(config, api_key: Optional[str] = None,
|
|||||||
config=config,
|
config=config,
|
||||||
logger=logger,
|
logger=logger,
|
||||||
)
|
)
|
||||||
|
return client
|
||||||
|
|
||||||
# 其他 OpenAI 兼容厂商:可直接复用 DeepSeekClient(继承其协议解析),
|
# 其他 OpenAI 兼容厂商:可直接复用 DeepSeekClient(继承其协议解析),
|
||||||
# 或在此接入专有实现。新增厂商时在此扩展即可。
|
# 或在此接入专有实现。新增厂商时在此扩展即可。
|
||||||
|
|||||||
@@ -0,0 +1,415 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
大模型接入点(模型档案)交互式管理工具
|
||||||
|
|
||||||
|
不用手写 config.json:菜单式列出 / 添加 / 删除 / 连通性验证
|
||||||
|
`models` 节的模型接入点(OpenAI Chat Completions 兼容端点:
|
||||||
|
DeepSeek / Kimi / Qwen / GLM / 本地 Ollama 等)。
|
||||||
|
|
||||||
|
用法:
|
||||||
|
python model_config.py # 交互式菜单
|
||||||
|
python model_config.py --list # 非交互:列出当前接入点
|
||||||
|
|
||||||
|
安全约定(与 aibox 审计一致):API Key 一律不写入配置文件——
|
||||||
|
只保存环境变量名(api_key_env),或留空表示免密本地端点。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
|
||||||
|
# 允许从仓库根目录或 aibox/ 目录启动
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
|
from common import ConfigManager # noqa: E402
|
||||||
|
|
||||||
|
PROFILE_NAME_RE = re.compile(r"^[A-Za-z0-9_-]+$")
|
||||||
|
|
||||||
|
# 常见厂商端点提示(首行是默认建议值)
|
||||||
|
PROVIDER_HINTS = {
|
||||||
|
"DeepSeek": "https://api.deepseek.com/v1/chat/completions",
|
||||||
|
"Kimi (Moonshot)": "https://api.moonshot.cn/v1/chat/completions",
|
||||||
|
"Qwen (阿里云百炼)": "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions",
|
||||||
|
"GLM (智谱)": "https://open.bigmodel.cn/api/paas/v4/chat/completions",
|
||||||
|
"Ollama 本地": "http://localhost:11434/v1/chat/completions",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 核心逻辑(可单测,不依赖 input) ====================
|
||||||
|
|
||||||
|
def get_models(config: ConfigManager) -> Dict:
|
||||||
|
"""取 models 节(不存在则初始化为空 dict)。"""
|
||||||
|
models = config.get("models") or {}
|
||||||
|
if not isinstance(models, dict):
|
||||||
|
models = {}
|
||||||
|
return models
|
||||||
|
|
||||||
|
|
||||||
|
def list_profiles(config: ConfigManager) -> List[Dict]:
|
||||||
|
"""列出所有档案的归一化展示数据。"""
|
||||||
|
models = get_models(config)
|
||||||
|
default = config.get("api.default_model") or ""
|
||||||
|
profiles = []
|
||||||
|
for name, profile in models.items():
|
||||||
|
if not isinstance(profile, dict):
|
||||||
|
continue
|
||||||
|
profiles.append({
|
||||||
|
"name": name,
|
||||||
|
"is_default": name == default,
|
||||||
|
"display_name": profile.get("display_name") or name,
|
||||||
|
"api_url": profile.get("api_url", ""),
|
||||||
|
"model": profile.get("model", ""),
|
||||||
|
"api_key_env": profile.get("api_key_env", ""),
|
||||||
|
"reasoning_enabled": bool(profile.get("reasoning_enabled", False)),
|
||||||
|
"temperature": profile.get("temperature"),
|
||||||
|
"timeout": profile.get("timeout"),
|
||||||
|
"fallback": profile.get("fallback") or [],
|
||||||
|
})
|
||||||
|
return profiles
|
||||||
|
|
||||||
|
|
||||||
|
def validate_new_profile(config: ConfigManager, name: str, api_url: str,
|
||||||
|
model: str) -> Optional[str]:
|
||||||
|
"""校验新档案字段,返回错误信息(合法返回 None)。"""
|
||||||
|
if not name:
|
||||||
|
return "档案名不能为空"
|
||||||
|
if not PROFILE_NAME_RE.match(name):
|
||||||
|
return f"档案名只能包含字母/数字/下划线/短横线(当前: {name})"
|
||||||
|
models = get_models(config)
|
||||||
|
if name in models:
|
||||||
|
return f"档案名 '{name}' 已存在"
|
||||||
|
if not api_url or not str(api_url).strip().lower().startswith(("http://", "https://")):
|
||||||
|
return "API 地址必须是 http:// 或 https:// 开头的完整 URL"
|
||||||
|
if not model or not str(model).strip():
|
||||||
|
return "模型名不能为空(如 deepseek-chat / gpt-4o / qwen-plus)"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def add_profile(config: ConfigManager, name: str, api_url: str, model: str,
|
||||||
|
api_key_env: str = "", display_name: str = None,
|
||||||
|
temperature: float = None, timeout: int = None,
|
||||||
|
reasoning_enabled: bool = False,
|
||||||
|
fallback: Optional[List[str]] = None,
|
||||||
|
set_default: bool = False) -> List[str]:
|
||||||
|
"""添加档案。返回提示信息列表;字段非法时返回带 ❌ 的错误信息。"""
|
||||||
|
err = validate_new_profile(config, name, api_url, model)
|
||||||
|
if err:
|
||||||
|
return [f"❌ {err}"]
|
||||||
|
|
||||||
|
models = get_models(config)
|
||||||
|
profile = {
|
||||||
|
"api_url": str(api_url).strip(),
|
||||||
|
"model": str(model).strip(),
|
||||||
|
}
|
||||||
|
if api_key_env:
|
||||||
|
profile["api_key_env"] = api_key_env.strip()
|
||||||
|
else:
|
||||||
|
# 空串 = 免密本地端点(Ollama/vLLM),显式声明
|
||||||
|
profile["api_key_env"] = ""
|
||||||
|
if display_name:
|
||||||
|
profile["display_name"] = display_name.strip()
|
||||||
|
if temperature is not None:
|
||||||
|
try:
|
||||||
|
profile["temperature"] = max(0.0, min(2.0, float(temperature)))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return ["❌ 温度必须是 0.0~2.0 的数字"]
|
||||||
|
if timeout is not None:
|
||||||
|
try:
|
||||||
|
profile["timeout"] = max(5, int(timeout))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return ["❌ 超时必须是整数秒"]
|
||||||
|
if reasoning_enabled:
|
||||||
|
profile["reasoning_enabled"] = True
|
||||||
|
if fallback:
|
||||||
|
# 只保留已存在的档案,避免悬空引用(L4a 启动期硬性报错)
|
||||||
|
known = set(models.keys())
|
||||||
|
invalid = [f for f in fallback if f not in known or f == name]
|
||||||
|
if invalid:
|
||||||
|
return [f"❌ 失败转移包含不存在的档案或自身: {invalid},请重新选择"]
|
||||||
|
profile["fallback"] = list(fallback)
|
||||||
|
|
||||||
|
models[name] = profile
|
||||||
|
config._config["models"] = models
|
||||||
|
if set_default:
|
||||||
|
config._config.setdefault("api", {})["default_model"] = name
|
||||||
|
|
||||||
|
saved = config._save_config()
|
||||||
|
msgs = [f"✅ 接入点 '{name}' 已添加({profile['api_url']},模型 {profile['model']})"]
|
||||||
|
if not saved:
|
||||||
|
msgs.append("⚠️ 保存失败(仅本次运行生效,重启后丢失)——请检查配置文件目录权限")
|
||||||
|
if set_default:
|
||||||
|
msgs.append(f"⭐ 已设为默认接入点(--model 不指定时使用)")
|
||||||
|
if not api_key_env:
|
||||||
|
msgs.append("ℹ️ 免密端点:请求不携带 API Key(适用于 Ollama/vLLM 等本地服务)")
|
||||||
|
else:
|
||||||
|
msgs.append(f"🔑 Key 从环境变量 {api_key_env} 读取(未设置时该档案不可用)")
|
||||||
|
return msgs
|
||||||
|
|
||||||
|
|
||||||
|
def delete_profile(config: ConfigManager, name: str) -> List[str]:
|
||||||
|
"""删除档案,并自动清理其他档案 fallback 中的悬空引用。"""
|
||||||
|
models = get_models(config)
|
||||||
|
if name not in models:
|
||||||
|
return [f"❌ 档案 '{name}' 不存在"]
|
||||||
|
|
||||||
|
del models[name]
|
||||||
|
|
||||||
|
# 清理其他档案对该档案的 fallback 引用(悬空引用会导致启动期报错)
|
||||||
|
cleaned = []
|
||||||
|
for other, profile in models.items():
|
||||||
|
if not isinstance(profile, dict):
|
||||||
|
continue
|
||||||
|
fb = profile.get("fallback") or []
|
||||||
|
if name in fb:
|
||||||
|
profile["fallback"] = [f for f in fb if f != name]
|
||||||
|
cleaned.append(other)
|
||||||
|
config._config["models"] = models
|
||||||
|
|
||||||
|
# 若删除的是默认档案,清除默认标记(回退旧 api 节语义)
|
||||||
|
if config.get("api.default_model") == name:
|
||||||
|
config._config.setdefault("api", {}).pop("default_model", None)
|
||||||
|
|
||||||
|
saved = config._save_config()
|
||||||
|
msgs = [f"✅ 接入点 '{name}' 已删除"]
|
||||||
|
if cleaned:
|
||||||
|
msgs.append(f"ℹ️ 已同时清理 {len(cleaned)} 个档案的失败转移引用: {', '.join(cleaned)}")
|
||||||
|
if not saved:
|
||||||
|
msgs.append("⚠️ 保存失败——请检查配置文件目录权限")
|
||||||
|
return msgs
|
||||||
|
|
||||||
|
|
||||||
|
def test_profile(config: ConfigManager, name: str, logger=None) -> str:
|
||||||
|
"""连通性验证:向该档案端点发一次最小请求。"""
|
||||||
|
models = get_models(config)
|
||||||
|
if name not in models:
|
||||||
|
return f"❌ 档案 '{name}' 不存在"
|
||||||
|
try:
|
||||||
|
from llm_client import create_llm_client
|
||||||
|
client = create_llm_client(config, api_key=None, logger=logger, model=name)
|
||||||
|
result = client.chat_completion(
|
||||||
|
[{"role": "user", "content": "ping"}], max_tokens=5)
|
||||||
|
if result is None or not str(result).strip():
|
||||||
|
return f"❌ 档案 '{name}' 验证失败:端点返回空响应(多为 4xx 认证/参数错误)"
|
||||||
|
return f"✅ 档案 '{name}' 连通正常:{str(result).strip()[:60]}"
|
||||||
|
except Exception as e:
|
||||||
|
return f"❌ 档案 '{name}' 验证失败:{str(e)}"
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 交互式菜单 ====================
|
||||||
|
|
||||||
|
def _ask(prompt: str, default: str = "") -> str:
|
||||||
|
if default:
|
||||||
|
val = input(f"{prompt}(回车默认: {default}): ").strip()
|
||||||
|
return val if val else default
|
||||||
|
return input(f"{prompt}: ").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _ask_yes_no(prompt: str, default: bool = False) -> bool:
|
||||||
|
hint = "y/n" if not default else "Y/n"
|
||||||
|
while True:
|
||||||
|
val = input(f"{prompt} [{hint}] ").strip().lower()
|
||||||
|
if not val:
|
||||||
|
return default
|
||||||
|
if val in ("y", "yes", "是"):
|
||||||
|
return True
|
||||||
|
if val in ("n", "no", "否"):
|
||||||
|
return False
|
||||||
|
print(" 请输入 y 或 n")
|
||||||
|
|
||||||
|
|
||||||
|
def _choose(prompt: str, options: List[str], multi: bool = False) -> List[str]:
|
||||||
|
"""编号选择(单选或多选)。多选用逗号分隔,直接回车 = 不选。"""
|
||||||
|
for i, opt in enumerate(options, 1):
|
||||||
|
print(f" {i}. {opt}")
|
||||||
|
if multi:
|
||||||
|
while True:
|
||||||
|
raw = input(f"{prompt}(多选,逗号分隔,直接回车不选): ").strip()
|
||||||
|
if not raw:
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
idxs = [int(x) for x in raw.replace(",", ",").split(",") if x.strip()]
|
||||||
|
except ValueError:
|
||||||
|
print(" 请输入数字编号,逗号分隔")
|
||||||
|
continue
|
||||||
|
if all(1 <= i <= len(options) for i in idxs):
|
||||||
|
return [options[i - 1] for i in idxs]
|
||||||
|
print(f" 编号超出范围(1~{len(options)})")
|
||||||
|
else:
|
||||||
|
while True:
|
||||||
|
raw = input(f"{prompt}(1~{len(options)},直接回车不选): ").strip()
|
||||||
|
if not raw:
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
idx = int(raw)
|
||||||
|
except ValueError:
|
||||||
|
print(" 请输入数字编号")
|
||||||
|
continue
|
||||||
|
if 1 <= idx <= len(options):
|
||||||
|
return [options[idx - 1]]
|
||||||
|
print(f" 编号超出范围(1~{len(options)})")
|
||||||
|
|
||||||
|
|
||||||
|
def _print_profiles(profiles: List[Dict]):
|
||||||
|
if not profiles:
|
||||||
|
print(" (当前没有任何接入点。旧版 api 节仍可用,或选 2 添加)")
|
||||||
|
return
|
||||||
|
for p in profiles:
|
||||||
|
mark = " ⭐默认" if p["is_default"] else ""
|
||||||
|
key_desc = f"环境变量 {p['api_key_env']}" if p["api_key_env"] else "免密"
|
||||||
|
fb = f",失败转移 → {', '.join(p['fallback'])}" if p["fallback"] else ""
|
||||||
|
print(f" {p['name']}{mark} [{p['display_name']}]")
|
||||||
|
print(f" 地址: {p['api_url']} 模型: {p['model']}")
|
||||||
|
print(f" Key: {key_desc} 温度: {p['temperature'] or '默认'} 超时: {p['timeout'] or '默认'}{fb}")
|
||||||
|
|
||||||
|
|
||||||
|
def _add_flow(config: ConfigManager):
|
||||||
|
print("\n── 添加接入点 ──")
|
||||||
|
print("常见厂商端点参考:")
|
||||||
|
for k, v in PROVIDER_HINTS.items():
|
||||||
|
print(f" {k}: {v}")
|
||||||
|
name = _ask("档案名(英文,如 deepseek/kimi/local)")
|
||||||
|
if not name:
|
||||||
|
print("❌ 已取消(档案名不能为空)")
|
||||||
|
return
|
||||||
|
if not PROFILE_NAME_RE.match(name):
|
||||||
|
print(f"❌ 档案名只能包含字母/数字/下划线/短横线")
|
||||||
|
return
|
||||||
|
if name in get_models(config):
|
||||||
|
print(f"❌ 档案名 '{name}' 已存在")
|
||||||
|
return
|
||||||
|
api_url = _ask("API 地址")
|
||||||
|
model = _ask("模型名(如 deepseek-chat / qwen-plus)")
|
||||||
|
|
||||||
|
print("Key 方式:")
|
||||||
|
print(" 1. 环境变量名(推荐,如 DEEPSEEK_API_KEY——Key 不落盘,重启不丢)")
|
||||||
|
print(" 2. 免密本地端点(如 Ollama,不需要 Key)")
|
||||||
|
key_mode = _ask("选择 [1/2]", "1")
|
||||||
|
api_key_env = ""
|
||||||
|
if key_mode == "1":
|
||||||
|
api_key_env = _ask("环境变量名", "DEEPSEEK_API_KEY")
|
||||||
|
elif key_mode != "2":
|
||||||
|
print("❌ 已取消(Key 方式无效)")
|
||||||
|
return
|
||||||
|
|
||||||
|
display_name = _ask("显示名(可选,回车用档案名)")
|
||||||
|
temperature_raw = _ask("温度 0.0~2.0(可选,越低越确定)")
|
||||||
|
timeout_raw = _ask("超时秒数(可选)")
|
||||||
|
reasoning = _ask_yes_no("启用思考模式?", default=False)
|
||||||
|
|
||||||
|
profiles = list_profiles(config)
|
||||||
|
fallback = []
|
||||||
|
if profiles:
|
||||||
|
print("失败转移:主接入点故障时按顺序尝试的备用接入点(可选)")
|
||||||
|
fallback = _choose("选择备用接入点", [p["name"] for p in profiles], multi=True)
|
||||||
|
|
||||||
|
set_default = _ask_yes_no("设为默认接入点?", default=(len(profiles) == 0))
|
||||||
|
|
||||||
|
print("\n── 确认 ──")
|
||||||
|
print(f" 档案名: {name}")
|
||||||
|
print(f" 地址: {api_url} 模型: {model}")
|
||||||
|
print(f" Key: {'环境变量 ' + api_key_env if api_key_env else '免密(不携带)'}")
|
||||||
|
if display_name:
|
||||||
|
print(f" 显示名: {display_name}")
|
||||||
|
if temperature_raw:
|
||||||
|
print(f" 温度: {temperature_raw}")
|
||||||
|
if timeout_raw:
|
||||||
|
print(f" 超时: {timeout_raw}s")
|
||||||
|
if reasoning:
|
||||||
|
print(" 思考模式: 开启")
|
||||||
|
if fallback:
|
||||||
|
print(f" 失败转移: {', '.join(fallback)}")
|
||||||
|
if not _ask_yes_no("确认保存?", default=True):
|
||||||
|
print("已取消")
|
||||||
|
return
|
||||||
|
|
||||||
|
msgs = add_profile(config, name, api_url, model,
|
||||||
|
api_key_env=api_key_env, display_name=display_name,
|
||||||
|
temperature=float(temperature_raw) if temperature_raw else None,
|
||||||
|
timeout=int(timeout_raw) if timeout_raw else None,
|
||||||
|
reasoning_enabled=reasoning, fallback=fallback,
|
||||||
|
set_default=set_default)
|
||||||
|
for m in msgs:
|
||||||
|
print(m)
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_flow(config: ConfigManager):
|
||||||
|
profiles = list_profiles(config)
|
||||||
|
if not profiles:
|
||||||
|
print("\n(当前没有任何接入点)")
|
||||||
|
return
|
||||||
|
print("\n── 删除接入点 ──")
|
||||||
|
picked = _choose("选择要删除的接入点", [p["name"] for p in profiles])
|
||||||
|
if not picked:
|
||||||
|
print("已取消")
|
||||||
|
return
|
||||||
|
name = picked[0]
|
||||||
|
if not _ask_yes_no(f"确认删除 '{name}'?(其他档案的失败转移引用会自动清理)", default=False):
|
||||||
|
print("已取消")
|
||||||
|
return
|
||||||
|
for m in delete_profile(config, name):
|
||||||
|
print(m)
|
||||||
|
|
||||||
|
|
||||||
|
def _test_flow(config: ConfigManager):
|
||||||
|
profiles = list_profiles(config)
|
||||||
|
if not profiles:
|
||||||
|
print("\n(当前没有任何接入点)")
|
||||||
|
return
|
||||||
|
print("\n── 连通性验证 ──")
|
||||||
|
picked = _choose("选择要验证的接入点", [p["name"] for p in profiles])
|
||||||
|
if not picked:
|
||||||
|
print("已取消")
|
||||||
|
return
|
||||||
|
print(" 正在请求端点(最长 20 秒)...")
|
||||||
|
print(test_profile(config, picked[0]))
|
||||||
|
|
||||||
|
|
||||||
|
def run_interactive(config: ConfigManager = None):
|
||||||
|
"""交互式主循环。config 可注入(测试用)。"""
|
||||||
|
config = config or ConfigManager()
|
||||||
|
config_path = getattr(config, "_config_path", None) or "(默认位置)"
|
||||||
|
print("=" * 56)
|
||||||
|
print(" aibox 大模型接入点管理工具")
|
||||||
|
print(f" 配置文件: {config_path}")
|
||||||
|
print(" 说明:API Key 只存环境变量名,不落盘;免密端点适合 Ollama 等本地服务")
|
||||||
|
print("=" * 56)
|
||||||
|
while True:
|
||||||
|
print("\n请选择操作:")
|
||||||
|
print(" 1. 📋 列出接入点")
|
||||||
|
print(" 2. ➕ 添加接入点")
|
||||||
|
print(" 3. 🗑️ 删除接入点")
|
||||||
|
print(" 4. 🔍 验证接入点连通性")
|
||||||
|
print(" 0. 退出")
|
||||||
|
choice = input("> ").strip()
|
||||||
|
if choice == "1":
|
||||||
|
_print_profiles(list_profiles(config))
|
||||||
|
elif choice == "2":
|
||||||
|
_add_flow(config)
|
||||||
|
elif choice == "3":
|
||||||
|
_delete_flow(config)
|
||||||
|
elif choice == "4":
|
||||||
|
_test_flow(config)
|
||||||
|
elif choice in ("0", "q", "exit", "退出"):
|
||||||
|
print("再见")
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
print("无效选择,请输入 0~4")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description="aibox 大模型接入点管理工具")
|
||||||
|
parser.add_argument("--list", action="store_true", help="非交互:列出当前接入点")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
config = ConfigManager()
|
||||||
|
if args.list:
|
||||||
|
_print_profiles(list_profiles(config))
|
||||||
|
return
|
||||||
|
run_interactive(config)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -514,7 +514,8 @@ class ConversationCompressor:
|
|||||||
final_tokens = self.estimate_tokens([m for m in final_messages if m.get("role") != "system"])
|
final_tokens = self.estimate_tokens([m for m in final_messages if m.get("role") != "system"])
|
||||||
|
|
||||||
print(f"\n✅ {i18n.get('conversation_compressed', original_count, final_count)}")
|
print(f"\n✅ {i18n.get('conversation_compressed', original_count, final_count)}")
|
||||||
print(f" Token: {original_tokens} -> {final_tokens} (压缩率: {final_tokens/original_tokens:.1%})")
|
# L6:压缩块全空时 original_tokens 可能为 0,除零防护
|
||||||
|
print(f" Token: {original_tokens} -> {final_tokens} (压缩率: {final_tokens / max(original_tokens, 1):.1%})")
|
||||||
|
|
||||||
# 检查是否压缩到阈值以下
|
# 检查是否压缩到阈值以下
|
||||||
if final_count >= threshold:
|
if final_count >= threshold:
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ subsystems.conversation_history - 对话历史管理器
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import time
|
import time
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -61,14 +62,26 @@ class ConversationHistory:
|
|||||||
try:
|
try:
|
||||||
with open(self.history_file, 'r', encoding='utf-8') as f:
|
with open(self.history_file, 'r', encoding='utf-8') as f:
|
||||||
return json.load(f)
|
return json.load(f)
|
||||||
except:
|
except Exception as e:
|
||||||
|
# N16:收窄为 Exception——裸 except 连 KeyboardInterrupt/SystemExit
|
||||||
|
# 也吞掉;损坏时重置为空索引并告警
|
||||||
|
self.logger.warning(f"历史索引文件损坏,已重置: {e}")
|
||||||
return {"conversations": [], "last_id": 0}
|
return {"conversations": [], "last_id": 0}
|
||||||
return {"conversations": [], "last_id": 0}
|
return {"conversations": [], "last_id": 0}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _atomic_write_json(path: Path, data) -> None:
|
||||||
|
"""原子写 JSON(I 批,审计 2026-08-18):先写同目录临时文件再
|
||||||
|
os.replace——此前 'w' 直写,进程中断/崩溃留半截 JSON(current 损坏
|
||||||
|
后仅静默置 None,历史内容丢失)"""
|
||||||
|
tmp = path.with_suffix(path.suffix + ".tmp")
|
||||||
|
with open(tmp, 'w', encoding='utf-8') as f:
|
||||||
|
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
def _save_history_index(self):
|
def _save_history_index(self):
|
||||||
"""保存历史索引"""
|
"""保存历史索引"""
|
||||||
with open(self.history_file, 'w', encoding='utf-8') as f:
|
self._atomic_write_json(self.history_file, self.history_index)
|
||||||
json.dump(self.history_index, f, ensure_ascii=False, indent=2)
|
|
||||||
|
|
||||||
def _load_current_conversation(self):
|
def _load_current_conversation(self):
|
||||||
"""加载当前的对话(如果有)"""
|
"""加载当前的对话(如果有)"""
|
||||||
@@ -83,8 +96,7 @@ class ConversationHistory:
|
|||||||
def save_current_conversation(self):
|
def save_current_conversation(self):
|
||||||
"""保存当前对话"""
|
"""保存当前对话"""
|
||||||
if self.current_conversation:
|
if self.current_conversation:
|
||||||
with open(self.current_file, 'w', encoding='utf-8') as f:
|
self._atomic_write_json(self.current_file, self.current_conversation)
|
||||||
json.dump(self.current_conversation, f, ensure_ascii=False, indent=2)
|
|
||||||
|
|
||||||
def start_new_conversation(self, topic: str = None) -> Dict:
|
def start_new_conversation(self, topic: str = None) -> Dict:
|
||||||
"""开始新对话"""
|
"""开始新对话"""
|
||||||
@@ -132,12 +144,17 @@ class ConversationHistory:
|
|||||||
importance=2
|
importance=2
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# N16:归档文件名加序号防同秒覆盖——DATE_FORMAT 精度为秒,
|
||||||
|
# 同一秒内两次归档(如测试/快速重置)会静默覆盖前一份
|
||||||
timestamp = datetime.now().strftime(constants.DATE_FORMAT)
|
timestamp = datetime.now().strftime(constants.DATE_FORMAT)
|
||||||
archive_filename = f"conversation_{timestamp}.json"
|
archive_filename = f"conversation_{timestamp}.json"
|
||||||
|
counter = 1
|
||||||
|
while (self.archive_dir / archive_filename).exists():
|
||||||
|
archive_filename = f"conversation_{timestamp}_{counter}.json"
|
||||||
|
counter += 1
|
||||||
archive_path = self.archive_dir / archive_filename
|
archive_path = self.archive_dir / archive_filename
|
||||||
|
|
||||||
with open(archive_path, 'w', encoding='utf-8') as f:
|
self._atomic_write_json(archive_path, self.current_conversation)
|
||||||
json.dump(self.current_conversation, f, ensure_ascii=False, indent=2)
|
|
||||||
|
|
||||||
self.history_index["conversations"].append({
|
self.history_index["conversations"].append({
|
||||||
"id": self.current_conversation["id"],
|
"id": self.current_conversation["id"],
|
||||||
@@ -341,7 +358,13 @@ class ConversationHistory:
|
|||||||
if file_name == constants.CURRENT_CONVERSATION_FILE and self.current_conversation:
|
if file_name == constants.CURRENT_CONVERSATION_FILE and self.current_conversation:
|
||||||
return self.current_conversation
|
return self.current_conversation
|
||||||
|
|
||||||
file_path = self.archive_dir / file_name
|
# I 批(审计 2026-08-18):file_name 目前无 LLM 可控调用面,但一旦
|
||||||
|
# 接入即成为任意 .json 读取通道——realpath 解析软链后再做目录前缀校验
|
||||||
|
archive_root = os.path.normcase(os.path.realpath(str(self.archive_dir)))
|
||||||
|
target = os.path.normcase(os.path.realpath(str(self.archive_dir / file_name)))
|
||||||
|
if not target.startswith(archive_root + os.sep):
|
||||||
|
return None
|
||||||
|
file_path = Path(target)
|
||||||
if file_path.exists():
|
if file_path.exists():
|
||||||
with open(file_path, 'r', encoding='utf-8') as f:
|
with open(file_path, 'r', encoding='utf-8') as f:
|
||||||
return json.load(f)
|
return json.load(f)
|
||||||
|
|||||||
+102
-55
@@ -36,6 +36,8 @@ class MemoryDatabase:
|
|||||||
def _init_database(self):
|
def _init_database(self):
|
||||||
"""初始化数据库表"""
|
"""初始化数据库表"""
|
||||||
with sqlite3.connect(self.db_path) as conn:
|
with sqlite3.connect(self.db_path) as conn:
|
||||||
|
# WAL 模式:读写并发不互斥,缓解多 CLI 进程并发写锁等待
|
||||||
|
conn.execute("PRAGMA journal_mode=WAL")
|
||||||
cursor = conn.cursor()
|
cursor = conn.cursor()
|
||||||
cursor.execute('''
|
cursor.execute('''
|
||||||
CREATE TABLE IF NOT EXISTS memories (
|
CREATE TABLE IF NOT EXISTS memories (
|
||||||
@@ -174,7 +176,10 @@ class MemoryDatabase:
|
|||||||
def _search_via_fts(self, cursor, keywords: List[str], match_all: bool, limit: int) -> List[Dict]:
|
def _search_via_fts(self, cursor, keywords: List[str], match_all: bool, limit: int) -> List[Dict]:
|
||||||
"""FTS5 全文检索(关键词含特殊字符导致 MATCH 语法错误时返回空,走 LIKE 兜底)"""
|
"""FTS5 全文检索(关键词含特殊字符导致 MATCH 语法错误时返回空,走 LIKE 兜底)"""
|
||||||
joiner = " AND " if match_all else " OR "
|
joiner = " AND " if match_all else " OR "
|
||||||
query = joiner.join([f'"{kw}"' for kw in keywords])
|
# N7:短语内双引号翻倍转义——关键词含 " 时若不翻倍会提前闭合短语
|
||||||
|
# 边界,破坏 MATCH 语法或改变匹配语义(FTS5 短语引号转义规则)
|
||||||
|
terms = [f'"{kw.replace(chr(34), chr(34) * 2)}"' for kw in keywords]
|
||||||
|
query = joiner.join(terms)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
cursor.execute('''
|
cursor.execute('''
|
||||||
@@ -227,14 +232,15 @@ class MemoryDatabase:
|
|||||||
return results
|
return results
|
||||||
|
|
||||||
def get_recent_memories(self, limit: int = 10) -> List[Dict]:
|
def get_recent_memories(self, limit: int = 10) -> List[Dict]:
|
||||||
"""获取最近的记忆。limit=None 时不限制返回条数(供分页使用)。"""
|
"""获取最近的记忆。limit=None 时使用默认上限(M4:此前 None 无 LIMIT
|
||||||
|
全表加载,记忆库增长后单次调用 fetchall 全量入内存;与 search_memories
|
||||||
|
的 None→默认上限语义对齐)。"""
|
||||||
|
if limit is None:
|
||||||
|
limit = constants.MEMORY_MAX_SEARCH_RESULTS
|
||||||
with sqlite3.connect(self.db_path) as conn:
|
with sqlite3.connect(self.db_path) as conn:
|
||||||
conn.row_factory = sqlite3.Row
|
conn.row_factory = sqlite3.Row
|
||||||
cursor = conn.cursor()
|
cursor = conn.cursor()
|
||||||
if limit is None:
|
cursor.execute('SELECT * FROM memories ORDER BY end_time DESC LIMIT ?', (limit,))
|
||||||
cursor.execute('SELECT * FROM memories ORDER BY end_time DESC')
|
|
||||||
else:
|
|
||||||
cursor.execute('SELECT * FROM memories ORDER BY end_time DESC LIMIT ?', (limit,))
|
|
||||||
|
|
||||||
results = []
|
results = []
|
||||||
for row in cursor.fetchall():
|
for row in cursor.fetchall():
|
||||||
@@ -396,11 +402,14 @@ class MemoDatabase:
|
|||||||
# 如果传入了 before_time,使用它,否则使用当前时间
|
# 如果传入了 before_time,使用它,否则使用当前时间
|
||||||
compare_time = before_time if before_time else now
|
compare_time = before_time if before_time else now
|
||||||
|
|
||||||
|
# I 批(审计 2026-08-18):去掉 SQL 层 reminder_time <= ? 预过滤——
|
||||||
|
# 字符串比较会把格式异常的时间(如 "不是时间")静默排除在 SQL 层,
|
||||||
|
# Python 二次验证永远看不到它们(Q5:格式异常任务按到期返回)。
|
||||||
|
# 时间过滤改由下方 Python 解析做权威判断(本地小库全扫描可忽略)。
|
||||||
query = '''
|
query = '''
|
||||||
SELECT * FROM memos
|
SELECT * FROM memos
|
||||||
WHERE is_completed = 0
|
WHERE is_completed = 0
|
||||||
AND reminder_time IS NOT NULL
|
AND reminder_time IS NOT NULL
|
||||||
AND reminder_time <= ? -- 严格小于等于比较时间
|
|
||||||
AND (repeat_end_time IS NULL OR repeat_end_time >= ?)
|
AND (repeat_end_time IS NULL OR repeat_end_time >= ?)
|
||||||
'''
|
'''
|
||||||
|
|
||||||
@@ -409,10 +418,13 @@ class MemoDatabase:
|
|||||||
|
|
||||||
query += ' ORDER BY reminder_time ASC, priority DESC'
|
query += ' ORDER BY reminder_time ASC, priority DESC'
|
||||||
|
|
||||||
params = [compare_time, compare_time]
|
params = [compare_time]
|
||||||
|
|
||||||
if limit:
|
# I 批:limit=0 语义修正——此前 `if limit:` 使 0 等同"不限",
|
||||||
query += f' LIMIT {limit}'
|
# 与 None=不限的惯例冲突;改为 None 才不加 LIMIT
|
||||||
|
if limit is not None:
|
||||||
|
query += ' LIMIT ?'
|
||||||
|
params.append(max(0, int(limit)))
|
||||||
|
|
||||||
cursor.execute(query, params)
|
cursor.execute(query, params)
|
||||||
|
|
||||||
@@ -433,7 +445,11 @@ class MemoDatabase:
|
|||||||
# 调试日志
|
# 调试日志
|
||||||
self.logger.debug(f"过滤掉未到期任务 #{memo['id']}: {memo['reminder_time']} > {compare_time}")
|
self.logger.debug(f"过滤掉未到期任务 #{memo['id']}: {memo['reminder_time']} > {compare_time}")
|
||||||
except (ValueError, TypeError) as e:
|
except (ValueError, TypeError) as e:
|
||||||
self.logger.warning(f"解析任务 #{memo['id']} 的提醒时间失败: {e}")
|
# I 批(审计 2026-08-18):格式异常此前被静默丢弃——
|
||||||
|
# 任务永不执行且用户无感知。改为按"到期"返回并告警,
|
||||||
|
# 至少保证任务被处理而非永久丢失
|
||||||
|
self.logger.warning(f"解析任务 #{memo['id']} 的提醒时间失败,按到期处理: {e}")
|
||||||
|
results.append(memo)
|
||||||
else:
|
else:
|
||||||
# 没有提醒时间的任务不应该在待执行列表中
|
# 没有提醒时间的任务不应该在待执行列表中
|
||||||
self.logger.warning(f"任务 #{memo['id']} 没有提醒时间,但被返回为待执行")
|
self.logger.warning(f"任务 #{memo['id']} 没有提醒时间,但被返回为待执行")
|
||||||
@@ -453,10 +469,14 @@ class MemoDatabase:
|
|||||||
ORDER BY created_at ASC
|
ORDER BY created_at ASC
|
||||||
'''
|
'''
|
||||||
|
|
||||||
if limit:
|
params = []
|
||||||
query += f' LIMIT {limit}'
|
# I 批:limit=0 语义修正——此前 `if limit:` 使 0 等同"不限",
|
||||||
|
# 与 None=不限的惯例冲突;改为 None 才不加 LIMIT
|
||||||
|
if limit is not None:
|
||||||
|
query += ' LIMIT ?'
|
||||||
|
params.append(max(0, int(limit)))
|
||||||
|
|
||||||
cursor.execute(query)
|
cursor.execute(query, params)
|
||||||
|
|
||||||
results = []
|
results = []
|
||||||
for row in cursor.fetchall():
|
for row in cursor.fetchall():
|
||||||
@@ -506,64 +526,82 @@ class MemoDatabase:
|
|||||||
|
|
||||||
def mark_as_triggered(self, memo_id: int) -> Optional[str]:
|
def mark_as_triggered(self, memo_id: int) -> Optional[str]:
|
||||||
"""标记任务已被触发,并根据重复规则计算下一次提醒时间"""
|
"""标记任务已被触发,并根据重复规则计算下一次提醒时间"""
|
||||||
memo = self.get_memo(memo_id)
|
# N10:读取与更新收进同一连接的 BEGIN IMMEDIATE 事务。此前先
|
||||||
if not memo:
|
# get_memo(独立连接)后 UPDATE,check-then-act 之间存在竞态:
|
||||||
return None
|
# 调度器并发触发同一任务会双计数/双推进 repeat。IMMEDIATE 在事务
|
||||||
|
# 开始即取写锁,串行化并发触发。
|
||||||
now = datetime.now()
|
now = datetime.now()
|
||||||
next_reminder = None
|
next_reminder = None
|
||||||
|
|
||||||
with sqlite3.connect(self.db_path) as conn:
|
with sqlite3.connect(self.db_path) as conn:
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
cursor = conn.cursor()
|
cursor = conn.cursor()
|
||||||
|
# I 批(审计 2026-08-18):多进程同库(CLI --memo-add + 服务调度)
|
||||||
|
# 写竞争时默认 5s 后直接报 database is locked——busy_timeout 让
|
||||||
|
# SQLite 等待锁释放(最长 10s)而不是立即失败
|
||||||
|
cursor.execute("PRAGMA busy_timeout = 10000")
|
||||||
|
cursor.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
cursor.execute('SELECT * FROM memos WHERE id = ?', (memo_id,))
|
||||||
|
row = cursor.fetchone()
|
||||||
|
if row is None:
|
||||||
|
conn.rollback()
|
||||||
|
return None
|
||||||
|
memo = dict(row)
|
||||||
|
if memo.get('tags'):
|
||||||
|
memo['tags'] = memo['tags'].split(',')
|
||||||
|
|
||||||
cursor.execute('''
|
cursor.execute('''
|
||||||
UPDATE memos
|
UPDATE memos
|
||||||
SET last_triggered_at = ?, trigger_count = trigger_count + 1
|
SET last_triggered_at = ?, trigger_count = trigger_count + 1
|
||||||
WHERE id = ?
|
WHERE id = ?
|
||||||
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
||||||
|
|
||||||
# 处理重复任务
|
# 处理重复任务
|
||||||
if memo['repeat_type'] != 'none' and not self._should_end_repeat(memo, now):
|
if memo['repeat_type'] != 'none' and not self._should_end_repeat(memo, now):
|
||||||
# 使用当前时间作为基准来计算下一次执行时间
|
# 使用当前时间作为基准来计算下一次执行时间
|
||||||
next_reminder = self._calculate_next_reminder(memo, now)
|
next_reminder = self._calculate_next_reminder(memo, now)
|
||||||
|
|
||||||
if next_reminder:
|
if next_reminder:
|
||||||
cursor.execute('''
|
cursor.execute('''
|
||||||
UPDATE memos
|
UPDATE memos
|
||||||
SET reminder_time = ?
|
SET reminder_time = ?
|
||||||
WHERE id = ?
|
WHERE id = ?
|
||||||
''', (next_reminder.strftime(constants.DATETIME_FORMAT), memo_id))
|
''', (next_reminder.strftime(constants.DATETIME_FORMAT), memo_id))
|
||||||
|
|
||||||
self.logger.info(f"任务 #{memo_id} 已更新下一次执行时间: {next_reminder}")
|
self.logger.info(f"任务 #{memo_id} 已更新下一次执行时间: {next_reminder}")
|
||||||
else:
|
else:
|
||||||
# 重复结束,标记为完成
|
# 重复结束,标记为完成
|
||||||
|
cursor.execute('''
|
||||||
|
UPDATE memos
|
||||||
|
SET is_completed = 1, completed_at = ?
|
||||||
|
WHERE id = ?
|
||||||
|
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
||||||
|
|
||||||
|
self.logger.info(f"任务 #{memo_id} 重复结束,已标记为完成")
|
||||||
|
elif memo['is_immediate']:
|
||||||
|
# 即时任务执行后标记为完成
|
||||||
cursor.execute('''
|
cursor.execute('''
|
||||||
UPDATE memos
|
UPDATE memos
|
||||||
SET is_completed = 1, completed_at = ?
|
SET is_completed = 1, completed_at = ?
|
||||||
WHERE id = ?
|
WHERE id = ?
|
||||||
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
||||||
|
|
||||||
self.logger.info(f"任务 #{memo_id} 重复结束,已标记为完成")
|
self.logger.info(f"即时任务 #{memo_id} 已完成")
|
||||||
elif memo['is_immediate']:
|
else:
|
||||||
# 即时任务执行后标记为完成
|
# 一次性任务,标记为完成
|
||||||
cursor.execute('''
|
cursor.execute('''
|
||||||
UPDATE memos
|
UPDATE memos
|
||||||
SET is_completed = 1, completed_at = ?
|
SET is_completed = 1, completed_at = ?
|
||||||
WHERE id = ?
|
WHERE id = ?
|
||||||
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
||||||
|
|
||||||
self.logger.info(f"即时任务 #{memo_id} 已完成")
|
self.logger.info(f"任务 #{memo_id} 已完成")
|
||||||
else:
|
|
||||||
# 一次性任务,标记为完成
|
|
||||||
cursor.execute('''
|
|
||||||
UPDATE memos
|
|
||||||
SET is_completed = 1, completed_at = ?
|
|
||||||
WHERE id = ?
|
|
||||||
''', (now.strftime(constants.DATETIME_FORMAT), memo_id))
|
|
||||||
|
|
||||||
self.logger.info(f"任务 #{memo_id} 已完成")
|
conn.commit()
|
||||||
|
except Exception:
|
||||||
conn.commit()
|
conn.rollback()
|
||||||
|
raise
|
||||||
|
|
||||||
return next_reminder.strftime(constants.DATETIME_FORMAT) if next_reminder else None
|
return next_reminder.strftime(constants.DATETIME_FORMAT) if next_reminder else None
|
||||||
|
|
||||||
@@ -650,6 +688,15 @@ class MemoDatabase:
|
|||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def delete_memo(self, memo_id: int) -> bool:
|
||||||
|
"""物理删除任务(M5:delete 语义此前被 complete_memo 顶替——只标记
|
||||||
|
完成不删行,重复任务还会被推进而非移除,stats 计数失真)"""
|
||||||
|
with sqlite3.connect(self.db_path) as conn:
|
||||||
|
cursor = conn.cursor()
|
||||||
|
cursor.execute("DELETE FROM memos WHERE id = ?", (memo_id,))
|
||||||
|
conn.commit()
|
||||||
|
return cursor.rowcount > 0
|
||||||
|
|
||||||
def get_memo_stats(self) -> Dict:
|
def get_memo_stats(self) -> Dict:
|
||||||
"""获取任务统计信息"""
|
"""获取任务统计信息"""
|
||||||
with sqlite3.connect(self.db_path) as conn:
|
with sqlite3.connect(self.db_path) as conn:
|
||||||
|
|||||||
+148
-84
@@ -13,6 +13,7 @@ subsystems.executors - 命令执行器
|
|||||||
|
|
||||||
import os
|
import os
|
||||||
import queue
|
import queue
|
||||||
|
import signal
|
||||||
import subprocess
|
import subprocess
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
@@ -71,62 +72,148 @@ class SubprocessCommandExecutor(CommandExecutor):
|
|||||||
def __init__(self, logger=None):
|
def __init__(self, logger=None):
|
||||||
super().__init__(logger)
|
super().__init__(logger)
|
||||||
self._current_process = None
|
self._current_process = None
|
||||||
|
self._current_pgid = None # L9:Popen 后立即捕获,shell 死亡后 getpgid 会失败
|
||||||
self._process_lock = threading.Lock()
|
self._process_lock = threading.Lock()
|
||||||
|
|
||||||
|
def _terminate_tree(self, proc, force: bool = False):
|
||||||
|
"""终止整个进程组(L9:POSIX 下 shell=True 只杀 shell 时,孙进程
|
||||||
|
(`(x &)` 等)仍持管道写端——reader 阻塞无 EOF、fd/线程悬挂至孙
|
||||||
|
进程退出,写满 64KB 管道缓冲后孙进程也卡死。start_new_session 使
|
||||||
|
shell 成为会话首进程,按组信号可一次覆盖整棵进程树)。
|
||||||
|
组 ID 在 Popen 后立即保存:shell 死亡后 os.getpgid(pid) 会抛
|
||||||
|
ProcessLookupError,届时无法再从进程反查组。
|
||||||
|
Windows 无进程组概念,退回单进程 terminate/kill。"""
|
||||||
|
try:
|
||||||
|
if os.name == "posix":
|
||||||
|
sig = signal.SIGKILL if force else signal.SIGTERM
|
||||||
|
with self._process_lock:
|
||||||
|
pgid = self._current_pgid
|
||||||
|
if pgid is None:
|
||||||
|
pgid = os.getpgid(proc.pid)
|
||||||
|
os.killpg(pgid, sig)
|
||||||
|
elif force:
|
||||||
|
proc.kill()
|
||||||
|
else:
|
||||||
|
proc.terminate()
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
proc.kill()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
def cancel(self):
|
def cancel(self):
|
||||||
"""取消当前正在运行的命令(kill子进程)。线程安全。"""
|
"""取消当前正在运行的命令(kill子进程组)。线程安全。"""
|
||||||
with self._process_lock:
|
with self._process_lock:
|
||||||
proc = self._current_process
|
proc = self._current_process
|
||||||
if proc is not None:
|
if proc is not None:
|
||||||
try:
|
try:
|
||||||
if proc.poll() is None:
|
if proc.poll() is None:
|
||||||
proc.kill()
|
self._terminate_tree(proc, force=True)
|
||||||
self.logger.info("已终止正在运行的子进程")
|
self.logger.info("已终止正在运行的子进程组")
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
def _read_with_limit(self, process, timeout: int, start_time: float, output_queue=None) -> Tuple[str, bool, bool]:
|
def _read_with_limit(self, process, timeout: int, start_time: float, output_queue=None) -> Tuple[str, bool, bool]:
|
||||||
"""从进程读取输出,限制总大小和超时。
|
"""从进程读取输出,限制总大小和超时。返回 (output_str, timeout_occurred, truncated)。
|
||||||
返回 (output_str, timeout_occurred, truncated)
|
|
||||||
|
H2 修复:阻塞 readline 在独立 reader 线程执行,主循环带超时轮询队列。
|
||||||
|
此前超时检查只在两次 readline 之间执行,无输出命令会让 readline
|
||||||
|
阻塞到进程自然退出,timeout 形同虚设(审计 PoC 实证)。
|
||||||
"""
|
"""
|
||||||
output_chunks = []
|
output_chunks = []
|
||||||
total_bytes = 0
|
total_bytes = 0
|
||||||
timeout_occurred = False
|
timeout_occurred = False
|
||||||
truncated = False
|
truncated = False
|
||||||
|
|
||||||
|
line_queue: queue.Queue = queue.Queue()
|
||||||
|
reader_done = threading.Event()
|
||||||
|
|
||||||
|
def _reader():
|
||||||
|
"""后台 reader 线程:阻塞 readline,行入队;EOF/异常结束"""
|
||||||
|
try:
|
||||||
|
for line in iter(process.stdout.readline, ""):
|
||||||
|
line_queue.put(line)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
reader_done.set()
|
||||||
|
|
||||||
|
reader = threading.Thread(target=_reader, daemon=True)
|
||||||
|
reader.start()
|
||||||
|
|
||||||
|
shell_dead_since = None # shell 进程已死但 reader 未收到 EOF 的起始时刻
|
||||||
while True:
|
while True:
|
||||||
# 检查超时
|
# 检查超时(readline 阻塞不再影响本判断)
|
||||||
if timeout > 0 and time.time() - start_time > timeout:
|
if timeout > 0 and time.time() - start_time > timeout:
|
||||||
process.terminate()
|
self._terminate_tree(process)
|
||||||
timeout_occurred = True
|
timeout_occurred = True
|
||||||
break
|
break
|
||||||
|
|
||||||
# 检查输出大小
|
# 检查输出大小
|
||||||
if total_bytes >= self.MAX_OUTPUT_BYTES:
|
if total_bytes >= self.MAX_OUTPUT_BYTES:
|
||||||
truncated = True
|
truncated = True
|
||||||
process.terminate()
|
self._terminate_tree(process)
|
||||||
break
|
break
|
||||||
|
|
||||||
# 读取一行
|
try:
|
||||||
line = process.stdout.readline()
|
line = line_queue.get(timeout=0.1)
|
||||||
if not line and process.poll() is not None:
|
shell_dead_since = None
|
||||||
break
|
except queue.Empty:
|
||||||
if line:
|
if reader_done.is_set():
|
||||||
line_bytes = len(line.encode('utf-8', errors='replace'))
|
|
||||||
if total_bytes + line_bytes > self.MAX_OUTPUT_BYTES and not truncated:
|
|
||||||
allowed = self.MAX_OUTPUT_BYTES - total_bytes
|
|
||||||
if allowed > 0:
|
|
||||||
output_chunks.append(line.encode('utf-8', errors='replace')[:allowed].decode('utf-8', errors='replace'))
|
|
||||||
total_bytes = self.MAX_OUTPUT_BYTES
|
|
||||||
truncated = True
|
|
||||||
process.terminate()
|
|
||||||
break
|
break
|
||||||
|
# shell 已被 kill/退出但 reader 仍阻塞(如 Windows shell=True
|
||||||
|
# 下 cancel 只杀死包装进程,孙进程仍持有管道写端导致无 EOF):
|
||||||
|
# 宽限 1 秒排空残留输出后放弃等待
|
||||||
|
if process.poll() is not None:
|
||||||
|
if shell_dead_since is None:
|
||||||
|
shell_dead_since = time.time()
|
||||||
|
elif time.time() - shell_dead_since > 1.0:
|
||||||
|
# L9:shell 已死但孙进程仍持管道写端(`(sleep 300 &) ;
|
||||||
|
# echo done` 形态)。只靠 daemon reader 兜底会悬挂线程
|
||||||
|
# 与管道 fd 至孙进程退出,写满 64KB 缓冲后孙进程自身
|
||||||
|
# 也卡死。此处强杀进程组收尾(nohup 重定向输出的
|
||||||
|
# 真正脱离场景不持管道,不会走到这里)。
|
||||||
|
self._terminate_tree(process, force=True)
|
||||||
|
break
|
||||||
|
continue
|
||||||
|
|
||||||
output_chunks.append(line)
|
line_bytes = len(line.encode('utf-8', errors='replace'))
|
||||||
total_bytes += line_bytes
|
if total_bytes + line_bytes > self.MAX_OUTPUT_BYTES and not truncated:
|
||||||
|
allowed = self.MAX_OUTPUT_BYTES - total_bytes
|
||||||
|
if allowed > 0:
|
||||||
|
# L5:decode 用 errors='ignore'——按字节截断可能落在多字节
|
||||||
|
# 字符中间,errors='replace' 会在行尾留下半个字符(�)
|
||||||
|
output_chunks.append(line.encode('utf-8', errors='replace')[:allowed].decode('utf-8', errors='ignore'))
|
||||||
|
total_bytes = self.MAX_OUTPUT_BYTES
|
||||||
|
truncated = True
|
||||||
|
self._terminate_tree(process)
|
||||||
|
break
|
||||||
|
|
||||||
if output_queue:
|
output_chunks.append(line)
|
||||||
output_queue.put(("line", line.rstrip('\n') + "\n"))
|
total_bytes += line_bytes
|
||||||
|
|
||||||
|
if output_queue:
|
||||||
|
output_queue.put(("line", line.rstrip('\n') + "\n"))
|
||||||
|
|
||||||
|
# N4 修复(审计 2026-08-18,二次修正):reader 线程收尾。
|
||||||
|
# 超时/截断/宽限到期 break 后,reader 可能仍阻塞在 readline
|
||||||
|
# (Windows shell=True 下孙进程持管道写端,永远等不到 EOF)。
|
||||||
|
# 注意:此时不能 close stdout——CPython 流对象内部有锁,
|
||||||
|
# readline 持锁阻塞时 close() 会一同挂起直到 readline 返回
|
||||||
|
# (实测命令超时被拖满整个 Start-Sleep 时长)。reader 为 daemon
|
||||||
|
# 线程,孙进程退出收到 EOF 后自然结束,不阻塞主流程;
|
||||||
|
# reader 已正常退出的场景才 close 显式释放管道 fd。
|
||||||
|
if reader_done.is_set():
|
||||||
|
try:
|
||||||
|
process.stdout.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
reader.join(timeout=2)
|
||||||
|
if reader.is_alive():
|
||||||
|
self.logger.warning(
|
||||||
|
"reader 线程仍阻塞在管道读取(孙进程持写端),已放弃等待,"
|
||||||
|
"线程由 daemon 兜底、随管道 EOF 自然退出")
|
||||||
|
|
||||||
return ''.join(output_chunks), timeout_occurred, truncated
|
return ''.join(output_chunks), timeout_occurred, truncated
|
||||||
|
|
||||||
@@ -153,30 +240,35 @@ class SubprocessCommandExecutor(CommandExecutor):
|
|||||||
shell=True,
|
shell=True,
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.STDOUT,
|
stderr=subprocess.STDOUT,
|
||||||
stdin=subprocess.PIPE,
|
stdin=subprocess.DEVNULL, # H3 修复:不挂 stdin,防等待输入命令永久挂起
|
||||||
text=True,
|
text=True,
|
||||||
bufsize=1,
|
bufsize=1,
|
||||||
universal_newlines=True,
|
universal_newlines=True,
|
||||||
env=exec_env,
|
env=exec_env,
|
||||||
cwd=os.path.expanduser(work_dir)
|
cwd=os.path.expanduser(work_dir),
|
||||||
|
# L9:POSIX 下 shell 成为会话首进程,超时/取消可按进程组
|
||||||
|
# 一次终止整棵进程树(含 & 后台孙进程),消除管道悬挂
|
||||||
|
start_new_session=(os.name == "posix")
|
||||||
)
|
)
|
||||||
# 保存进程引用以便cancel()可以kill它
|
# 保存进程引用以便cancel()可以kill它
|
||||||
with self._process_lock:
|
with self._process_lock:
|
||||||
self._current_process = process
|
self._current_process = process
|
||||||
|
if os.name == "posix":
|
||||||
|
try:
|
||||||
|
self._current_pgid = os.getpgid(process.pid)
|
||||||
|
except Exception:
|
||||||
|
self._current_pgid = None
|
||||||
|
|
||||||
start_time = time.time()
|
start_time = time.time()
|
||||||
output_str, timeout_occurred, truncated = self._read_with_limit(process, timeout, start_time)
|
output_str, timeout_occurred, truncated = self._read_with_limit(process, timeout, start_time)
|
||||||
|
|
||||||
# 确保进程终止
|
# 确保进程终止
|
||||||
if process.poll() is None:
|
if process.poll() is None:
|
||||||
if timeout_occurred or truncated:
|
self._terminate_tree(process, force=timeout_occurred or truncated)
|
||||||
process.kill()
|
|
||||||
else:
|
|
||||||
process.terminate()
|
|
||||||
try:
|
try:
|
||||||
process.wait(timeout=2)
|
process.wait(timeout=2)
|
||||||
except:
|
except:
|
||||||
process.kill()
|
self._terminate_tree(process, force=True)
|
||||||
|
|
||||||
return_code = process.returncode if process.returncode is not None else -1
|
return_code = process.returncode if process.returncode is not None else -1
|
||||||
|
|
||||||
@@ -204,10 +296,11 @@ class SubprocessCommandExecutor(CommandExecutor):
|
|||||||
return f"❌ 执行错误: {str(e)}"
|
return f"❌ 执行错误: {str(e)}"
|
||||||
finally:
|
finally:
|
||||||
if process and process.poll() is None:
|
if process and process.poll() is None:
|
||||||
process.kill()
|
self._terminate_tree(process, force=True)
|
||||||
# 清除当前进程引用
|
# 清除当前进程引用
|
||||||
with self._process_lock:
|
with self._process_lock:
|
||||||
self._current_process = None
|
self._current_process = None
|
||||||
|
self._current_pgid = None
|
||||||
|
|
||||||
def execute_streaming(self, command: str, output_queue: queue.Queue, timeout: int = 30,
|
def execute_streaming(self, command: str, output_queue: queue.Queue, timeout: int = 30,
|
||||||
work_dir: str = None, env: Dict = None, **kwargs):
|
work_dir: str = None, env: Dict = None, **kwargs):
|
||||||
@@ -240,67 +333,39 @@ class SubprocessCommandExecutor(CommandExecutor):
|
|||||||
shell=True,
|
shell=True,
|
||||||
stdout=subprocess.PIPE,
|
stdout=subprocess.PIPE,
|
||||||
stderr=subprocess.STDOUT,
|
stderr=subprocess.STDOUT,
|
||||||
stdin=subprocess.PIPE,
|
stdin=subprocess.DEVNULL, # H3 修复:不挂 stdin,防等待输入命令永久挂起
|
||||||
text=True,
|
text=True,
|
||||||
bufsize=1,
|
bufsize=1,
|
||||||
universal_newlines=True,
|
universal_newlines=True,
|
||||||
env=exec_env,
|
env=exec_env,
|
||||||
cwd=os.path.expanduser(work_dir)
|
cwd=os.path.expanduser(work_dir),
|
||||||
|
# L9:POSIX 下 shell 成为会话首进程,超时/取消可按进程组
|
||||||
|
# 一次终止整棵进程树(含 & 后台孙进程),消除管道悬挂
|
||||||
|
start_new_session=(os.name == "posix")
|
||||||
)
|
)
|
||||||
self.process = process
|
self.process = process
|
||||||
# 保存进程引用以便cancel()可以kill它
|
# 保存进程引用以便cancel()可以kill它
|
||||||
with self._process_lock:
|
with self._process_lock:
|
||||||
self._current_process = process
|
self._current_process = process
|
||||||
|
if os.name == "posix":
|
||||||
|
try:
|
||||||
|
self._current_pgid = os.getpgid(process.pid)
|
||||||
|
except Exception:
|
||||||
|
self._current_pgid = None
|
||||||
|
|
||||||
start_time = time.time()
|
start_time = time.time()
|
||||||
output_lines = []
|
# H2 修复:复用线程化读取(reader 线程 + 超时轮询),行经 output_queue 流式转发
|
||||||
total_bytes = 0
|
output_str, timeout_occurred, truncated = self._read_with_limit(
|
||||||
timeout_occurred = False
|
process, timeout, start_time, output_queue=output_queue)
|
||||||
truncated = False
|
|
||||||
|
|
||||||
# 实时读取输出
|
if timeout_occurred:
|
||||||
while True:
|
output_queue.put(("line", f"\n⏰ 命令执行超时({timeout}秒)\n"))
|
||||||
# 检查超时
|
elif truncated:
|
||||||
if timeout > 0 and time.time() - start_time > timeout:
|
output_queue.put(("line", f"\n📦 输出超过 {self.MAX_OUTPUT_BYTES/1024/1024:.0f}MB 限制,结果已截断\n"))
|
||||||
process.terminate()
|
|
||||||
timeout_occurred = True
|
|
||||||
output_queue.put(("line", f"\n⏰ 命令执行超时({timeout}秒)\n"))
|
|
||||||
break
|
|
||||||
|
|
||||||
# 检查输出大小
|
|
||||||
if total_bytes >= self.MAX_OUTPUT_BYTES:
|
|
||||||
truncated = True
|
|
||||||
process.terminate()
|
|
||||||
output_queue.put(("line", f"\n📦 输出超过 {self.MAX_OUTPUT_BYTES/1024/1024:.0f}MB 限制,结果已截断\n"))
|
|
||||||
break
|
|
||||||
|
|
||||||
# 读取一行输出
|
|
||||||
line = process.stdout.readline()
|
|
||||||
if not line and process.poll() is not None:
|
|
||||||
break
|
|
||||||
if line:
|
|
||||||
line_bytes = len(line.encode('utf-8', errors='replace'))
|
|
||||||
|
|
||||||
if total_bytes + line_bytes > self.MAX_OUTPUT_BYTES and not truncated:
|
|
||||||
allowed = self.MAX_OUTPUT_BYTES - total_bytes
|
|
||||||
if allowed > 0:
|
|
||||||
truncated_line = line.encode('utf-8', errors='replace')[:allowed].decode('utf-8', errors='replace')
|
|
||||||
output_queue.put(("line", truncated_line))
|
|
||||||
output_lines.append(truncated_line)
|
|
||||||
total_bytes = self.MAX_OUTPUT_BYTES
|
|
||||||
truncated = True
|
|
||||||
process.terminate()
|
|
||||||
output_queue.put(("line", f"\n📦 输出超过 {self.MAX_OUTPUT_BYTES/1024/1024:.0f}MB 限制,结果已截断\n"))
|
|
||||||
break
|
|
||||||
|
|
||||||
stripped = line.rstrip('\n')
|
|
||||||
output_queue.put(("line", stripped + "\n"))
|
|
||||||
output_lines.append(line)
|
|
||||||
total_bytes += line_bytes
|
|
||||||
|
|
||||||
# 如果超时或截断,确保进程被终止
|
# 如果超时或截断,确保进程被终止
|
||||||
if (timeout_occurred or truncated) and process.poll() is None:
|
if (timeout_occurred or truncated) and process.poll() is None:
|
||||||
process.kill()
|
self._terminate_tree(process, force=True)
|
||||||
try:
|
try:
|
||||||
process.wait(timeout=2)
|
process.wait(timeout=2)
|
||||||
except:
|
except:
|
||||||
@@ -317,10 +382,8 @@ class SubprocessCommandExecutor(CommandExecutor):
|
|||||||
else:
|
else:
|
||||||
output_queue.put(("line", f"✅ 命令执行完成,返回码: {return_code}\n"))
|
output_queue.put(("line", f"✅ 命令执行完成,返回码: {return_code}\n"))
|
||||||
|
|
||||||
# 准备完整结果
|
|
||||||
full_output = "".join(output_lines)
|
|
||||||
data = {
|
data = {
|
||||||
"输出": full_output.strip() if full_output.strip() else "(无输出)",
|
"输出": output_str.strip() if output_str.strip() else "(无输出)",
|
||||||
"返回码": return_code
|
"返回码": return_code
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -339,11 +402,12 @@ class SubprocessCommandExecutor(CommandExecutor):
|
|||||||
output_queue.put(("complete", f"❌ 执行错误: {str(e)}"))
|
output_queue.put(("complete", f"❌ 执行错误: {str(e)}"))
|
||||||
finally:
|
finally:
|
||||||
if process and process.poll() is None:
|
if process and process.poll() is None:
|
||||||
process.terminate()
|
self._terminate_tree(process)
|
||||||
try:
|
try:
|
||||||
process.wait(timeout=2)
|
process.wait(timeout=2)
|
||||||
except:
|
except:
|
||||||
process.kill()
|
self._terminate_tree(process, force=True)
|
||||||
self.process = None
|
self.process = None
|
||||||
with self._process_lock:
|
with self._process_lock:
|
||||||
self._current_process = None
|
self._current_process = None
|
||||||
|
self._current_pgid = None
|
||||||
@@ -18,6 +18,7 @@ subsystems.prompt_manager - 提示词管理器
|
|||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
|
import threading
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from typing import Dict, List, Optional
|
from typing import Dict, List, Optional
|
||||||
|
|
||||||
@@ -35,6 +36,7 @@ class PromptManager:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
_instance = None
|
_instance = None
|
||||||
|
_instance_lock = threading.Lock() # I 批:并发首例化竞态防护
|
||||||
_prompts = []
|
_prompts = []
|
||||||
_prompts_dir = None
|
_prompts_dir = None
|
||||||
|
|
||||||
@@ -43,12 +45,18 @@ class PromptManager:
|
|||||||
STATUS_FILE = "100_STATUS.md"
|
STATUS_FILE = "100_STATUS.md"
|
||||||
SOUL_FILE = "99_SOUL.md" # 智能体灵魂文件 - 所有角色定位内容
|
SOUL_FILE = "99_SOUL.md" # 智能体灵魂文件 - 所有角色定位内容
|
||||||
|
|
||||||
|
# N9(审计 2026-08-18):Agent 可控内容直写 00/99/100 文件并作为后续
|
||||||
|
# 会话系统提示词加载(持久化注入面)。内容长度上限约束注入规模,
|
||||||
|
# 超长内容拒绝写入(工具层有更友好的提示)。
|
||||||
|
MAX_CONTENT_LENGTH = 10000
|
||||||
|
|
||||||
# 基础技能文件名模式:N_xxx.md(N 为数字,如 1_memory_search.md、10_task_handling.md)
|
# 基础技能文件名模式:N_xxx.md(N 为数字,如 1_memory_search.md、10_task_handling.md)
|
||||||
_SKILL_FILE_RE = re.compile(r'^(\d+)_')
|
_SKILL_FILE_RE = re.compile(r'^(\d+)_')
|
||||||
|
|
||||||
def __new__(cls):
|
def __new__(cls):
|
||||||
if cls._instance is None:
|
with cls._instance_lock:
|
||||||
cls._instance = super().__new__(cls)
|
if cls._instance is None:
|
||||||
|
cls._instance = super().__new__(cls)
|
||||||
return cls._instance
|
return cls._instance
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
@@ -60,10 +68,16 @@ class PromptManager:
|
|||||||
prompts_dir = config.get("system.prompts_dir", "~/.aibox/prompts")
|
prompts_dir = config.get("system.prompts_dir", "~/.aibox/prompts")
|
||||||
return os.path.expanduser(prompts_dir)
|
return os.path.expanduser(prompts_dir)
|
||||||
|
|
||||||
def _load_prompts(self):
|
def _load_prompts(self, target: Optional[List[Dict]] = None):
|
||||||
|
"""加载提示词到 target 列表(缺省 self._prompts)。
|
||||||
|
|
||||||
|
M6:reload 传入新列表本地构建,完成后由 reload 原子替换——
|
||||||
|
此前先清空 self._prompts 再逐文件重读,并发读者(其他连接构建
|
||||||
|
system 消息)会读到空/半截提示词列表。
|
||||||
|
"""
|
||||||
|
prompts = target if target is not None else self._prompts
|
||||||
self._prompts_dir = self._get_prompts_dir()
|
self._prompts_dir = self._get_prompts_dir()
|
||||||
os.makedirs(self._prompts_dir, exist_ok=True)
|
os.makedirs(self._prompts_dir, exist_ok=True)
|
||||||
self._prompts = []
|
|
||||||
|
|
||||||
# 确保核心文件存在
|
# 确保核心文件存在
|
||||||
self._ensure_identity_file()
|
self._ensure_identity_file()
|
||||||
@@ -86,17 +100,14 @@ class PromptManager:
|
|||||||
with open(filepath, 'r', encoding='utf-8') as f:
|
with open(filepath, 'r', encoding='utf-8') as f:
|
||||||
content = f.read().strip()
|
content = f.read().strip()
|
||||||
if content:
|
if content:
|
||||||
self._prompts.append({
|
prompts.append({
|
||||||
'file': filename,
|
'file': filename,
|
||||||
'content': content
|
'content': content
|
||||||
})
|
})
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"⚠️ 读取提示词文件 {filename} 失败: {e}")
|
print(f"⚠️ 读取提示词文件 {filename} 失败: {e}")
|
||||||
|
|
||||||
essential_files = [self.IDENTITY_FILE, self.STATUS_FILE, self.SOUL_FILE]
|
if not prompts:
|
||||||
other_files = [f for f in prompt_files if f not in essential_files] if prompt_files else []
|
|
||||||
|
|
||||||
if not self._prompts:
|
|
||||||
self._create_default_prompts()
|
self._create_default_prompts()
|
||||||
|
|
||||||
def _get_template_dir(self) -> str:
|
def _get_template_dir(self) -> str:
|
||||||
@@ -148,6 +159,9 @@ class PromptManager:
|
|||||||
|
|
||||||
def update_identity(self, content: str) -> bool:
|
def update_identity(self, content: str) -> bool:
|
||||||
"""更新用户身份信息"""
|
"""更新用户身份信息"""
|
||||||
|
if len(content or "") > self.MAX_CONTENT_LENGTH:
|
||||||
|
print(f"⚠️ 身份内容超长({len(content)} > {self.MAX_CONTENT_LENGTH} 字符),拒绝写入")
|
||||||
|
return False
|
||||||
identity_path = os.path.join(self._prompts_dir, self.IDENTITY_FILE)
|
identity_path = os.path.join(self._prompts_dir, self.IDENTITY_FILE)
|
||||||
try:
|
try:
|
||||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
@@ -168,6 +182,9 @@ class PromptManager:
|
|||||||
|
|
||||||
def update_status(self, content: str) -> bool:
|
def update_status(self, content: str) -> bool:
|
||||||
"""更新智能体状态"""
|
"""更新智能体状态"""
|
||||||
|
if len(content or "") > self.MAX_CONTENT_LENGTH:
|
||||||
|
print(f"⚠️ 状态内容超长({len(content)} > {self.MAX_CONTENT_LENGTH} 字符),拒绝写入")
|
||||||
|
return False
|
||||||
status_path = os.path.join(self._prompts_dir, self.STATUS_FILE)
|
status_path = os.path.join(self._prompts_dir, self.STATUS_FILE)
|
||||||
try:
|
try:
|
||||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
@@ -188,6 +205,9 @@ class PromptManager:
|
|||||||
|
|
||||||
def update_soul(self, content: str) -> bool:
|
def update_soul(self, content: str) -> bool:
|
||||||
"""更新智能体灵魂定义 - 所有角色定位内容"""
|
"""更新智能体灵魂定义 - 所有角色定位内容"""
|
||||||
|
if len(content or "") > self.MAX_CONTENT_LENGTH:
|
||||||
|
print(f"⚠️ 灵魂内容超长({len(content)} > {self.MAX_CONTENT_LENGTH} 字符),拒绝写入")
|
||||||
|
return False
|
||||||
soul_path = os.path.join(self._prompts_dir, self.SOUL_FILE)
|
soul_path = os.path.join(self._prompts_dir, self.SOUL_FILE)
|
||||||
try:
|
try:
|
||||||
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
@@ -240,10 +260,6 @@ class PromptManager:
|
|||||||
return """我是DeepSeek,一个智能AI助手。
|
return """我是DeepSeek,一个智能AI助手。
|
||||||
请创建 99_SOUL.md 文件来定义我的完整身份、角色、性格和能力边界。"""
|
请创建 99_SOUL.md 文件来定义我的完整身份、角色、性格和能力边界。"""
|
||||||
|
|
||||||
def get_skill_prompts(self) -> List[Dict]:
|
|
||||||
"""获取所有基础技能提示词(以1_开头的文件)"""
|
|
||||||
return [p for p in self._prompts if p['file'].startswith(self.SKILL_PREFIX)]
|
|
||||||
|
|
||||||
def get_soul_prompt(self) -> Optional[Dict]:
|
def get_soul_prompt(self) -> Optional[Dict]:
|
||||||
"""获取灵魂提示词(99_SOUL.md)"""
|
"""获取灵魂提示词(99_SOUL.md)"""
|
||||||
for p in self._prompts:
|
for p in self._prompts:
|
||||||
@@ -296,9 +312,10 @@ class PromptManager:
|
|||||||
return [p for p in self._prompts if self._skill_sort_key(p['file']) is not None]
|
return [p for p in self._prompts if self._skill_sort_key(p['file']) is not None]
|
||||||
|
|
||||||
def reload(self):
|
def reload(self):
|
||||||
"""重新加载所有提示词"""
|
"""重新加载所有提示词(M6:构建新列表后原子替换,并发读者始终读到完整列表)"""
|
||||||
self._prompts = []
|
new_prompts: List[Dict] = []
|
||||||
self._load_prompts()
|
self._load_prompts(new_prompts)
|
||||||
|
self._prompts = new_prompts
|
||||||
return len(self._prompts)
|
return len(self._prompts)
|
||||||
|
|
||||||
def get_prompt_count(self) -> int:
|
def get_prompt_count(self) -> int:
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""审计修复 PoC 回归验证(H1/H2/H3 + 1MB + cancel)。
|
||||||
|
|
||||||
|
验证 2026-08-17 审计五项修复的实际运行效果:
|
||||||
|
PoC-1 (H2): 无输出命令 + timeout=3s → 应 ~3s 返回超时(修复前阻塞 15s+)
|
||||||
|
PoC-2 (H3): 等待 stdin 的命令 + timeout=3s → 应超时返回而非永久挂起
|
||||||
|
PoC-3 : 输出 >1MB → 应截断且总输出 ≈1MB
|
||||||
|
PoC-4 : cancel() 应在宽限时间内终止长命令
|
||||||
|
PoC-5 (H1) : CommandLineTool(工具层,非 executor)跑 >1MB 输出 →
|
||||||
|
截断生效即证明已走 executor 委托路径
|
||||||
|
"""
|
||||||
|
|
||||||
|
import queue
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import types
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
AIBOX_DIR = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(AIBOX_DIR))
|
||||||
|
sys.modules.setdefault("websockets", types.ModuleType("websockets"))
|
||||||
|
|
||||||
|
from subsystems.executors import SubprocessCommandExecutor
|
||||||
|
from tools.command_tool import CommandLineTool
|
||||||
|
|
||||||
|
results = []
|
||||||
|
|
||||||
|
|
||||||
|
def check(name, ok, detail):
|
||||||
|
results.append((name, ok, detail))
|
||||||
|
print(f"[{'PASS' if ok else 'FAIL'}] {name}: {detail}")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- PoC-1 (H2): 无输出命令超时 ----------
|
||||||
|
ex = SubprocessCommandExecutor()
|
||||||
|
t0 = time.time()
|
||||||
|
r = ex.execute("powershell -NoProfile -Command Start-Sleep 15", timeout=3)
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
check("PoC-1 无输出命令超时(H2)", elapsed < 6 and "超时" in r,
|
||||||
|
f"{elapsed:.1f}s 返回(修复前阻塞15s+),结果含超时: {'超时' in r}")
|
||||||
|
|
||||||
|
# ---------- PoC-2 (H3): 等待 stdin 的命令 ----------
|
||||||
|
# stdin=DEVNULL 下命令立即收到 EOF:要么快速退出,要么按 timeout 返回;
|
||||||
|
# 判定标准 = 不永久挂起(修复前 readline 无 EOF 永久阻塞)。
|
||||||
|
t0 = time.time()
|
||||||
|
r = ex.execute("powershell -NoProfile -Command Read-Host", timeout=3)
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
check("PoC-2 stdin等待命令不挂起(H3)", elapsed < 6,
|
||||||
|
f"{elapsed:.1f}s 返回(修复前永久挂起)")
|
||||||
|
|
||||||
|
# ---------- PoC-3: 1MB 输出截断 ----------
|
||||||
|
t0 = time.time()
|
||||||
|
r = ex.execute(
|
||||||
|
'powershell -NoProfile -Command "$s=\'x\'*2048; 1..1024 | ForEach-Object { $s }"',
|
||||||
|
timeout=60)
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
size_mb = len(r.encode("utf-8", errors="replace")) / 1024 / 1024
|
||||||
|
check("PoC-3 1MB输出截断", "截断" in r and size_mb < 1.15,
|
||||||
|
f"{elapsed:.1f}s,结果 {size_mb:.2f}MB,含截断标记: {'截断' in r}")
|
||||||
|
|
||||||
|
# ---------- PoC-4: cancel() 中断 ----------
|
||||||
|
out_q = queue.Queue()
|
||||||
|
done = threading.Event()
|
||||||
|
|
||||||
|
|
||||||
|
def _run():
|
||||||
|
ex.execute_streaming(
|
||||||
|
"powershell -NoProfile -Command Start-Sleep 60",
|
||||||
|
out_q, timeout=120)
|
||||||
|
done.set()
|
||||||
|
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
th = threading.Thread(target=_run, daemon=True)
|
||||||
|
th.start()
|
||||||
|
time.sleep(2)
|
||||||
|
ex.cancel()
|
||||||
|
done.wait(timeout=10)
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
check("PoC-4 cancel中断", done.is_set() and elapsed < 8,
|
||||||
|
f"cancel 后 {elapsed:.1f}s 结束(修复前杀不到进程需等满120s)")
|
||||||
|
|
||||||
|
# ---------- PoC-5 (H1): 工具层委托 executor(截断保护贯通) ----------
|
||||||
|
tool = CommandLineTool(executor=ex)
|
||||||
|
t0 = time.time()
|
||||||
|
r = tool.execute(
|
||||||
|
command='powershell -NoProfile -Command "$s=\'y\'*2048; 1..1024 | ForEach-Object { $s }"',
|
||||||
|
timeout=60)
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
size_mb = len(r.encode("utf-8", errors="replace")) / 1024 / 1024
|
||||||
|
check("PoC-5 工具层1MB贯通(H1)", "截断" in r and size_mb < 1.15,
|
||||||
|
f"{elapsed:.1f}s,工具层结果 {size_mb:.2f}MB(修复前无限累积)")
|
||||||
|
|
||||||
|
# ---------- 汇总 ----------
|
||||||
|
fails = [n for n, ok, _ in results if not ok]
|
||||||
|
print("\n==== 汇总: %d/%d PASS ====" % (len(results) - len(fails), len(results)))
|
||||||
|
sys.exit(1 if fails else 0)
|
||||||
@@ -13,6 +13,8 @@ from pathlib import Path
|
|||||||
# 把 aibox 目录加入 sys.path(无需安装为包即可 import tools/common)
|
# 把 aibox 目录加入 sys.path(无需安装为包即可 import tools/common)
|
||||||
AIBOX_DIR = Path(__file__).resolve().parent.parent
|
AIBOX_DIR = Path(__file__).resolve().parent.parent
|
||||||
sys.path.insert(0, str(AIBOX_DIR))
|
sys.path.insert(0, str(AIBOX_DIR))
|
||||||
|
# tests 目录也加入:测试文件互相导入(如 test_models_profile 复用 test_once_mode 的 _make_ai)
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
|
||||||
# 桩掉 websockets(aibox.py 顶部 import websockets,测试环境可能未安装)
|
# 桩掉 websockets(aibox.py 顶部 import websockets,测试环境可能未安装)
|
||||||
if "websockets" not in sys.modules:
|
if "websockets" not in sys.modules:
|
||||||
|
|||||||
+155
-1
@@ -15,7 +15,7 @@ from datetime import datetime, timedelta
|
|||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from common import constants
|
from common import ConfigManager, constants
|
||||||
from subsystems import (
|
from subsystems import (
|
||||||
MemoryDatabase, MemoDatabase, PromptManager, SkillManager,
|
MemoryDatabase, MemoDatabase, PromptManager, SkillManager,
|
||||||
SubprocessCommandExecutor,
|
SubprocessCommandExecutor,
|
||||||
@@ -425,6 +425,160 @@ def test_get_proxy_when_disabled():
|
|||||||
assert "未启用" in r or "启用" in r
|
assert "未启用" in r or "启用" in r
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- SetProxyTool 连通性验证(设置前验证,失败不落盘) ----------
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def fresh_config(tmp_path, monkeypatch):
|
||||||
|
"""隔离 ConfigManager 单例:AI_CONFIG_PATH 指向临时文件,前后重置类状态"""
|
||||||
|
cfg_file = tmp_path / "config.json"
|
||||||
|
monkeypatch.setenv("AI_CONFIG_PATH", str(cfg_file))
|
||||||
|
|
||||||
|
def _reset():
|
||||||
|
ConfigManager._instance = None
|
||||||
|
ConfigManager._config = None
|
||||||
|
ConfigManager._config_path = None
|
||||||
|
|
||||||
|
_reset()
|
||||||
|
yield ConfigManager
|
||||||
|
_reset()
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResp:
|
||||||
|
def __init__(self, status_code):
|
||||||
|
self.status_code = status_code
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_session_factory(exc=None, resp=None):
|
||||||
|
"""构造替代 requests.Session 的假类:get() 抛异常或返回假响应"""
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
class _FakeSession:
|
||||||
|
def __init__(self):
|
||||||
|
self.trust_env = True
|
||||||
|
|
||||||
|
def get(self, *args, **kwargs):
|
||||||
|
calls.append((args, kwargs))
|
||||||
|
if exc:
|
||||||
|
raise exc
|
||||||
|
return resp
|
||||||
|
|
||||||
|
_FakeSession.calls = calls
|
||||||
|
return _FakeSession
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_connectivity_ok(monkeypatch):
|
||||||
|
fake = _fake_session_factory(resp=_FakeResp(204))
|
||||||
|
monkeypatch.setattr("tools.proxy_tools.requests.Session", fake)
|
||||||
|
t = SetProxyTool()
|
||||||
|
ok, kind, detail = t._check_connectivity("127.0.0.1", 1080, None, None)
|
||||||
|
assert ok and kind == ""
|
||||||
|
# trust_env 必须关掉:排除系统 http_proxy 环境变量干扰
|
||||||
|
assert fake.calls, "应发出探测请求"
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_connectivity_classification(monkeypatch, tmp_path):
|
||||||
|
"""异常类型/消息 → 错误类型分类(与系统语言无关)"""
|
||||||
|
import requests as requests_lib
|
||||||
|
cases = [
|
||||||
|
# Windows 本地化的连接拒绝消息:靠异常类型而非英文关键词分类
|
||||||
|
(requests_lib.exceptions.ConnectionError(
|
||||||
|
"[WinError 10061] 由于目标计算机积极拒绝,无法连接。"), "connect_failed"),
|
||||||
|
# ConnectTimeout 同时继承 ConnectionError/Timeout,必须按 Timeout 归类
|
||||||
|
(requests_lib.exceptions.ConnectTimeout("connection timeout"), "timeout"),
|
||||||
|
(requests_lib.exceptions.ConnectionError(
|
||||||
|
"SOCKS5 authentication failed"), "auth_failed"),
|
||||||
|
(requests_lib.exceptions.ConnectionError(
|
||||||
|
"Missing dependencies for SOCKS support."), "missing_dependency"),
|
||||||
|
(Exception("Read timed out"), "timeout"), # 兜底:非 requests 异常按消息
|
||||||
|
(Exception("Connection refused"), "connect_failed"),
|
||||||
|
(Exception("something weird"), "error"),
|
||||||
|
]
|
||||||
|
t = SetProxyTool()
|
||||||
|
for exc, expect_kind in cases:
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"tools.proxy_tools.requests.Session",
|
||||||
|
_fake_session_factory(exc=exc))
|
||||||
|
ok, kind, _ = t._check_connectivity("127.0.0.1", 1080, "u", "p")
|
||||||
|
assert not ok
|
||||||
|
assert kind == expect_kind, f"{exc!r} 应分类为 {expect_kind},实际 {kind}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_connectivity_bad_status(monkeypatch):
|
||||||
|
"""探测端点返回非 200/204 → probe_failed"""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"tools.proxy_tools.requests.Session",
|
||||||
|
_fake_session_factory(resp=_FakeResp(502)))
|
||||||
|
t = SetProxyTool()
|
||||||
|
ok, kind, detail = t._check_connectivity("127.0.0.1", 1080, None, None)
|
||||||
|
assert not ok and kind == "probe_failed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_proxy_rejected_when_unreachable(fresh_config, monkeypatch):
|
||||||
|
"""验证失败:返回 ❌ 且配置不落盘"""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
SetProxyTool, "_check_connectivity",
|
||||||
|
lambda self, h, p, u, pw: (False, "connect_failed", "Connection refused"))
|
||||||
|
t = SetProxyTool()
|
||||||
|
r = t.execute(host="10.255.255.1", port=1080)
|
||||||
|
assert "❌" in r
|
||||||
|
assert "未保存" in r
|
||||||
|
# 配置未写入:代理仍未启用
|
||||||
|
assert fresh_config().get_proxy() is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_proxy_saved_when_check_passes(fresh_config, monkeypatch):
|
||||||
|
"""验证通过:配置保存且结果带连通性详情"""
|
||||||
|
monkeypatch.setattr(
|
||||||
|
SetProxyTool, "_check_connectivity",
|
||||||
|
lambda self, h, p, u, pw: (True, "", "探测成功 HTTP 204"))
|
||||||
|
t = SetProxyTool()
|
||||||
|
r = t.execute(host="127.0.0.1", port=1080)
|
||||||
|
assert "✅" in r
|
||||||
|
assert "探测成功" in r
|
||||||
|
proxies = fresh_config().get_proxy()
|
||||||
|
assert proxies and "127.0.0.1:1080" in proxies["http"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_proxy_skip_check_bypasses_validation(fresh_config, monkeypatch):
|
||||||
|
"""skip_check=True:不验证直接保存(代理未启动的预配置场景)"""
|
||||||
|
called = []
|
||||||
|
|
||||||
|
def _no_call(self, *a, **kw):
|
||||||
|
called.append(1)
|
||||||
|
raise AssertionError("skip_check=True 时不应调用验证")
|
||||||
|
|
||||||
|
monkeypatch.setattr(SetProxyTool, "_check_connectivity", _no_call)
|
||||||
|
t = SetProxyTool()
|
||||||
|
r = t.execute(host="127.0.0.1", port=1080, skip_check=True)
|
||||||
|
assert "✅" in r
|
||||||
|
assert not called
|
||||||
|
assert fresh_config().get_proxy() is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_proxy_disabled_skips_check(fresh_config, monkeypatch):
|
||||||
|
"""enabled=False:禁用代理无需验证"""
|
||||||
|
def _no_call(self, *a, **kw):
|
||||||
|
raise AssertionError("禁用代理不应触发验证")
|
||||||
|
|
||||||
|
monkeypatch.setattr(SetProxyTool, "_check_connectivity", _no_call)
|
||||||
|
t = SetProxyTool()
|
||||||
|
r = t.execute(host="127.0.0.1", port=1080, enabled=False)
|
||||||
|
assert "✅" in r
|
||||||
|
assert fresh_config().get_proxy() is None # 禁用态 get_proxy 返回 None
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_proxy_empty_env_hint(fresh_config, monkeypatch):
|
||||||
|
"""username + password_env 指向未设环境变量:精准提示而非裸 auth 失败"""
|
||||||
|
monkeypatch.delenv("AIBOX_TEST_SOCKS_PW", raising=False)
|
||||||
|
t = SetProxyTool()
|
||||||
|
r = t.execute(host="127.0.0.1", port=1080,
|
||||||
|
username="u1", password_env="AIBOX_TEST_SOCKS_PW")
|
||||||
|
assert "❌" in r
|
||||||
|
assert "环境变量" in r
|
||||||
|
assert "未保存" in r
|
||||||
|
assert fresh_config().get_proxy() is None
|
||||||
|
|
||||||
|
|
||||||
# ==================== AskUserTool ====================
|
# ==================== AskUserTool ====================
|
||||||
|
|
||||||
def test_ask_user_interface():
|
def test_ask_user_interface():
|
||||||
|
|||||||
@@ -0,0 +1,100 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""F1(N1 高危)修复回归测试:代理密码不落盘。
|
||||||
|
|
||||||
|
覆盖三点:
|
||||||
|
1. set_proxy(password=..., save=True) 后配置文件无明文密码(_save_config 脱敏)
|
||||||
|
2. password_env 解析:get_proxy() 从环境变量取密码拼 socks5:// URL
|
||||||
|
3. 旧版明文配置迁移:加载后盘上明文被清除,内存中本会话仍可用
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from common.config import ConfigManager
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def fresh_config(tmp_path, monkeypatch):
|
||||||
|
"""隔离 ConfigManager 单例:AI_CONFIG_PATH 指向临时文件,前后重置类状态"""
|
||||||
|
cfg_file = tmp_path / "config.json"
|
||||||
|
monkeypatch.setenv("AI_CONFIG_PATH", str(cfg_file))
|
||||||
|
|
||||||
|
def _reset():
|
||||||
|
ConfigManager._instance = None
|
||||||
|
ConfigManager._config = None
|
||||||
|
ConfigManager._config_path = None
|
||||||
|
|
||||||
|
_reset()
|
||||||
|
yield ConfigManager
|
||||||
|
_reset()
|
||||||
|
|
||||||
|
|
||||||
|
def _read_cfg_file() -> dict:
|
||||||
|
path = ConfigManager()._config_path
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
||||||
|
|
||||||
|
def test_set_proxy_sanitizes_password_on_save(fresh_config, tmp_path):
|
||||||
|
"""save=True 落盘后,文件中不得出现明文密码(纵深防御脱敏)"""
|
||||||
|
cfg = fresh_config()
|
||||||
|
cfg.set_proxy("127.0.0.1", 1080, username="u1", password="secret123",
|
||||||
|
password_env="MY_SOCKS_PW", save=True)
|
||||||
|
|
||||||
|
on_disk = _read_cfg_file()
|
||||||
|
socks = on_disk["proxy"]["socks5"]
|
||||||
|
assert socks["password"] in (None, ""), "落盘文件不得含明文密码"
|
||||||
|
assert socks["password_env"] == "MY_SOCKS_PW", "password_env 应持久化"
|
||||||
|
|
||||||
|
raw = (tmp_path / "config.json").read_text(encoding="utf-8")
|
||||||
|
assert "secret123" not in raw, "整个配置文件不得出现明文密码字符串"
|
||||||
|
|
||||||
|
# 内存中本会话仍可用
|
||||||
|
proxies = cfg.get_proxy()
|
||||||
|
assert proxies is not None
|
||||||
|
assert "u1:secret123@" in proxies["https"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_password_env_resolution(fresh_config, monkeypatch):
|
||||||
|
"""password 为空时按 password_env 从环境变量解析"""
|
||||||
|
monkeypatch.setenv("TEST_SOCKS_PW", "env-secret-42")
|
||||||
|
cfg = fresh_config()
|
||||||
|
cfg.set_proxy("10.0.0.8", 7890, username="u2", password=None,
|
||||||
|
password_env="TEST_SOCKS_PW", save=True)
|
||||||
|
|
||||||
|
proxies = cfg.get_proxy()
|
||||||
|
assert proxies is not None
|
||||||
|
assert "u2:env-secret-42@10.0.0.8:7890" in proxies["https"]
|
||||||
|
|
||||||
|
# get_proxy_info 不泄漏密码值,只给布尔 + 变量名
|
||||||
|
info = cfg.get_proxy_info()
|
||||||
|
assert info["password"] is True
|
||||||
|
assert info["password_env"] == "TEST_SOCKS_PW"
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_plaintext_migration(fresh_config, tmp_path):
|
||||||
|
"""旧版明文配置:加载后盘上清除,内存保留供本会话使用"""
|
||||||
|
cfg_file = tmp_path / "config.json"
|
||||||
|
legacy = {
|
||||||
|
"proxy": {
|
||||||
|
"socks5": {
|
||||||
|
"enabled": True, "host": "1.2.3.4", "port": 1080,
|
||||||
|
"username": "old_u", "password": "legacy_plain_pw"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cfg_file.write_text(json.dumps(legacy, ensure_ascii=False), encoding="utf-8")
|
||||||
|
|
||||||
|
cfg = fresh_config() # 触发 _load_config → 迁移
|
||||||
|
|
||||||
|
# 盘上已脱敏
|
||||||
|
on_disk = _read_cfg_file()
|
||||||
|
assert on_disk["proxy"]["socks5"]["password"] in (None, "")
|
||||||
|
raw = cfg_file.read_text(encoding="utf-8")
|
||||||
|
assert "legacy_plain_pw" not in raw
|
||||||
|
|
||||||
|
# 内存中本会话仍可解析出旧密码(代理不断)
|
||||||
|
proxies = cfg.get_proxy()
|
||||||
|
assert proxies is not None
|
||||||
|
assert "old_u:legacy_plain_pw@1.2.3.4:1080" in proxies["https"]
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""F2 修复回归测试:N6 registry 超时清理 + N4 reader 线程收尾。
|
||||||
|
|
||||||
|
N6:流式转发超时后必须清理 tool.streaming_output / tool.output_queue,
|
||||||
|
防止僵尸 daemon 线程经实例属性把输出混入后续会话队列。
|
||||||
|
N4:_read_with_limit 超时/宽限 break 后必须 close stdout + join reader,
|
||||||
|
防止 Windows shell=True 孙进程持管道场景下 reader 线程与 fd 泄漏。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import queue
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from subsystems.executors import SubprocessCommandExecutor
|
||||||
|
from tools.base import Tool
|
||||||
|
from tools.registry import ToolRegistry
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- N6:registry 超时路径清理 ----------
|
||||||
|
|
||||||
|
class _StuckStreamTool(Tool):
|
||||||
|
"""execute_streaming 永不结束(模拟卡死的工具,如等待外部资源)"""
|
||||||
|
|
||||||
|
def get_name(self):
|
||||||
|
return "stuck_tool"
|
||||||
|
|
||||||
|
def get_description(self):
|
||||||
|
return "test only"
|
||||||
|
|
||||||
|
def get_parameters(self):
|
||||||
|
return {"type": "object", "properties": {}}
|
||||||
|
|
||||||
|
def execute(self, **kwargs):
|
||||||
|
time.sleep(30)
|
||||||
|
return "should not reach"
|
||||||
|
|
||||||
|
def execute_streaming(self, output_queue: queue.Queue, **kwargs):
|
||||||
|
self.output_queue = output_queue
|
||||||
|
self.streaming_output = True
|
||||||
|
try:
|
||||||
|
time.sleep(30) # 卡死,永不产出 complete
|
||||||
|
finally:
|
||||||
|
self.streaming_output = False
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_timeout_cleans_tool_state():
|
||||||
|
"""转发总超时后:返回超时消息,且工具实例属性被清理"""
|
||||||
|
reg = ToolRegistry()
|
||||||
|
reg.stream_tool_timeout = 1 # 测试调小(默认 3600)
|
||||||
|
tool = _StuckStreamTool()
|
||||||
|
reg.register(tool)
|
||||||
|
reg.set_output_queue(queue.Queue())
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
result = reg.execute_tool("stuck_tool")
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
|
||||||
|
assert "超时" in result, f"应返回超时消息,实际: {result}"
|
||||||
|
assert elapsed < 10, f"应 ~1s 返回,实际 {elapsed:.1f}s"
|
||||||
|
# N6 核心:超时路径不得跳过清理
|
||||||
|
assert tool.output_queue is None, "超时后 tool.output_queue 必须置 None(防僵尸输出串流)"
|
||||||
|
assert tool.streaming_output is False, "超时后 tool.streaming_output 必须复位"
|
||||||
|
|
||||||
|
|
||||||
|
def test_registry_normal_path_cleans_tool_state():
|
||||||
|
"""正常流式完成后:清理同样执行,结果透传"""
|
||||||
|
reg = ToolRegistry()
|
||||||
|
|
||||||
|
class _OkTool(_StuckStreamTool):
|
||||||
|
def get_name(self):
|
||||||
|
return "ok_tool"
|
||||||
|
|
||||||
|
def execute_streaming(self, output_queue: queue.Queue, **kwargs):
|
||||||
|
self.output_queue = output_queue
|
||||||
|
self.streaming_output = True
|
||||||
|
try:
|
||||||
|
output_queue.put(("line", "hello"))
|
||||||
|
output_queue.put(("complete", "done-result"))
|
||||||
|
finally:
|
||||||
|
pass
|
||||||
|
|
||||||
|
tool = _OkTool()
|
||||||
|
reg.register(tool)
|
||||||
|
reg.set_output_queue(queue.Queue())
|
||||||
|
|
||||||
|
result = reg.execute_tool("ok_tool")
|
||||||
|
assert result == "done-result"
|
||||||
|
assert tool.output_queue is None
|
||||||
|
assert tool.streaming_output is False
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- N4:reader 线程收尾 ----------
|
||||||
|
|
||||||
|
class _FakeStdout:
|
||||||
|
"""模拟管道 stdout,可配置阻塞时长"""
|
||||||
|
|
||||||
|
def __init__(self, block_seconds: float = 8):
|
||||||
|
self.closed = False
|
||||||
|
self._block = block_seconds
|
||||||
|
|
||||||
|
def readline(self):
|
||||||
|
time.sleep(self._block) # > join 宽限 2s 即模拟"无 EOF 阻塞"
|
||||||
|
return ""
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
self.closed = True
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeProcess:
|
||||||
|
def __init__(self, block_seconds: float = 8):
|
||||||
|
self.stdout = _FakeStdout(block_seconds)
|
||||||
|
|
||||||
|
def poll(self):
|
||||||
|
return None # 进程"活着"
|
||||||
|
|
||||||
|
def terminate(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def kill(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_with_limit_timeout_returns_promptly():
|
||||||
|
"""超时 break 后 reader 仍阻塞:只做有限 join,不 close、不挂死。
|
||||||
|
|
||||||
|
CPython 流锁决定了此时 close(stdout) 会阻塞到 readline 返回,
|
||||||
|
故收尾策略为放弃等待(daemon 兜底),主流程必须及时返回。
|
||||||
|
"""
|
||||||
|
ex = SubprocessCommandExecutor()
|
||||||
|
proc = _FakeProcess(block_seconds=8)
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
output, timeout_occurred, truncated = ex._read_with_limit(
|
||||||
|
proc, timeout=1, start_time=time.time())
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
|
||||||
|
assert timeout_occurred is True
|
||||||
|
assert truncated is False
|
||||||
|
# 超时 1s + join 宽限 2s,绝不能等到 block 8s
|
||||||
|
assert elapsed < 5, f"应收尾后及时返回,实际 {elapsed:.1f}s"
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_with_limit_grace_path_returns_promptly():
|
||||||
|
"""shell 死亡宽限 break 后 reader 仍阻塞:同样及时返回"""
|
||||||
|
ex = SubprocessCommandExecutor()
|
||||||
|
proc = _FakeProcess(block_seconds=8)
|
||||||
|
proc.poll = lambda: 0 # shell 已死,但 reader 无 EOF(孙进程持管道)
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
output, timeout_occurred, truncated = ex._read_with_limit(
|
||||||
|
proc, timeout=0, start_time=time.time()) # timeout=0 不触发超时
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
|
||||||
|
assert timeout_occurred is False
|
||||||
|
# 宽限 1s + join 2s 上界,不能等到 block 8s
|
||||||
|
assert elapsed < 5, f"宽限+收尾应 ~3s 内返回,实际 {elapsed:.1f}s"
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_with_limit_closes_stdout_after_normal_eof():
|
||||||
|
"""reader 正常收到 EOF 退出后:显式 close stdout 释放管道 fd"""
|
||||||
|
ex = SubprocessCommandExecutor()
|
||||||
|
proc = _FakeProcess(block_seconds=0) # readline 立即返回 ""(EOF)
|
||||||
|
|
||||||
|
output, timeout_occurred, truncated = ex._read_with_limit(
|
||||||
|
proc, timeout=5, start_time=time.time())
|
||||||
|
|
||||||
|
assert timeout_occurred is False
|
||||||
|
assert output == ""
|
||||||
|
assert proc.stdout.closed is True, "reader 正常退出后必须 close stdout 释放 fd"
|
||||||
@@ -0,0 +1,359 @@
|
|||||||
|
"""aibox 第三轮审计 Batch G 修复回归测试(2026-08-18,审计报告 §9.4)
|
||||||
|
|
||||||
|
覆盖:
|
||||||
|
- P1:非瞬时 HTTP 错误响应体截断(密钥/超大错误页不进异常栈)
|
||||||
|
- P2:Retry-After 解析上限 cap 3600s
|
||||||
|
- P3:fallback 档案缺 model → 跳过并告警(与缺 Key 行为对齐)
|
||||||
|
- P5:LLMClient 死状态 _cancelled 已删除
|
||||||
|
- P6:fallback 列表重复项去重
|
||||||
|
- N2:drain 线程批次结束即回收(不再泄漏常驻线程)
|
||||||
|
- N5:子代理共享父代理取消事件
|
||||||
|
- N7:FTS5 关键词内双引号翻倍转义
|
||||||
|
- N8:registry 流式转发受 TOOL_MAX_RESULT_LENGTH 约束
|
||||||
|
- N10:mark_as_triggered 单事务(缺失 id 路径回归)
|
||||||
|
- N12:persona 路径边界校验(拒绝越界路径)
|
||||||
|
- N16:同秒归档不再覆盖(文件名加序号)
|
||||||
|
|
||||||
|
N3(/new 移线程池)/N13(broadcast)/N14(任务登记集合)为 async WebSocket
|
||||||
|
路径,依赖事件循环与真实连接对象,由现有 WebSocket 测试面回归,不在此单测。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import queue
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import types
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# ---- 环境桩:websockets 未安装时保证 import aibox.py 不失败 ----
|
||||||
|
if "websockets" not in sys.modules:
|
||||||
|
sys.modules["websockets"] = types.ModuleType("websockets")
|
||||||
|
|
||||||
|
import aibox # noqa: E402
|
||||||
|
from aibox import DeepSeekChat, run_full_subagent # noqa: E402
|
||||||
|
|
||||||
|
from llm_client import ( # noqa: E402
|
||||||
|
DeepSeekClient,
|
||||||
|
FailoverClient,
|
||||||
|
_retry_after_of,
|
||||||
|
create_llm_client,
|
||||||
|
)
|
||||||
|
from subsystems.db import MemoryDatabase, MemoDatabase # noqa: E402
|
||||||
|
from subsystems.conversation_history import ConversationHistory # noqa: E402
|
||||||
|
from tools.base import Tool # noqa: E402
|
||||||
|
from tools.registry import ToolRegistry # noqa: E402
|
||||||
|
from common import constants # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== P1/P2:错误响应体截断 + Retry-After cap ====================
|
||||||
|
|
||||||
|
class TestP1P2LlmClient:
|
||||||
|
def test_p1_non_transient_error_body_truncated(self):
|
||||||
|
"""400 响应体 1000 字符 → 异常消息仅含前 200 字符"""
|
||||||
|
c = DeepSeekClient(api_key="k", model="m", base_url="https://x")
|
||||||
|
resp = mock.MagicMock()
|
||||||
|
resp.status_code = 400
|
||||||
|
resp.text = "A" * 1000
|
||||||
|
resp.headers = {}
|
||||||
|
with mock.patch("llm_client.requests.post", return_value=resp):
|
||||||
|
with pytest.raises(RuntimeError) as ei:
|
||||||
|
list(c.stream_chat([{"role": "user", "content": "hi"}]))
|
||||||
|
msg = str(ei.value)
|
||||||
|
assert "HTTP 400" in msg
|
||||||
|
assert len(msg) < 300 # 200 字符正文 + 状态行前缀
|
||||||
|
assert msg.count("A") <= 200
|
||||||
|
|
||||||
|
def test_p2_retry_after_capped(self):
|
||||||
|
"""异常/恶意 Retry-After=999999 → cap 到 3600s"""
|
||||||
|
resp = mock.MagicMock()
|
||||||
|
resp.headers = {"Retry-After": "999999"}
|
||||||
|
assert _retry_after_of(resp) == 3600.0
|
||||||
|
|
||||||
|
def test_p2_retry_after_normal_value_kept(self):
|
||||||
|
resp = mock.MagicMock()
|
||||||
|
resp.headers = {"Retry-After": "30"}
|
||||||
|
assert _retry_after_of(resp) == 30.0
|
||||||
|
|
||||||
|
def test_p2_retry_after_invalid_returns_none(self):
|
||||||
|
resp = mock.MagicMock()
|
||||||
|
resp.headers = {"Retry-After": "not-a-number"}
|
||||||
|
assert _retry_after_of(resp) is None
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== P3/P5/P6:fallback 链构造 ====================
|
||||||
|
|
||||||
|
def _g_cfg():
|
||||||
|
"""主档案 main 的 fallback 链:缺 model 的 broken + 重复 dupe + 正常 ok2"""
|
||||||
|
return {
|
||||||
|
"models": {
|
||||||
|
"main": {
|
||||||
|
"api_url": "https://a.example/v1/chat/completions",
|
||||||
|
"model": "m1", "api_key_env": "MAIN_KEY",
|
||||||
|
"fallback": ["nomodel", "dupe", "dupe", "ok2"],
|
||||||
|
},
|
||||||
|
"nomodel": { # P3:缺 model 字段
|
||||||
|
"api_url": "https://b.example/v1/chat/completions",
|
||||||
|
"api_key_env": "B_KEY",
|
||||||
|
},
|
||||||
|
"dupe": {
|
||||||
|
"api_url": "https://c.example/v1/chat/completions",
|
||||||
|
"model": "c1", "api_key_env": "C_KEY",
|
||||||
|
},
|
||||||
|
"ok2": {
|
||||||
|
"api_url": "https://d.example/v1/chat/completions",
|
||||||
|
"model": "d1", "api_key_env": "D_KEY",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class TestP3P5P6FallbackChain:
|
||||||
|
def test_p3_p6_fallback_missing_model_skipped_and_deduped(self):
|
||||||
|
"""缺 model 的 fallback 跳过;重复项去重;链 = [main, dupe, ok2]"""
|
||||||
|
env = {k: f"sk-{k.lower()}" for k in
|
||||||
|
("MAIN_KEY", "B_KEY", "C_KEY", "D_KEY")}
|
||||||
|
with mock.patch.dict(os.environ, env):
|
||||||
|
c = create_llm_client(_g_cfg(), model="main")
|
||||||
|
assert isinstance(c, FailoverClient)
|
||||||
|
names = [cl.name for cl in c._clients]
|
||||||
|
assert names == ["main", "dupe", "ok2"]
|
||||||
|
|
||||||
|
def test_p5_dead_cancel_flag_removed(self):
|
||||||
|
"""LLMClient 不再持有只写不读的 _cancelled 死状态"""
|
||||||
|
c = DeepSeekClient(api_key="k", model="m", base_url="https://x")
|
||||||
|
assert not hasattr(c, "_cancelled")
|
||||||
|
c.cancel() # 无在途响应时为安全 no-op
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N2:drain 线程生命周期 ====================
|
||||||
|
|
||||||
|
class _EchoTool(Tool):
|
||||||
|
def get_name(self):
|
||||||
|
return "g_echo"
|
||||||
|
|
||||||
|
def get_description(self):
|
||||||
|
return "测试用回显工具"
|
||||||
|
|
||||||
|
def get_parameters(self):
|
||||||
|
return {"type": "object", "properties": {}}
|
||||||
|
|
||||||
|
def execute(self, **kwargs):
|
||||||
|
return "ok"
|
||||||
|
|
||||||
|
|
||||||
|
class TestN2DrainThread:
|
||||||
|
def test_drain_thread_reclaimed_after_batch(self):
|
||||||
|
"""_handle_tool_calls 返回后 drain 线程必须已退出(finally join)"""
|
||||||
|
from test_once_mode import _make_ai
|
||||||
|
ai = _make_ai()
|
||||||
|
ai.tool_registry.register(_EchoTool(ai.logger))
|
||||||
|
tool_calls = [{"id": "c1", "function": {"name": "g_echo", "arguments": "{}"}}]
|
||||||
|
ai._handle_tool_calls(tool_calls, output_callback=lambda t, c: None)
|
||||||
|
drains = [t for t in threading.enumerate() if t.name == "tool-output-drain"]
|
||||||
|
assert drains == [] or all(not t.is_alive() for t in drains)
|
||||||
|
|
||||||
|
@pytest.mark.filterwarnings("ignore::pytest.PytestUnhandledThreadExceptionWarning")
|
||||||
|
def test_drain_thread_reclaimed_on_tool_exception(self):
|
||||||
|
"""工具执行抛异常时 drain 线程同样被回收(finally 路径)"""
|
||||||
|
from test_once_mode import _make_ai
|
||||||
|
|
||||||
|
class _BoomTool(_EchoTool):
|
||||||
|
def get_name(self):
|
||||||
|
return "g_boom"
|
||||||
|
|
||||||
|
def execute(self, **kwargs):
|
||||||
|
raise RuntimeError("boom")
|
||||||
|
|
||||||
|
ai = _make_ai()
|
||||||
|
ai.tool_registry.register(_BoomTool(ai.logger))
|
||||||
|
tool_calls = [{"id": "c1", "function": {"name": "g_boom", "arguments": "{}"}}]
|
||||||
|
ai._handle_tool_calls(tool_calls, output_callback=lambda t, c: None)
|
||||||
|
drains = [t for t in threading.enumerate() if t.name == "tool-output-drain"]
|
||||||
|
assert drains == [] or all(not t.is_alive() for t in drains)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N5:子代理取消传播 ====================
|
||||||
|
|
||||||
|
def _fake_sub_chat():
|
||||||
|
fake = mock.MagicMock()
|
||||||
|
fake.think_and_respond = mock.MagicMock(return_value="任务完成报告")
|
||||||
|
fake.stats = {}
|
||||||
|
fake.messages = []
|
||||||
|
fake._cancel_event = threading.Event()
|
||||||
|
return fake
|
||||||
|
|
||||||
|
|
||||||
|
class TestN5SubagentCancel:
|
||||||
|
def test_subagent_shares_parent_cancel_event(self):
|
||||||
|
"""cancel_event 传入 → 子代理取消事件被替换为父事件(检查点联动)"""
|
||||||
|
fake = _fake_sub_chat()
|
||||||
|
parent_event = threading.Event()
|
||||||
|
with mock.patch("aibox.DeepSeekChat", return_value=fake), \
|
||||||
|
mock.patch("aibox.create_llm_client"), \
|
||||||
|
mock.patch("aibox.SubprocessCommandExecutor"):
|
||||||
|
run_full_subagent("r", "t", None, cancel_event=parent_event)
|
||||||
|
assert fake._cancel_event is parent_event
|
||||||
|
|
||||||
|
def test_subagent_keeps_own_event_when_not_provided(self):
|
||||||
|
"""cancel_event=None → 兼容旧行为,子代理保留自身事件"""
|
||||||
|
fake = _fake_sub_chat()
|
||||||
|
own = fake._cancel_event
|
||||||
|
with mock.patch("aibox.DeepSeekChat", return_value=fake), \
|
||||||
|
mock.patch("aibox.create_llm_client"), \
|
||||||
|
mock.patch("aibox.SubprocessCommandExecutor"):
|
||||||
|
run_full_subagent("r", "t", None)
|
||||||
|
assert fake._cancel_event is own
|
||||||
|
|
||||||
|
def test_subagent_tool_registered_with_parent_event(self):
|
||||||
|
"""DeepSeekChat 注册 SubAgentTool 时注入自身 _cancel_event(N5 链路)"""
|
||||||
|
from test_once_mode import _make_ai
|
||||||
|
ai = _make_ai()
|
||||||
|
tool = next((t for t in ai.tool_registry.get_all_tools()
|
||||||
|
if t.get_name() == "create_subagent"), None)
|
||||||
|
assert tool is not None
|
||||||
|
assert tool.cancel_event is ai._cancel_event
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N7:FTS5 引号转义 ====================
|
||||||
|
|
||||||
|
class TestN7FtsQuoteEscape:
|
||||||
|
def test_keyword_with_quote_does_not_break_search(self, tmp_path):
|
||||||
|
"""关键词含双引号 → MATCH 语法合法(翻倍转义),不抛异常"""
|
||||||
|
db = MemoryDatabase(db_path=str(tmp_path / "mem.db"))
|
||||||
|
db.add_memory(topic="引号话题", summary="摘要",
|
||||||
|
content='含"引号"的记忆内容',
|
||||||
|
start_time="2026-08-18 00:00:00",
|
||||||
|
end_time="2026-08-18 00:00:01", importance=2)
|
||||||
|
res = db.search_memories(['带"引号"的关键词'])
|
||||||
|
assert isinstance(res, list) # 不抛 OperationalError 即通过
|
||||||
|
|
||||||
|
def test_normal_keyword_still_matches(self, tmp_path):
|
||||||
|
"""常规关键词检索行为不回归"""
|
||||||
|
db = MemoryDatabase(db_path=str(tmp_path / "mem.db"))
|
||||||
|
db.add_memory(topic="普通话题", summary="天气摘要",
|
||||||
|
content="今天天气晴朗",
|
||||||
|
start_time="2026-08-18 00:00:00",
|
||||||
|
end_time="2026-08-18 00:00:01", importance=2)
|
||||||
|
res = db.search_memories(["天气"])
|
||||||
|
assert len(res) >= 1
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N8:流式转发长度上限 ====================
|
||||||
|
|
||||||
|
class _BigStreamTool(Tool):
|
||||||
|
"""流式工具:发出 3 行 × 40000 字符(总 120000 > 100000 上限)"""
|
||||||
|
|
||||||
|
def get_name(self):
|
||||||
|
return "g_bigstream"
|
||||||
|
|
||||||
|
def get_description(self):
|
||||||
|
return "测试用超长流式工具"
|
||||||
|
|
||||||
|
def get_parameters(self):
|
||||||
|
return {"type": "object", "properties": {}}
|
||||||
|
|
||||||
|
def execute(self, **kwargs):
|
||||||
|
return "done"
|
||||||
|
|
||||||
|
def execute_streaming(self, output_queue, **kwargs):
|
||||||
|
for _ in range(3):
|
||||||
|
output_queue.put(("line", "x" * 40000))
|
||||||
|
output_queue.put(("complete", "done"))
|
||||||
|
|
||||||
|
|
||||||
|
class TestN8StreamingLimit:
|
||||||
|
def test_streaming_forward_truncated_to_budget(self):
|
||||||
|
registry = ToolRegistry()
|
||||||
|
registry.register(_BigStreamTool())
|
||||||
|
captured = queue.Queue()
|
||||||
|
registry.set_output_queue(captured)
|
||||||
|
|
||||||
|
result = registry.execute_tool("g_bigstream")
|
||||||
|
|
||||||
|
lines = []
|
||||||
|
while not captured.empty():
|
||||||
|
msg_type, content = captured.get_nowait()
|
||||||
|
if msg_type == "line":
|
||||||
|
lines.append(content)
|
||||||
|
budget = constants.TOOL_MAX_RESULT_LENGTH
|
||||||
|
total = sum(len(ln) for ln in lines)
|
||||||
|
# 截断标记行本身占少量额外字符(含提示文案)
|
||||||
|
assert total <= budget + 200
|
||||||
|
assert any("已停止转发" in ln or "已截断" in ln for ln in lines)
|
||||||
|
assert result == "done"
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N10:mark_as_triggered 单事务 ====================
|
||||||
|
|
||||||
|
class TestN10MarkTriggered:
|
||||||
|
def test_missing_id_returns_none_via_transaction(self, tmp_path):
|
||||||
|
"""事务内读取不存在行 → rollback + None(不再走独立 get_memo)"""
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "memo.db"))
|
||||||
|
assert db.mark_as_triggered(99999) is None
|
||||||
|
|
||||||
|
def test_single_shot_memo_completes(self, tmp_path):
|
||||||
|
"""一次性任务触发 → completed + 返回 None(无下次提醒)"""
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "memo.db"))
|
||||||
|
mid = db.add_memo(title="一次性", content="c",
|
||||||
|
reminder_time="2026-08-18 00:00:00")
|
||||||
|
assert mid is not None
|
||||||
|
assert db.mark_as_triggered(mid) is None
|
||||||
|
memo = db.get_memo(mid)
|
||||||
|
assert memo["is_completed"] == 1
|
||||||
|
assert memo["trigger_count"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N12:persona 路径边界 ====================
|
||||||
|
|
||||||
|
class TestN12PersonaBoundary:
|
||||||
|
def _tool(self):
|
||||||
|
return aibox.ExpressEmotionTool()
|
||||||
|
|
||||||
|
def test_traversal_persona_rejected(self, tmp_path):
|
||||||
|
"""persona 含 ../ 越界 → 拒绝(返回空串,不读文件)"""
|
||||||
|
tool = self._tool()
|
||||||
|
skills = tmp_path / "skills"
|
||||||
|
skills.mkdir()
|
||||||
|
with mock.patch.object(type(aibox.constants), "SKILLS_DIR",
|
||||||
|
new_callable=mock.PropertyMock(return_value=str(skills))):
|
||||||
|
assert tool._load_persona("../../etc/passwd", None) == ""
|
||||||
|
assert tool._load_persona("../outside", None) == ""
|
||||||
|
|
||||||
|
def test_absolute_persona_rejected(self, tmp_path):
|
||||||
|
"""persona 为绝对路径 → 解析结果不在技能目录内,拒绝"""
|
||||||
|
tool = self._tool()
|
||||||
|
skills = tmp_path / "skills"
|
||||||
|
skills.mkdir()
|
||||||
|
with mock.patch.object(type(aibox.constants), "SKILLS_DIR",
|
||||||
|
new_callable=mock.PropertyMock(return_value=str(skills))):
|
||||||
|
assert tool._load_persona(str(tmp_path / "evil"), None) == ""
|
||||||
|
|
||||||
|
def test_normal_persona_loads(self, tmp_path):
|
||||||
|
"""合法 persona(技能目录内)正常加载"""
|
||||||
|
tool = self._tool()
|
||||||
|
skills = tmp_path / "skills"
|
||||||
|
skills.mkdir()
|
||||||
|
(skills / "ok.md").write_text("人格内容", encoding="utf-8")
|
||||||
|
with mock.patch.object(type(aibox.constants), "SKILLS_DIR",
|
||||||
|
new_callable=mock.PropertyMock(return_value=str(skills))):
|
||||||
|
assert "人格内容" in tool._load_persona("ok", None)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N16:同秒归档防覆盖 ====================
|
||||||
|
|
||||||
|
class TestN16ArchiveCollision:
|
||||||
|
def test_same_second_archive_no_overwrite(self, tmp_path):
|
||||||
|
"""同一秒内两次归档 → 文件名加序号,互不覆盖"""
|
||||||
|
hm = ConversationHistory(save_dir=str(tmp_path))
|
||||||
|
hm.start_new_conversation(topic="t1")
|
||||||
|
hm.add_message("user", "hi", speaker="u")
|
||||||
|
p1 = hm.archive_current_conversation()
|
||||||
|
|
||||||
|
hm.start_new_conversation(topic="t2")
|
||||||
|
hm.add_message("user", "hi2", speaker="u")
|
||||||
|
p2 = hm.archive_current_conversation()
|
||||||
|
|
||||||
|
assert p1 != p2
|
||||||
|
assert os.path.exists(p1) and os.path.exists(p2)
|
||||||
@@ -0,0 +1,408 @@
|
|||||||
|
"""aibox 第四轮审计 Batch H 修复回归测试(2026-08-18,审计报告 §十一)
|
||||||
|
|
||||||
|
覆盖:
|
||||||
|
- H1:子代理共享父取消事件后首轮 think 不再清除父事件(N5 回归)+ 父已取消时跳过启动
|
||||||
|
- M3:调度器任务级超时兜底(AI think 卡死不再停摆调度循环)
|
||||||
|
- M4:get_recent_memories(limit=None) 使用默认上限(不再全表加载)
|
||||||
|
- M5:memo delete 物理删除(此前只标记完成)
|
||||||
|
- M6:PromptManager.reload 原子替换(并发读者不再读到空/半截列表)
|
||||||
|
- L1:read_file 的 sed 参数强转 int(防引号注入)
|
||||||
|
- L2:set_proxy 落盘失败如实报告(不再静默报"已保存")
|
||||||
|
- L3:SSE 非 UTF-8 行不中断流(errors=replace)
|
||||||
|
- L4:persona 路径校验解析符号链接(软链越界拒绝)
|
||||||
|
- L5:1MB 截断不再产生半个字符(�)
|
||||||
|
- L6:compressor 空内容压缩不再除零
|
||||||
|
- L7:InstallSkillTool 指引命令路径引用(防拆词/注入)
|
||||||
|
- L8:ask_user 缺参返回明确错误(不再 TypeError)
|
||||||
|
- L10:web 工具仅允许 http/https(file:// 等拒绝)
|
||||||
|
- L11:registry 日志打码改包含匹配(apiKey/access_token 等变体)
|
||||||
|
- L12:FailoverClient 日志 Bearer 令牌脱敏
|
||||||
|
|
||||||
|
M1(/reload_prompts 移线程池)/M2(技能扫描启动前预热)为 async WebSocket /
|
||||||
|
启动路径,依赖事件循环与真实连接对象,由既有 WebSocket 测试面与启动逻辑回归。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import queue
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import types
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# ---- 环境桩:websockets 未安装时保证 import aibox.py 不失败 ----
|
||||||
|
if "websockets" not in sys.modules:
|
||||||
|
sys.modules["websockets"] = types.ModuleType("websockets")
|
||||||
|
|
||||||
|
import aibox # noqa: E402
|
||||||
|
from aibox import DeepSeekChat, TaskScheduler, run_full_subagent # noqa: E402
|
||||||
|
|
||||||
|
from llm_client import DeepSeekClient, FailoverClient, _sanitize_log_msg # noqa: E402
|
||||||
|
from subsystems.db import MemoryDatabase, MemoDatabase # noqa: E402
|
||||||
|
from subsystems.prompt_manager import PromptManager # noqa: E402
|
||||||
|
from subsystems.conversation_compressor import ConversationCompressor # noqa: E402
|
||||||
|
from common import constants # noqa: E402
|
||||||
|
|
||||||
|
from test_conversation_compressor import FakeLLMClient # noqa: E402
|
||||||
|
from test_once_mode import _make_ai # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_sub_chat():
|
||||||
|
fake = mock.MagicMock()
|
||||||
|
fake.think_and_respond = mock.MagicMock(return_value="任务完成报告")
|
||||||
|
fake.stats = {}
|
||||||
|
fake.messages = []
|
||||||
|
fake._cancel_event = threading.Event()
|
||||||
|
return fake
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== H1:子代理取消事件清除回归 ====================
|
||||||
|
|
||||||
|
class TestH1SubagentCancelClear:
|
||||||
|
def test_shared_event_not_cleared_on_first_think(self):
|
||||||
|
"""共享父事件后首轮 think 不清除事件——父级已置位的取消信号不被吞掉"""
|
||||||
|
ai = _make_ai()
|
||||||
|
parent_event = threading.Event()
|
||||||
|
parent_event.set()
|
||||||
|
ai._cancel_event = parent_event
|
||||||
|
ai._shared_cancel_event = True
|
||||||
|
with pytest.raises(InterruptedError):
|
||||||
|
ai._think_and_respond_impl("hi")
|
||||||
|
assert parent_event.is_set()
|
||||||
|
|
||||||
|
def test_own_event_still_cleared_on_first_think(self):
|
||||||
|
"""未共享(普通会话)时首轮仍清除自身事件——原语义不回归"""
|
||||||
|
import contextlib
|
||||||
|
ai = _make_ai()
|
||||||
|
ai._cancel_event.set()
|
||||||
|
with contextlib.suppress(Exception):
|
||||||
|
ai._think_and_respond_impl("hi")
|
||||||
|
assert not ai._cancel_event.is_set()
|
||||||
|
|
||||||
|
def test_subagent_not_started_when_parent_already_cancelled(self):
|
||||||
|
"""父事件已置位 → 直接返回取消提示,不启动子代理"""
|
||||||
|
fake = _fake_sub_chat()
|
||||||
|
parent_event = threading.Event()
|
||||||
|
parent_event.set()
|
||||||
|
with mock.patch("aibox.DeepSeekChat", return_value=fake), \
|
||||||
|
mock.patch("aibox.create_llm_client"), \
|
||||||
|
mock.patch("aibox.SubprocessCommandExecutor"):
|
||||||
|
result = run_full_subagent("r", "t", None, cancel_event=parent_event)
|
||||||
|
assert "取消" in result
|
||||||
|
fake.think_and_respond.assert_not_called()
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== M3:调度器任务超时 ====================
|
||||||
|
|
||||||
|
class _BlockingAI:
|
||||||
|
"""think 卡死直到被 cancel 的伪 AI(模拟 LLM 挂起)"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.cancelled = threading.Event()
|
||||||
|
self.cancel_called = 0
|
||||||
|
|
||||||
|
def cancel(self):
|
||||||
|
self.cancel_called += 1
|
||||||
|
self.cancelled.set()
|
||||||
|
|
||||||
|
def think_and_respond(self, input_text, output_callback=None,
|
||||||
|
is_continuation=False):
|
||||||
|
self.cancelled.wait(timeout=15)
|
||||||
|
raise InterruptedError("用户中断")
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeMemoDB:
|
||||||
|
def get_memo(self, memo_id):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class TestM3SchedulerTimeout:
|
||||||
|
def test_stuck_task_times_out_and_scheduler_continues(self):
|
||||||
|
from aibox import ConfigManager
|
||||||
|
loop = asyncio.new_event_loop()
|
||||||
|
try:
|
||||||
|
scheduler = TaskScheduler(loop=loop, logger=None)
|
||||||
|
fake_ai = _BlockingAI()
|
||||||
|
memo = {"id": 7, "title": "卡死任务", "content": "内容", "trigger_count": 0}
|
||||||
|
orig_get = ConfigManager.get
|
||||||
|
|
||||||
|
def fake_get(self, key, default=None):
|
||||||
|
# 只把任务超时压到 0.3s,其余配置读取走原实现(防止
|
||||||
|
# DATETIME_FORMAT 等属性读到错误类型)
|
||||||
|
if key == "scheduler.task_timeout":
|
||||||
|
return 0.3
|
||||||
|
return orig_get(self, key, default)
|
||||||
|
|
||||||
|
with mock.patch.object(ConfigManager, "get", fake_get), \
|
||||||
|
mock.patch.object(scheduler, "_create_task_ai", return_value=fake_ai):
|
||||||
|
result = loop.run_until_complete(
|
||||||
|
scheduler.execute_task_as_conversation(memo, _FakeMemoDB()))
|
||||||
|
assert "超时" in result
|
||||||
|
assert fake_ai.cancel_called >= 1
|
||||||
|
finally:
|
||||||
|
loop.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== M4/M5:记忆上限与任务删除 ====================
|
||||||
|
|
||||||
|
class TestM4M5Db:
|
||||||
|
def test_recent_memories_none_capped(self, tmp_path, monkeypatch):
|
||||||
|
"""limit=None 不再全表加载,使用默认上限(对齐 search_memories)"""
|
||||||
|
from test_conversation_compressor import patch_const
|
||||||
|
db = MemoryDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
for i in range(7):
|
||||||
|
db.add_memory(topic=f"t{i}", summary=f"s{i}", content="c",
|
||||||
|
start_time="2026-08-18 00:00:00",
|
||||||
|
end_time="2026-08-18 00:00:01", importance=1)
|
||||||
|
patch_const(monkeypatch, "MEMORY_MAX_SEARCH_RESULTS", 5)
|
||||||
|
res = db.get_recent_memories(limit=None)
|
||||||
|
assert len(res) == 5
|
||||||
|
|
||||||
|
def test_delete_memo_physically_removes(self, tmp_path):
|
||||||
|
"""delete_memo 真实删除行(此前 complete_memo 只标记完成)"""
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
mid = db.add_memo(title="待删任务", content="内容")
|
||||||
|
assert db.get_memo(mid) is not None
|
||||||
|
assert db.delete_memo(mid) is True
|
||||||
|
assert db.get_memo(mid) is None
|
||||||
|
assert db.delete_memo(mid) is False # 已不存在
|
||||||
|
|
||||||
|
def test_memo_tool_delete_uses_physical_delete(self, tmp_path):
|
||||||
|
"""MemoTool delete 走 delete_memo:行消失而非仅标记完成"""
|
||||||
|
from tools import MemoTool
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
mid = db.add_memo(title="重复任务", content="内容", repeat_type="daily")
|
||||||
|
tool = MemoTool(db, None)
|
||||||
|
result = tool.execute(action="delete", memo_id=mid, confirm=True)
|
||||||
|
assert "已删除" in result
|
||||||
|
assert db.get_memo(mid) is None
|
||||||
|
stats = db.get_memo_stats()
|
||||||
|
assert stats["total"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== M6:PromptManager 原子 reload ====================
|
||||||
|
|
||||||
|
class TestM6PromptReloadAtomic:
|
||||||
|
def test_reload_builds_new_list_then_swaps(self):
|
||||||
|
"""reload 在本地新列表构建完成后才原子替换——读者始终看到完整列表"""
|
||||||
|
pm = PromptManager()
|
||||||
|
old_list = pm._prompts
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
def fake_load(target):
|
||||||
|
seen.append(target)
|
||||||
|
target.append({"file": "X_test.md", "content": "y"})
|
||||||
|
|
||||||
|
with mock.patch.object(pm, "_load_prompts", side_effect=fake_load):
|
||||||
|
n = pm.reload()
|
||||||
|
assert len(seen) == 1
|
||||||
|
assert seen[0] is not old_list # 构建用的是新列表
|
||||||
|
assert pm._prompts is seen[0] # 完成后才替换
|
||||||
|
assert n == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L1:read_file 参数强转 ====================
|
||||||
|
|
||||||
|
class TestL1ReadFileCoercion:
|
||||||
|
def test_non_integer_lines_rejected(self, tmp_path):
|
||||||
|
from tools import ReadFileTool
|
||||||
|
f = tmp_path / "f.txt"
|
||||||
|
f.write_text("line1\nline2\n", encoding="utf-8")
|
||||||
|
tool = ReadFileTool()
|
||||||
|
r = tool.execute(filepath=str(f), start_line="1'; rm -rf /; echo '",
|
||||||
|
end_line=5)
|
||||||
|
assert "参数错误" in r
|
||||||
|
|
||||||
|
def test_valid_integers_still_work(self, tmp_path):
|
||||||
|
from tools import ReadFileTool
|
||||||
|
f = tmp_path / "f.txt"
|
||||||
|
f.write_text("line1\nline2\n", encoding="utf-8")
|
||||||
|
tool = ReadFileTool()
|
||||||
|
r = tool.execute(filepath=str(f), start_line=1, end_line=2)
|
||||||
|
assert "sed -n" in r and "参数错误" not in r
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L2:set_proxy 落盘失败如实报告 ====================
|
||||||
|
|
||||||
|
class TestL2ProxySaveFailure:
|
||||||
|
def test_set_proxy_returns_false_on_save_failure(self, tmp_path):
|
||||||
|
from common import ConfigManager
|
||||||
|
cm = ConfigManager()
|
||||||
|
cm._config_path = str(tmp_path / "config.json")
|
||||||
|
with mock.patch("builtins.open", side_effect=PermissionError("disk ro")):
|
||||||
|
ok = cm.set_proxy("127.0.0.1", 1080, save=True)
|
||||||
|
assert ok is False
|
||||||
|
|
||||||
|
def test_proxy_tool_reports_save_failure(self, tmp_path):
|
||||||
|
from tools import SetProxyTool
|
||||||
|
tool = SetProxyTool(None)
|
||||||
|
with mock.patch("builtins.open", side_effect=PermissionError("disk ro")):
|
||||||
|
r = tool.execute(host="127.0.0.1", port=1080, save=True,
|
||||||
|
skip_check=True, enabled=True)
|
||||||
|
assert "保存失败" in r
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L3:SSE 非 UTF-8 行不中断流 ====================
|
||||||
|
|
||||||
|
class TestL3SseDecode:
|
||||||
|
def test_non_utf8_line_does_not_break_stream(self):
|
||||||
|
c = DeepSeekClient(api_key="k", model="m", base_url="https://x")
|
||||||
|
resp = mock.MagicMock()
|
||||||
|
resp.status_code = 200
|
||||||
|
resp.iter_lines.return_value = [
|
||||||
|
b"data: \xff\xfe",
|
||||||
|
b'data: {"choices":[{"delta":{"content":"ok"}}]}',
|
||||||
|
b"data: [DONE]",
|
||||||
|
]
|
||||||
|
with mock.patch("llm_client.requests.post", return_value=resp):
|
||||||
|
events = list(c.stream_chat([{"role": "user", "content": "hi"}]))
|
||||||
|
assert any(e.get("type") == "content" and e.get("text") == "ok"
|
||||||
|
for e in events)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L4:persona 软链越界 ====================
|
||||||
|
|
||||||
|
class TestL4PersonaSymlink:
|
||||||
|
def test_symlink_outside_skills_dir_rejected(self, tmp_path, monkeypatch):
|
||||||
|
from test_conversation_compressor import patch_const
|
||||||
|
from aibox import ExpressEmotionTool
|
||||||
|
skills_dir = tmp_path / "skills"
|
||||||
|
skills_dir.mkdir()
|
||||||
|
outside = tmp_path / "outside.md"
|
||||||
|
outside.write_text("秘密内容", encoding="utf-8")
|
||||||
|
(skills_dir / "evil.md").symlink_to(outside)
|
||||||
|
tool = ExpressEmotionTool()
|
||||||
|
patch_const(monkeypatch, "SKILLS_DIR", str(skills_dir))
|
||||||
|
content = tool._load_persona("evil", "")
|
||||||
|
assert content == ""
|
||||||
|
|
||||||
|
def test_normal_persona_still_loads(self, tmp_path, monkeypatch):
|
||||||
|
from test_conversation_compressor import patch_const
|
||||||
|
from aibox import ExpressEmotionTool
|
||||||
|
skills_dir = tmp_path / "skills"
|
||||||
|
skills_dir.mkdir()
|
||||||
|
(skills_dir / "happy.md").write_text("快乐人格", encoding="utf-8")
|
||||||
|
tool = ExpressEmotionTool()
|
||||||
|
patch_const(monkeypatch, "SKILLS_DIR", str(skills_dir))
|
||||||
|
content = tool._load_persona("happy", "")
|
||||||
|
assert content == "快乐人格"
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L5:截断无半个字符 ====================
|
||||||
|
|
||||||
|
class TestL5NoPartialChar:
|
||||||
|
def test_truncated_output_has_no_replacement_char(self):
|
||||||
|
from subsystems.executors import SubprocessCommandExecutor
|
||||||
|
ex = SubprocessCommandExecutor()
|
||||||
|
r = ex.execute("python3 -c \"print('中' * 600000)\"", timeout=60)
|
||||||
|
assert "截断" in r
|
||||||
|
assert "\ufffd" not in r
|
||||||
|
|
||||||
|
@pytest.mark.skipif(os.name != "posix", reason="Linux 进程组语义")
|
||||||
|
def test_grandchild_holding_pipe_is_reaped(self):
|
||||||
|
"""`(sleep N &) ; echo done` 形态:shell 退出后孙进程持管道写端,
|
||||||
|
宽限窗口到期强杀进程组——不悬挂 reader、孙进程被清理"""
|
||||||
|
import subprocess as sp
|
||||||
|
from subsystems.executors import SubprocessCommandExecutor
|
||||||
|
marker = f"sleep {12340}"
|
||||||
|
ex = SubprocessCommandExecutor()
|
||||||
|
r = ex.execute(f"({marker} &) ; echo done", timeout=8)
|
||||||
|
assert "done" in r
|
||||||
|
leftover = sp.run(["pgrep", "-f", marker],
|
||||||
|
capture_output=True, text=True).stdout.strip()
|
||||||
|
assert leftover == ""
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L6:compressor 除零 ====================
|
||||||
|
|
||||||
|
class TestL6CompressorDivZero:
|
||||||
|
def test_empty_content_compress_no_div0(self, monkeypatch):
|
||||||
|
comp = ConversationCompressor(llm_client=FakeLLMClient(),
|
||||||
|
logger=None)
|
||||||
|
from test_conversation_compressor import patch_const
|
||||||
|
patch_const(monkeypatch, "COMPRESS_MESSAGE_THRESHOLD", 10)
|
||||||
|
messages = [{"role": "system", "content": "SYS"}] + [
|
||||||
|
{"role": "user" if i % 2 == 0 else "assistant", "content": ""}
|
||||||
|
for i in range(15)
|
||||||
|
]
|
||||||
|
result = comp.compress_conversation(messages)
|
||||||
|
assert isinstance(result, list)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L7:InstallSkillTool 路径引用 ====================
|
||||||
|
|
||||||
|
class TestL7SkillPathQuoting:
|
||||||
|
def test_guide_commands_quote_paths(self):
|
||||||
|
from tools import InstallSkillTool
|
||||||
|
tool = InstallSkillTool()
|
||||||
|
r = tool.execute(skill_path="/tmp/a b; rm -rf /")
|
||||||
|
assert "ls -l '/tmp/a b; rm -rf /'" in r
|
||||||
|
assert "cp '/tmp/a b; rm -rf /' " in r
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L8:ask_user 缺参 ====================
|
||||||
|
|
||||||
|
class TestL8AskUser:
|
||||||
|
def test_missing_question_returns_error(self):
|
||||||
|
from tools import AskUserTool
|
||||||
|
tool = AskUserTool()
|
||||||
|
assert "需要提供问题" in tool.execute()
|
||||||
|
|
||||||
|
def test_question_passed_through(self):
|
||||||
|
from tools import AskUserTool
|
||||||
|
tool = AskUserTool()
|
||||||
|
assert "测试问题" in tool.execute(question="测试问题")
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L10:web 工具 scheme 白名单 ====================
|
||||||
|
|
||||||
|
class TestL10WebScheme:
|
||||||
|
def test_file_url_rejected(self):
|
||||||
|
from tools import SimpleWebCrawlerTool
|
||||||
|
tool = SimpleWebCrawlerTool()
|
||||||
|
r = tool.execute(action="get", url="file:///etc/shadow")
|
||||||
|
assert "仅支持 http/https" in r
|
||||||
|
|
||||||
|
def test_http_url_accepted(self):
|
||||||
|
from tools import SimpleWebCrawlerTool
|
||||||
|
tool = SimpleWebCrawlerTool()
|
||||||
|
r = tool.execute(action="get", url="https://example.com")
|
||||||
|
assert "curl" in r and "仅支持" not in r
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L11:registry 打码包含匹配 ====================
|
||||||
|
|
||||||
|
class TestL11SanitizeSubstring:
|
||||||
|
def test_variant_keys_redacted(self):
|
||||||
|
from tools.registry import _sanitize_kwargs
|
||||||
|
out = _sanitize_kwargs({
|
||||||
|
"apiKey": "x", "access_token": "y", "passwd": "z",
|
||||||
|
"url": "https://ok",
|
||||||
|
})
|
||||||
|
assert out == {"apiKey": "***", "access_token": "***",
|
||||||
|
"passwd": "***", "url": "https://ok"}
|
||||||
|
|
||||||
|
def test_normal_keys_untouched(self):
|
||||||
|
from tools.registry import _sanitize_kwargs
|
||||||
|
out = _sanitize_kwargs({"filepath": "/tmp/a", "question": "hi"})
|
||||||
|
assert out == {"filepath": "/tmp/a", "question": "hi"}
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== L12:failover 日志 Bearer 脱敏 ====================
|
||||||
|
|
||||||
|
class TestL12LogSanitize:
|
||||||
|
def test_sanitize_msg_redacts_bearer(self):
|
||||||
|
msg = "瞬时故障: HTTP 401: Bearer sk-abc123xyz"
|
||||||
|
out = _sanitize_log_msg(msg)
|
||||||
|
assert "sk-abc123xyz" not in out
|
||||||
|
assert "Bearer ***" in out
|
||||||
|
|
||||||
|
def test_failover_log_redacts_bearer(self):
|
||||||
|
logger = mock.MagicMock()
|
||||||
|
c = FailoverClient([], logger=logger)
|
||||||
|
c._log("warning", "瞬时故障(Bearer sk-secret456)")
|
||||||
|
logged = logger.warning.call_args[0][0]
|
||||||
|
assert "sk-secret456" not in logged
|
||||||
|
assert "Bearer ***" in logged
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
"""aibox 第四轮审计 Batch I 修复回归测试(2026-08-18,审计报告 §十二)
|
||||||
|
|
||||||
|
覆盖:
|
||||||
|
- N9:prompt 写入内容长度上限(持久化注入面约束)
|
||||||
|
- Q7b:load_conversation 越界路径拒绝(realpath 前缀校验)
|
||||||
|
- F5:run_light_subagent 取消传播(流式可中断 + 父已取消早退)
|
||||||
|
- Q5:提醒时间格式异常的任务按到期返回(不再静默丢弃)
|
||||||
|
- Q8:对话文件原子写(无 .tmp 残留、JSON 完整)
|
||||||
|
- F6:WebSocketHandler 命令专用线程池存在且可释放
|
||||||
|
- limit=0 语义修正(None 才不加 LIMIT)
|
||||||
|
|
||||||
|
Q4(busy_timeout 10s)为时序性行为(默认 5s 等待已能通过短竞争),
|
||||||
|
不做慢速单测;F9(--once 硬超时)为进程级行为,由 CLI 集成面覆盖。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import queue
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import types
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# ---- 环境桩:websockets 未安装时保证 import aibox.py 不失败 ----
|
||||||
|
if "websockets" not in sys.modules:
|
||||||
|
sys.modules["websockets"] = types.ModuleType("websockets")
|
||||||
|
|
||||||
|
import aibox # noqa: E402
|
||||||
|
from aibox import run_light_subagent # noqa: E402
|
||||||
|
|
||||||
|
from subsystems.db import MemoryDatabase, MemoDatabase # noqa: E402
|
||||||
|
from subsystems.prompt_manager import PromptManager # noqa: E402
|
||||||
|
from subsystems.conversation_history import ConversationHistory # noqa: E402
|
||||||
|
from tools.prompt_tools import ( # noqa: E402
|
||||||
|
UpdateIdentityTool,
|
||||||
|
UpdateSoulTool,
|
||||||
|
SaveMemoryAndEndConversationTool,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== N9:prompt 写入长度上限 ====================
|
||||||
|
|
||||||
|
class TestN9PromptContentCap:
|
||||||
|
def test_update_identity_oversized_rejected(self, tmp_path, monkeypatch):
|
||||||
|
"""manager 层兜底:超长身份内容拒绝写入"""
|
||||||
|
pm = PromptManager()
|
||||||
|
with mock.patch.object(pm, "_prompts_dir", str(tmp_path)):
|
||||||
|
assert pm.update_identity("x" * (PromptManager.MAX_CONTENT_LENGTH + 1)) is False
|
||||||
|
assert not (tmp_path / pm.IDENTITY_FILE).exists()
|
||||||
|
|
||||||
|
def test_update_identity_tool_friendly_error(self, tmp_path):
|
||||||
|
"""工具层:超长内容返回明确提示而非写入"""
|
||||||
|
pm = PromptManager()
|
||||||
|
with mock.patch.object(pm, "_prompts_dir", str(tmp_path)):
|
||||||
|
tool = UpdateIdentityTool(pm, None)
|
||||||
|
r = tool.execute(content="x" * (PromptManager.MAX_CONTENT_LENGTH + 1))
|
||||||
|
assert "内容过长" in r
|
||||||
|
|
||||||
|
def test_update_soul_tool_friendly_error(self, tmp_path):
|
||||||
|
pm = PromptManager()
|
||||||
|
with mock.patch.object(pm, "_prompts_dir", str(tmp_path)):
|
||||||
|
tool = UpdateSoulTool(pm, None)
|
||||||
|
r = tool.execute(content="y" * (PromptManager.MAX_CONTENT_LENGTH + 1))
|
||||||
|
assert "内容过长" in r
|
||||||
|
|
||||||
|
def test_save_memory_status_oversized_rejected(self, tmp_path):
|
||||||
|
"""save_memory_and_end_conversation:状态内容超长拒绝"""
|
||||||
|
pm = PromptManager()
|
||||||
|
with mock.patch.object(pm, "_prompts_dir", str(tmp_path)):
|
||||||
|
tool = SaveMemoryAndEndConversationTool(
|
||||||
|
exit_callback=lambda *a: None,
|
||||||
|
memory_db=None, prompt_manager=pm, logger=None)
|
||||||
|
r = tool.execute(summary="这是一段足够长的对话总结" * 20,
|
||||||
|
current_status="x" * PromptManager.MAX_CONTENT_LENGTH)
|
||||||
|
assert "状态内容过长" in r
|
||||||
|
|
||||||
|
def test_normal_content_still_written(self, tmp_path):
|
||||||
|
pm = PromptManager()
|
||||||
|
with mock.patch.object(pm, "_prompts_dir", str(tmp_path)):
|
||||||
|
assert pm.update_identity("正常身份内容") is True
|
||||||
|
assert "正常身份内容" in (tmp_path / pm.IDENTITY_FILE).read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== Q7b:load_conversation 路径校验 ====================
|
||||||
|
|
||||||
|
class TestQ7bLoadConversationBoundary:
|
||||||
|
def test_traversal_rejected(self, tmp_path):
|
||||||
|
h = ConversationHistory(save_dir=str(tmp_path / "save"), load_history=False)
|
||||||
|
secret = tmp_path / "secret.json"
|
||||||
|
secret.write_text('{"leak": true}', encoding="utf-8")
|
||||||
|
assert h.load_conversation("../secret.json") is None
|
||||||
|
|
||||||
|
def test_normal_file_still_loads(self, tmp_path):
|
||||||
|
h = ConversationHistory(save_dir=str(tmp_path / "save"), load_history=False)
|
||||||
|
h.current_conversation = {
|
||||||
|
"id": "1", "start_time": "2026-08-18 00:00:00", "end_time": None,
|
||||||
|
"topic": "t", "messages": [{"role": "user", "content": "hi"}],
|
||||||
|
"tool_calls": [], "stats": {},
|
||||||
|
}
|
||||||
|
h.save_current_conversation()
|
||||||
|
h.archive_current_conversation()
|
||||||
|
archives = list(h.archive_dir.glob("conversation_*.json"))
|
||||||
|
assert archives
|
||||||
|
loaded = h.load_conversation(archives[0].name)
|
||||||
|
assert loaded is not None and loaded["id"] == "1"
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== F5:run_light_subagent 取消传播 ====================
|
||||||
|
|
||||||
|
class _BlockingLightClient:
|
||||||
|
"""流式产出首段后卡住,直到 cancel() 才中断的伪 LLM 客户端"""
|
||||||
|
|
||||||
|
model = "fake-model"
|
||||||
|
default_temperature = None
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.cancelled = threading.Event()
|
||||||
|
|
||||||
|
def cancel(self):
|
||||||
|
self.cancelled.set()
|
||||||
|
|
||||||
|
def stream_chat(self, messages, **kwargs):
|
||||||
|
yield {"type": "content", "text": "开头"}
|
||||||
|
while not self.cancelled.wait(timeout=0.1):
|
||||||
|
pass
|
||||||
|
raise RuntimeError("连接已中断(cancel)")
|
||||||
|
|
||||||
|
|
||||||
|
class TestF5LightSubagentCancel:
|
||||||
|
def test_cancelled_before_start_returns_early(self):
|
||||||
|
"""父事件已置位 → 直接返回取消提示,不创建客户端"""
|
||||||
|
parent = threading.Event()
|
||||||
|
parent.set()
|
||||||
|
with mock.patch("aibox.create_llm_client") as mk:
|
||||||
|
r = run_light_subagent("r", "t", None, cancel_event=parent)
|
||||||
|
assert "取消" in r
|
||||||
|
mk.assert_not_called()
|
||||||
|
|
||||||
|
def test_cancel_during_stream_aborts(self):
|
||||||
|
"""流式中父级取消 → 观察者线程 cancel 客户端 → 立即中断返回"""
|
||||||
|
parent = threading.Event()
|
||||||
|
fake = _BlockingLightClient()
|
||||||
|
with mock.patch("aibox.create_llm_client", return_value=fake):
|
||||||
|
result_holder = {}
|
||||||
|
|
||||||
|
def _run():
|
||||||
|
result_holder["r"] = run_light_subagent(
|
||||||
|
"r", "t", None, cancel_event=parent)
|
||||||
|
|
||||||
|
th = threading.Thread(target=_run, daemon=True)
|
||||||
|
th.start()
|
||||||
|
time.sleep(0.5) # 等待首段产出、观察者就绪
|
||||||
|
parent.set()
|
||||||
|
th.join(timeout=5)
|
||||||
|
assert not th.is_alive() # 未挂死
|
||||||
|
assert "失败" in result_holder.get("r", "")
|
||||||
|
|
||||||
|
def test_express_emotion_registered_with_parent_event(self):
|
||||||
|
"""DeepSeekChat 注册 ExpressEmotionTool 时注入自身取消事件"""
|
||||||
|
from test_once_mode import _make_ai
|
||||||
|
ai = _make_ai()
|
||||||
|
tool = next((t for t in ai.tool_registry.get_all_tools()
|
||||||
|
if t.get_name() == "express_emotion"), None)
|
||||||
|
assert tool is not None
|
||||||
|
assert tool.cancel_event is ai._cancel_event
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== Q5:格式异常任务不丢弃 ====================
|
||||||
|
|
||||||
|
class TestQ5MalformedReminderTime:
|
||||||
|
def test_malformed_time_task_returned_as_due(self, tmp_path):
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
mid = db.add_memo(title="格式异常", content="c", reminder_time="不是时间")
|
||||||
|
pending = db.get_pending_memos(before_time="2026-08-18 00:00:00")
|
||||||
|
assert any(m["id"] == mid for m in pending)
|
||||||
|
|
||||||
|
def test_normal_pending_still_filtered(self, tmp_path):
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
db.add_memo(title="未来任务", content="c", reminder_time="2099-01-01 00:00:00")
|
||||||
|
pending = db.get_pending_memos(before_time="2026-08-18 00:00:00")
|
||||||
|
assert pending == []
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== Q8:对话文件原子写 ====================
|
||||||
|
|
||||||
|
class TestQ8AtomicWrites:
|
||||||
|
def test_no_tmp_leftover_and_valid_json(self, tmp_path):
|
||||||
|
h = ConversationHistory(save_dir=str(tmp_path / "save"), load_history=False)
|
||||||
|
h.current_conversation = {
|
||||||
|
"id": "1", "start_time": "2026-08-18 00:00:00", "end_time": None,
|
||||||
|
"topic": "t", "messages": [], "tool_calls": [], "stats": {},
|
||||||
|
}
|
||||||
|
h.save_current_conversation()
|
||||||
|
assert h.current_file.exists() # 文件名可能被用户配置覆盖,用实例属性
|
||||||
|
import json
|
||||||
|
json.loads(h.current_file.read_text(encoding="utf-8")) # 不抛即合法 JSON
|
||||||
|
h.archive_current_conversation() # 归档后 current 文件按设计移除
|
||||||
|
assert list(Path(tmp_path).rglob("*.tmp")) == []
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== F6:WS 命令专用线程池 ====================
|
||||||
|
|
||||||
|
class TestF6CmdExecutor:
|
||||||
|
def test_handler_has_dedicated_executor(self):
|
||||||
|
from aibox import WebSocketHandler
|
||||||
|
h = WebSocketHandler(ai_manager=None)
|
||||||
|
assert h._cmd_executor is not None
|
||||||
|
h.shutdown()
|
||||||
|
assert h._cmd_executor is None
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== limit=0 语义 ====================
|
||||||
|
|
||||||
|
class TestLimitZeroSemantics:
|
||||||
|
def test_pending_memos_limit_zero_returns_none(self, tmp_path):
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
db.add_memo(title="t", content="c", reminder_time="2026-01-01 00:00:00")
|
||||||
|
assert db.get_pending_memos(limit=0, before_time="2026-08-18 00:00:00") == []
|
||||||
|
|
||||||
|
def test_immediate_memos_limit_zero_returns_none(self, tmp_path):
|
||||||
|
db = MemoDatabase(db_path=str(tmp_path / "m.db"))
|
||||||
|
db.add_memo(title="t", content="c", is_immediate=True)
|
||||||
|
assert db.get_immediate_memos(limit=0) == []
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
"""aibox 模型接入点管理工具(model_config.py)单元测试(2026-08-18)
|
||||||
|
|
||||||
|
覆盖:
|
||||||
|
- 列表归一化(默认标记/免密/失败转移展示字段)
|
||||||
|
- 添加:字段校验(档案名/地址/模型名/重复)、Key 两种方式、温度/超时钳制、
|
||||||
|
失败转移引用校验、设为默认、落盘(含脱敏)
|
||||||
|
- 删除:物理移除、fallback 悬空引用清理、默认档案标记清除
|
||||||
|
- 连通性验证:成功 / 失败 / 档案不存在
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
if "websockets" not in sys.modules:
|
||||||
|
sys.modules["websockets"] = types.ModuleType("websockets")
|
||||||
|
|
||||||
|
import model_config # noqa: E402
|
||||||
|
from common import ConfigManager # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def _cfg(tmp_path, initial: dict = None) -> ConfigManager:
|
||||||
|
"""构造指向临时配置文件的 ConfigManager(避免碰真实 ~/.aibox)"""
|
||||||
|
cm = ConfigManager()
|
||||||
|
cm._config_path = str(tmp_path / "config.json")
|
||||||
|
cm._config = initial if initial is not None else {}
|
||||||
|
return cm
|
||||||
|
|
||||||
|
|
||||||
|
def _base_models():
|
||||||
|
return {
|
||||||
|
"models": {
|
||||||
|
"deepseek": {
|
||||||
|
"api_url": "https://api.deepseek.com/v1/chat/completions",
|
||||||
|
"model": "deepseek-chat",
|
||||||
|
"api_key_env": "DEEPSEEK_API_KEY",
|
||||||
|
},
|
||||||
|
"local": {
|
||||||
|
"api_url": "http://localhost:11434/v1/chat/completions",
|
||||||
|
"model": "qwen3",
|
||||||
|
"api_key_env": "",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"api": {"default_model": "deepseek"},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 列表 ====================
|
||||||
|
|
||||||
|
class TestListProfiles:
|
||||||
|
def test_list_normalized(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
profiles = model_config.list_profiles(cm)
|
||||||
|
assert len(profiles) == 2
|
||||||
|
ds = next(p for p in profiles if p["name"] == "deepseek")
|
||||||
|
assert ds["is_default"] is True
|
||||||
|
assert ds["api_key_env"] == "DEEPSEEK_API_KEY"
|
||||||
|
local = next(p for p in profiles if p["name"] == "local")
|
||||||
|
assert local["api_key_env"] == ""
|
||||||
|
assert local["is_default"] is False
|
||||||
|
|
||||||
|
def test_list_empty_models_section(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {})
|
||||||
|
assert model_config.list_profiles(cm) == []
|
||||||
|
|
||||||
|
def test_list_ignores_malformed_entries(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {"models": {"bad": "not-a-dict"}})
|
||||||
|
assert model_config.list_profiles(cm) == []
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 校验 ====================
|
||||||
|
|
||||||
|
class TestValidateNewProfile:
|
||||||
|
def test_name_rules(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
assert model_config.validate_new_profile(cm, "", "https://x", "m")
|
||||||
|
assert model_config.validate_new_profile(cm, "坏名字!", "https://x", "m")
|
||||||
|
assert model_config.validate_new_profile(cm, "deepseek", "https://x", "m") # 已存在
|
||||||
|
assert model_config.validate_new_profile(cm, "ok_name", "https://x", "m") is None
|
||||||
|
|
||||||
|
def test_url_scheme_and_model_required(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
assert model_config.validate_new_profile(cm, "kimi", "ftp://x", "m")
|
||||||
|
assert model_config.validate_new_profile(cm, "kimi", "https://x", "")
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 添加 ====================
|
||||||
|
|
||||||
|
class TestAddProfile:
|
||||||
|
def test_add_with_env_key_and_default(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
msgs = model_config.add_profile(
|
||||||
|
cm, "kimi", "https://api.moonshot.cn/v1/chat/completions",
|
||||||
|
"moonshot-v1-8k", api_key_env="KIMI_KEY",
|
||||||
|
display_name="Kimi", set_default=True)
|
||||||
|
assert any("已添加" in m for m in msgs)
|
||||||
|
assert any("设为默认" in m for m in msgs)
|
||||||
|
assert cm.get("api.default_model") == "kimi"
|
||||||
|
assert cm.get("models.kimi.api_key_env") == "KIMI_KEY"
|
||||||
|
assert cm.get("models.kimi.api_url").startswith("https://")
|
||||||
|
# 落盘文件存在且可解析
|
||||||
|
import json
|
||||||
|
disk = json.loads((tmp_path / "config.json").read_text(encoding="utf-8"))
|
||||||
|
assert disk["models"]["kimi"]["model"] == "moonshot-v1-8k"
|
||||||
|
|
||||||
|
def test_add_without_key_is_exempt_endpoint(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {})
|
||||||
|
msgs = model_config.add_profile(
|
||||||
|
cm, "local", "http://localhost:11434/v1/chat/completions",
|
||||||
|
"qwen3", api_key_env="")
|
||||||
|
assert any("免密" in m for m in msgs)
|
||||||
|
assert cm.get("models.local.api_key_env") == ""
|
||||||
|
|
||||||
|
def test_add_rejects_invalid_fallback(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
msgs = model_config.add_profile(
|
||||||
|
cm, "kimi", "https://x/v1", "m1",
|
||||||
|
fallback=["ghost", "deepseek"])
|
||||||
|
assert any("失败转移" in m for m in msgs)
|
||||||
|
assert cm.get("models.kimi") is None # 未写入
|
||||||
|
|
||||||
|
def test_add_temperature_timeout_clamped(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {})
|
||||||
|
model_config.add_profile(cm, "a", "https://x/v1", "m1",
|
||||||
|
temperature=9.9, timeout=1)
|
||||||
|
assert cm.get("models.a.temperature") == 2.0
|
||||||
|
assert cm.get("models.a.timeout") == 5
|
||||||
|
|
||||||
|
def test_add_invalid_temperature_rejected(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {})
|
||||||
|
msgs = model_config.add_profile(cm, "a", "https://x/v1", "m1",
|
||||||
|
temperature="abc")
|
||||||
|
assert any("温度" in m for m in msgs)
|
||||||
|
assert cm.get("models.a") is None
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 删除 ====================
|
||||||
|
|
||||||
|
class TestDeleteProfile:
|
||||||
|
def test_delete_cleans_fallback_refs(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {
|
||||||
|
"models": {
|
||||||
|
"main": {"api_url": "https://a", "model": "m1",
|
||||||
|
"fallback": ["backup"]},
|
||||||
|
"backup": {"api_url": "https://b", "model": "m2"},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
msgs = model_config.delete_profile(cm, "backup")
|
||||||
|
assert any("已删除" in m for m in msgs)
|
||||||
|
assert any("失败转移引用" in m for m in msgs)
|
||||||
|
assert cm.get("models.main.fallback") == []
|
||||||
|
|
||||||
|
def test_delete_default_clears_default_flag(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
model_config.delete_profile(cm, "deepseek")
|
||||||
|
assert cm.get("api.default_model") is None
|
||||||
|
|
||||||
|
def test_delete_nonexistent(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
msgs = model_config.delete_profile(cm, "ghost")
|
||||||
|
assert any("不存在" in m for m in msgs)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 连通性验证 ====================
|
||||||
|
|
||||||
|
class TestTestProfile:
|
||||||
|
def test_connectivity_ok(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
fake = mock.MagicMock()
|
||||||
|
fake.chat_completion.return_value = "pong"
|
||||||
|
with mock.patch("llm_client.create_llm_client", return_value=fake):
|
||||||
|
r = model_config.test_profile(cm, "deepseek")
|
||||||
|
assert "连通正常" in r and "pong" in r
|
||||||
|
|
||||||
|
def test_connectivity_failure(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
fake = mock.MagicMock()
|
||||||
|
fake.chat_completion.side_effect = RuntimeError("连接被拒绝")
|
||||||
|
with mock.patch("llm_client.create_llm_client", return_value=fake):
|
||||||
|
r = model_config.test_profile(cm, "deepseek")
|
||||||
|
assert "验证失败" in r and "连接被拒绝" in r
|
||||||
|
|
||||||
|
def test_connectivity_missing_profile(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, _base_models())
|
||||||
|
assert "不存在" in model_config.test_profile(cm, "ghost")
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 落盘失败感知 ====================
|
||||||
|
|
||||||
|
class TestSaveFailure:
|
||||||
|
def test_add_reports_save_failure(self, tmp_path):
|
||||||
|
cm = _cfg(tmp_path, {})
|
||||||
|
with mock.patch("builtins.open", side_effect=PermissionError("disk ro")):
|
||||||
|
msgs = model_config.add_profile(
|
||||||
|
cm, "a", "https://x/v1", "m1", api_key_env="K")
|
||||||
|
assert any("保存失败" in m for m in msgs)
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,269 @@
|
|||||||
|
"""aibox CLI 一次性对话模式(--once)单元测试
|
||||||
|
|
||||||
|
覆盖:
|
||||||
|
- handle_once_mode 参数路由:未传 --once → 返回 False
|
||||||
|
- enable_memory=False → MemoryTool 不注册(干净禁用)
|
||||||
|
- enable_memory=True(默认)→ MemoryTool 正常注册(回归保证)
|
||||||
|
- cli_callback 流式分流:chunk→stdout / line→stderr / reasoning 受 --show-reasoning 控制
|
||||||
|
- 退出码:成功 0 / 工具失败 1 / API 失败 2
|
||||||
|
- stdout 不被压缩提示等 print 污染
|
||||||
|
|
||||||
|
通过 FakeLLMClient 桩掉 stream_chat,不依赖真实 API/网络。
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import types
|
||||||
|
import argparse
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# ---- 环境桩:websockets 未安装时保证 import aibox.py 不失败 ----
|
||||||
|
if "websockets" not in sys.modules:
|
||||||
|
sys.modules["websockets"] = types.ModuleType("websockets")
|
||||||
|
|
||||||
|
import aibox # noqa: E402
|
||||||
|
from aibox import DeepSeekChat, handle_once_mode # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== Fake LLM 客户端 ====================
|
||||||
|
|
||||||
|
class FakeLLMClient:
|
||||||
|
"""桩 LLMClient:按预设脚本产出统一事件流。"""
|
||||||
|
|
||||||
|
def __init__(self, events=None, raise_exc=None):
|
||||||
|
# events: list[dict],如 [{"type":"content","text":"你好"}, {"type":"done"}]
|
||||||
|
self.events = events if events is not None else [
|
||||||
|
{"type": "content", "text": "默认回答"},
|
||||||
|
{"type": "done"},
|
||||||
|
]
|
||||||
|
self.raise_exc = raise_exc
|
||||||
|
self.api_key = "fake-key"
|
||||||
|
self.name = "fake-model"
|
||||||
|
self.model = "fake-model"
|
||||||
|
|
||||||
|
def stream_chat(self, messages, temperature=0.7, max_tokens=8192,
|
||||||
|
tools=None, tool_choice=None, reasoning_enabled=False, **extra):
|
||||||
|
if self.raise_exc:
|
||||||
|
raise self.raise_exc
|
||||||
|
for ev in self.events:
|
||||||
|
yield ev
|
||||||
|
|
||||||
|
def chat_completion(self, messages, temperature=0.3, max_tokens=1024, **kwargs):
|
||||||
|
return "压缩摘要"
|
||||||
|
|
||||||
|
def cancel(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 公共工具 ====================
|
||||||
|
|
||||||
|
def _make_ai(events=None, raise_exc=None, enable_memory=True,
|
||||||
|
tool_profile=None):
|
||||||
|
"""构造一个注入 FakeLLMClient 的 DeepSeekChat,跳过真实网络/文件系统副作用。"""
|
||||||
|
from aibox import TOOL_PROFILE_ORCHESTRATOR
|
||||||
|
with mock.patch("aibox.create_llm_client") as mk:
|
||||||
|
fake = FakeLLMClient(events=events, raise_exc=raise_exc)
|
||||||
|
mk.return_value = fake
|
||||||
|
with mock.patch("aibox.PromptManager") as pm, \
|
||||||
|
mock.patch("aibox.SkillManager"):
|
||||||
|
pm.return_value = mock.MagicMock()
|
||||||
|
pm.return_value.get_all_prompts.return_value = []
|
||||||
|
ai = DeepSeekChat(
|
||||||
|
api_key="fake",
|
||||||
|
history_manager=None,
|
||||||
|
memory_db=mock.MagicMock(),
|
||||||
|
memo_db=mock.MagicMock(),
|
||||||
|
command_executor=mock.MagicMock(),
|
||||||
|
prompt_manager=pm.return_value,
|
||||||
|
enable_memory=enable_memory,
|
||||||
|
tool_profile=tool_profile or TOOL_PROFILE_ORCHESTRATOR,
|
||||||
|
)
|
||||||
|
return ai
|
||||||
|
|
||||||
|
|
||||||
|
def _ns(**over):
|
||||||
|
"""构造 argparse.Namespace,含 once 模式全部字段默认值。"""
|
||||||
|
base = dict(once=None, quiet=False, once_timeout=300,
|
||||||
|
no_save_memory=False, show_reasoning=False,
|
||||||
|
worker_only=False, model=None, temperature=None)
|
||||||
|
base.update(over)
|
||||||
|
return argparse.Namespace(**base)
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== enable_memory 开关 ====================
|
||||||
|
|
||||||
|
class TestEnableMemory:
|
||||||
|
def test_memory_tool_registered_by_default(self):
|
||||||
|
"""默认 enable_memory=True → memory 工具注册"""
|
||||||
|
ai = _make_ai(enable_memory=True)
|
||||||
|
names = [t.get_name() for t in ai.tool_registry.get_all_tools()]
|
||||||
|
assert "memory" in names
|
||||||
|
|
||||||
|
def test_memory_tool_disabled(self):
|
||||||
|
"""enable_memory=False → memory 工具不注册(干净禁用)"""
|
||||||
|
ai = _make_ai(enable_memory=False)
|
||||||
|
names = [t.get_name() for t in ai.tool_registry.get_all_tools()]
|
||||||
|
assert "memory" not in names
|
||||||
|
# 其余 orchestrator 管理面工具不受影响
|
||||||
|
assert "memo" in names
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== handle_once_mode 路由 ====================
|
||||||
|
|
||||||
|
class TestHandleOnceModeRouting:
|
||||||
|
def test_no_once_returns_false(self):
|
||||||
|
"""未传 --once → 返回 False,不进入单次模式"""
|
||||||
|
manager = mock.MagicMock()
|
||||||
|
assert handle_once_mode(manager, _ns(once=None)) is False
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== cli_callback 分流 ====================
|
||||||
|
|
||||||
|
class TestCliCallback分流:
|
||||||
|
"""通过捕获 sys.stdout / sys.stderr 验证分流(handle_once_mode 内部闭包,
|
||||||
|
用 subprocess 太重,这里直接测 DeepSeekChat + 回调契约)。"""
|
||||||
|
|
||||||
|
def _collect(self, ai, query, quiet=False, show_reasoning=False):
|
||||||
|
out, err = [], []
|
||||||
|
|
||||||
|
def cb(t, c):
|
||||||
|
if t == "chunk":
|
||||||
|
out.append(c)
|
||||||
|
elif t == "complete":
|
||||||
|
pass
|
||||||
|
elif t == "error":
|
||||||
|
err.append(c)
|
||||||
|
elif t == "line":
|
||||||
|
if not quiet:
|
||||||
|
err.append(c)
|
||||||
|
elif t in ("reasoning", "reasoning_start", "reasoning_end"):
|
||||||
|
if show_reasoning and not quiet:
|
||||||
|
err.append(c)
|
||||||
|
|
||||||
|
resp = ai.think_and_respond(query, cb)
|
||||||
|
return "".join(out), "".join(err), resp
|
||||||
|
|
||||||
|
def test_chunk_goes_to_stdout(self):
|
||||||
|
ai = _make_ai(events=[{"type": "content", "text": "你好世界"},
|
||||||
|
{"type": "done"}])
|
||||||
|
out, err, resp = self._collect(ai, "hi")
|
||||||
|
assert "你好世界" in out
|
||||||
|
|
||||||
|
def test_reasoning_hidden_by_default(self):
|
||||||
|
ai = _make_ai(events=[{"type": "reasoning", "text": "RSN_CONTENT"},
|
||||||
|
{"type": "content", "text": "答案"},
|
||||||
|
{"type": "done"}])
|
||||||
|
out, err, _ = self._collect(ai, "q", show_reasoning=False)
|
||||||
|
assert "RSN_CONTENT" not in err
|
||||||
|
|
||||||
|
def test_reasoning_shown_with_flag(self):
|
||||||
|
ai = _make_ai(events=[{"type": "reasoning", "text": "RSN_CONTENT"},
|
||||||
|
{"type": "content", "text": "答案"},
|
||||||
|
{"type": "done"}])
|
||||||
|
out, err, _ = self._collect(ai, "q", show_reasoning=True)
|
||||||
|
assert "RSN_CONTENT" in err
|
||||||
|
|
||||||
|
def test_reasoning_hidden_when_quiet(self):
|
||||||
|
ai = _make_ai(events=[{"type": "reasoning", "text": "RSN_CONTENT"},
|
||||||
|
{"type": "content", "text": "答案"},
|
||||||
|
{"type": "done"}])
|
||||||
|
out, err, _ = self._collect(ai, "q", quiet=True, show_reasoning=True)
|
||||||
|
assert "RSN_CONTENT" not in err
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 退出码相关状态 ====================
|
||||||
|
|
||||||
|
class TestExitCodeState:
|
||||||
|
def test_tool_failed_flag_default_false(self):
|
||||||
|
ai = _make_ai()
|
||||||
|
assert ai.stats["tool_failed"] is False
|
||||||
|
|
||||||
|
def test_tool_failed_flag_set_on_exception(self):
|
||||||
|
"""工具执行异常 → stats['tool_failed'] = True"""
|
||||||
|
ai = _make_ai()
|
||||||
|
# 模拟一个会抛异常的工具调用
|
||||||
|
ai.tool_registry.execute_tool = mock.MagicMock(side_effect=RuntimeError("boom"))
|
||||||
|
tool_calls = [{"id": "c1", "function": {"name": "memo", "arguments": "{}"}}]
|
||||||
|
ai._handle_tool_calls(tool_calls, None)
|
||||||
|
assert ai.stats["tool_failed"] is True
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== 压缩提示不污染 stdout ====================
|
||||||
|
|
||||||
|
class TestCompressPromptNotOnStdout:
|
||||||
|
def test_compress_notice_goes_to_callback_line(self, monkeypatch, capsys):
|
||||||
|
"""触发压缩时,提示走 output_callback('line') 而非 print(stdout)。"""
|
||||||
|
ai = _make_ai()
|
||||||
|
# 强制压缩触发
|
||||||
|
monkeypatch.setattr(ai.compressor, "should_compress",
|
||||||
|
lambda msgs: (True, "测试触发"))
|
||||||
|
monkeypatch.setattr(ai.compressor, "compress_conversation",
|
||||||
|
lambda msgs: msgs)
|
||||||
|
monkeypatch.setattr(ai.compressor, "estimate_tokens", lambda msgs: 100)
|
||||||
|
|
||||||
|
lines = []
|
||||||
|
def cb(t, c):
|
||||||
|
if t == "line":
|
||||||
|
lines.append(c)
|
||||||
|
ai.think_and_respond("你好", cb)
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "📦" not in captured.out # stdout 无压缩提示
|
||||||
|
assert any("📦" in l or "压缩" in l for l in lines) # 提示走了 line 回调
|
||||||
|
|
||||||
|
|
||||||
|
# ==================== Batch C:tool_profile / model / temperature / 子代理透传 ====================
|
||||||
|
|
||||||
|
from aibox import TOOL_PROFILE_ORCHESTRATOR, TOOL_PROFILE_WORKER, _SUBAGENT_TOOL_FAILED_MARKER # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
class TestWorkerOnly:
|
||||||
|
def test_worker_profile_registers_execution_tools(self):
|
||||||
|
"""tool_profile=worker → 注册执行面工具(run_command 等),无 memory"""
|
||||||
|
ai = _make_ai(enable_memory=True, tool_profile=TOOL_PROFILE_WORKER)
|
||||||
|
names = [t.get_name() for t in ai.tool_registry.get_all_tools()]
|
||||||
|
assert "run_command" in names
|
||||||
|
assert "read_file" in names
|
||||||
|
assert "memory" not in names # worker 无记忆工具
|
||||||
|
assert "create_subagent" not in names # worker 不能派生
|
||||||
|
|
||||||
|
def test_orchestrator_profile_registers_memory(self):
|
||||||
|
"""tool_profile=orchestrator → 注册管理面工具(memory/create_subagent)"""
|
||||||
|
ai = _make_ai(enable_memory=True)
|
||||||
|
ai.tool_profile = TOOL_PROFILE_ORCHESTRATOR
|
||||||
|
ai._register_tools()
|
||||||
|
names = [t.get_name() for t in ai.tool_registry.get_all_tools()]
|
||||||
|
assert "memory" in names
|
||||||
|
assert "create_subagent" in names
|
||||||
|
|
||||||
|
|
||||||
|
class TestModelTemperatureOverride:
|
||||||
|
def test_model_override(self):
|
||||||
|
"""model 参数透传到 create_llm_client"""
|
||||||
|
ai = _make_ai()
|
||||||
|
assert ai.client.model == "fake-model" # FakeLLMClient.model
|
||||||
|
# 注:真实链路 create_llm_client(model=...) 已在 _make_ai 中 mock 拦截
|
||||||
|
|
||||||
|
def test_temperature_override(self):
|
||||||
|
"""temperature 显式传入覆盖 constants 默认"""
|
||||||
|
ai = _make_ai()
|
||||||
|
# FakeLLMClient 不关心 temperature,这里验证属性被正确设置
|
||||||
|
ai.temperature = 0.1
|
||||||
|
assert ai.temperature == 0.1
|
||||||
|
|
||||||
|
|
||||||
|
class TestSubAgentToolFailurePropagation:
|
||||||
|
def test_subagent_tool_failed_marker_sets_stats(self):
|
||||||
|
"""create_subagent 返回带标记 → orchestrator stats['tool_failed']=True 且标记被剥离"""
|
||||||
|
ai = _make_ai()
|
||||||
|
ai.tool_registry.execute_tool = mock.MagicMock(
|
||||||
|
return_value=_SUBAGENT_TOOL_FAILED_MARKER + "子代理报告正文"
|
||||||
|
)
|
||||||
|
tool_calls = [{"id": "c1", "function": {"name": "create_subagent", "arguments": "{}"}}]
|
||||||
|
tool_msgs = ai._handle_tool_calls(tool_calls, None)
|
||||||
|
assert ai.stats["tool_failed"] is True
|
||||||
|
# 标记已从结果中剥离(LLM 不应看到内部标记)
|
||||||
|
assert tool_msgs and tool_msgs[0]["role"] == "tool"
|
||||||
|
assert "_SUBAGENT_TOOL_FAILED" not in tool_msgs[0]["content"]
|
||||||
|
assert "子代理报告正文" in tool_msgs[0]["content"]
|
||||||
@@ -96,6 +96,16 @@ class ReadFileTool(FileToolBase):
|
|||||||
|
|
||||||
# 限制max_lines上限,防止请求超大范围
|
# 限制max_lines上限,防止请求超大范围
|
||||||
max_lines = min(max_lines, 500)
|
max_lines = min(max_lines, 500)
|
||||||
|
# L1(审计 2026-08-18):start/end_line 为 LLM 可控参数,此前未强转
|
||||||
|
# 直拼 sed 单引号模板——LLM 传含引号的字符串可闭合引号注入命令。
|
||||||
|
# 强转 int(非法输入在此拦截)并夹取合理范围。
|
||||||
|
try:
|
||||||
|
start_line = int(start_line)
|
||||||
|
end_line = int(end_line)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return "❌ 参数错误: start_line/end_line 必须为整数"
|
||||||
|
start_line = max(1, start_line)
|
||||||
|
end_line = max(0, min(end_line, 10_000_000))
|
||||||
quoted = shlex.quote(abs_path)
|
quoted = shlex.quote(abs_path)
|
||||||
|
|
||||||
# 构造等价的命令行读取命令
|
# 构造等价的命令行读取命令
|
||||||
@@ -175,8 +185,15 @@ class WriteFileTool(FileToolBase):
|
|||||||
dir_cmd = f"mkdir -p {shlex.quote(directory)}" if directory else ""
|
dir_cmd = f"mkdir -p {shlex.quote(directory)}" if directory else ""
|
||||||
|
|
||||||
# 内容写入使用 heredoc,避免引号/特殊字符转义问题
|
# 内容写入使用 heredoc,避免引号/特殊字符转义问题
|
||||||
|
# M1 修复(审计 2026-08-17):定界符唯一化——若 content 中含独立行
|
||||||
|
# 与定界符相同(如 "AIBOX_EOF"),heredoc 会提前终止,其后内容被
|
||||||
|
# 当作 shell 命令执行(注入)。冲突时递增后缀直到唯一。
|
||||||
|
delimiter = "AIBOX_EOF"
|
||||||
|
content_lines = {line.strip() for line in content.split("\n")}
|
||||||
|
while delimiter in content_lines:
|
||||||
|
delimiter += "_X"
|
||||||
redirect = ">" if mode == "w" else ">>"
|
redirect = ">" if mode == "w" else ">>"
|
||||||
write_cmd = f"cat {redirect} {quoted} <<'AIBOX_EOF'\n{content}\nAIBOX_EOF"
|
write_cmd = f"cat {redirect} {quoted} <<'{delimiter}'\n{content}\n{delimiter}"
|
||||||
if dir_cmd:
|
if dir_cmd:
|
||||||
write_cmd = f"{dir_cmd} && {write_cmd}"
|
write_cmd = f"{dir_cmd} && {write_cmd}"
|
||||||
|
|
||||||
@@ -313,6 +330,12 @@ class SimpleWebCrawlerTool(Tool):
|
|||||||
if action not in ("get", "text", "title", "links", "search"):
|
if action not in ("get", "text", "title", "links", "search"):
|
||||||
return f"❌ 未知操作: {action}"
|
return f"❌ 未知操作: {action}"
|
||||||
|
|
||||||
|
# L10(审计 2026-08-18):url 为 LLM 可控输入(可被网页内容诱导),
|
||||||
|
# 此前不限 scheme——file:// 可经生成的 curl 指引读取本地文件(SSRF)。
|
||||||
|
# 白名单 http/https 后其余协议直接拒绝。
|
||||||
|
if not url or not str(url).strip().lower().startswith(("http://", "https://")):
|
||||||
|
return "❌ 仅支持 http/https 协议地址(已拦截 file:// 等非网络协议)"
|
||||||
|
|
||||||
quoted_url = shlex.quote(url)
|
quoted_url = shlex.quote(url)
|
||||||
timeout = constants.WEB_TIMEOUT
|
timeout = constants.WEB_TIMEOUT
|
||||||
user_agent = shlex.quote(constants.WEB_USER_AGENT)
|
user_agent = shlex.quote(constants.WEB_USER_AGENT)
|
||||||
|
|||||||
+12
-153
@@ -5,14 +5,15 @@ tools.command_tool - 命令行执行工具
|
|||||||
抽离自 aibox.py,使用抽象的 CommandExecutor 实现命令执行(流式 + 超时控制)。
|
抽离自 aibox.py,使用抽象的 CommandExecutor 实现命令执行(流式 + 超时控制)。
|
||||||
工具本身不持有 subprocess 逻辑,而是委托给 SubprocessCommandExecutor。
|
工具本身不持有 subprocess 逻辑,而是委托给 SubprocessCommandExecutor。
|
||||||
|
|
||||||
|
H1 修复(审计 2026-08-17):此前本类内置一套重复的 Popen 读取循环,
|
||||||
|
完全绕过注入的 executor,导致 1MB 输出上限、cancel() 三层中断、
|
||||||
|
超时机制三项保护同时失效。现改为纯委托,行为与 executor 完全一致。
|
||||||
|
|
||||||
依赖:common (ConfigManager, constants) + tools.base.Tool + subsystems.executors
|
依赖:common (ConfigManager, constants) + tools.base.Tool + subsystems.executors
|
||||||
不反向依赖 aibox 任何符号。
|
不反向依赖 aibox 任何符号。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
|
||||||
import queue
|
import queue
|
||||||
import subprocess
|
|
||||||
import time
|
|
||||||
from typing import Dict
|
from typing import Dict
|
||||||
|
|
||||||
from common import ConfigManager
|
from common import ConfigManager
|
||||||
@@ -60,163 +61,21 @@ class CommandLineTool(Tool):
|
|||||||
}
|
}
|
||||||
|
|
||||||
def execute_streaming(self, output_queue: queue.Queue, **kwargs):
|
def execute_streaming(self, output_queue: queue.Queue, **kwargs):
|
||||||
"""流式执行命令,支持超时控制"""
|
"""流式执行命令:纯委托给 executor(1MB 上限/超时/cancel 全部生效)"""
|
||||||
command = kwargs.get("command")
|
command = kwargs.get("command")
|
||||||
timeout = kwargs.get("timeout", 30)
|
|
||||||
work_dir = kwargs.get("work_dir", ".")
|
|
||||||
extra_env = kwargs.get("env", {})
|
|
||||||
|
|
||||||
if not command:
|
if not command:
|
||||||
output_queue.put(("line", "❌ 错误:缺少命令参数\n"))
|
output_queue.put(("line", "❌ 错误:缺少命令参数\n"))
|
||||||
output_queue.put(("complete", "❌ 错误:缺少命令参数"))
|
output_queue.put(("complete", "❌ 错误:缺少命令参数"))
|
||||||
return
|
return
|
||||||
|
|
||||||
# 发送开始执行的消息
|
self.executor.execute_streaming(
|
||||||
output_queue.put(("line", f"\n🔧 执行命令: {command}\n"))
|
command,
|
||||||
output_queue.put(("line", f"⏱️ 超时设置: {timeout} 秒\n"))
|
output_queue,
|
||||||
if work_dir != ".":
|
timeout=kwargs.get("timeout", 30),
|
||||||
output_queue.put(("line", f"📁 工作目录: {work_dir}\n"))
|
work_dir=kwargs.get("work_dir", "."),
|
||||||
output_queue.put(("line", "-" * 50 + "\n"))
|
env=kwargs.get("env"),
|
||||||
|
)
|
||||||
# 准备环境变量
|
|
||||||
env = os.environ.copy()
|
|
||||||
if extra_env:
|
|
||||||
env.update(extra_env)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# 使用 subprocess.Popen 实现流式输出,支持超时
|
|
||||||
process = subprocess.Popen(
|
|
||||||
command,
|
|
||||||
shell=True,
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.STDOUT,
|
|
||||||
stdin=subprocess.PIPE,
|
|
||||||
text=True,
|
|
||||||
bufsize=1,
|
|
||||||
universal_newlines=True,
|
|
||||||
env=env,
|
|
||||||
cwd=os.path.expanduser(work_dir)
|
|
||||||
)
|
|
||||||
|
|
||||||
start_time = time.time()
|
|
||||||
output_lines = []
|
|
||||||
timeout_occurred = False
|
|
||||||
|
|
||||||
# 实时读取输出
|
|
||||||
while True:
|
|
||||||
# 检查超时
|
|
||||||
elapsed = time.time() - start_time
|
|
||||||
if timeout > 0 and elapsed > timeout:
|
|
||||||
process.terminate()
|
|
||||||
timeout_occurred = True
|
|
||||||
output_queue.put(("line", f"\n⏰ 命令执行超时({timeout}秒),已终止\n"))
|
|
||||||
break
|
|
||||||
|
|
||||||
# 使用非阻塞方式读取一行
|
|
||||||
import select
|
|
||||||
import sys
|
|
||||||
|
|
||||||
# 检查是否有输出可读
|
|
||||||
if sys.platform == 'win32':
|
|
||||||
# Windows 下使用 readline 带超时
|
|
||||||
try:
|
|
||||||
import msvcrt
|
|
||||||
if msvcrt.kbhit():
|
|
||||||
pass
|
|
||||||
line = process.stdout.readline()
|
|
||||||
if not line and process.poll() is not None:
|
|
||||||
break
|
|
||||||
if line:
|
|
||||||
line = line.rstrip('\n')
|
|
||||||
output_queue.put(("line", line + "\n"))
|
|
||||||
output_lines.append(line)
|
|
||||||
else:
|
|
||||||
time.sleep(0.05)
|
|
||||||
except:
|
|
||||||
line = process.stdout.readline()
|
|
||||||
if not line and process.poll() is not None:
|
|
||||||
break
|
|
||||||
if line:
|
|
||||||
line = line.rstrip('\n')
|
|
||||||
output_queue.put(("line", line + "\n"))
|
|
||||||
output_lines.append(line)
|
|
||||||
else:
|
|
||||||
# Unix 系统可以使用 select
|
|
||||||
import select
|
|
||||||
rlist, _, _ = select.select([process.stdout], [], [], 0.5)
|
|
||||||
if rlist:
|
|
||||||
line = process.stdout.readline()
|
|
||||||
if not line and process.poll() is not None:
|
|
||||||
break
|
|
||||||
if line:
|
|
||||||
line = line.rstrip('\n')
|
|
||||||
output_queue.put(("line", line + "\n"))
|
|
||||||
output_lines.append(line)
|
|
||||||
else:
|
|
||||||
# 检查进程是否结束
|
|
||||||
if process.poll() is not None:
|
|
||||||
# 读取剩余输出
|
|
||||||
remaining = process.stdout.read()
|
|
||||||
if remaining:
|
|
||||||
remaining = remaining.rstrip('\n')
|
|
||||||
if remaining:
|
|
||||||
output_queue.put(("line", remaining + "\n"))
|
|
||||||
output_lines.append(remaining)
|
|
||||||
break
|
|
||||||
|
|
||||||
# 如果超时,确保进程被终止
|
|
||||||
if timeout_occurred and process.poll() is None:
|
|
||||||
process.kill()
|
|
||||||
process.wait(timeout=2)
|
|
||||||
|
|
||||||
# 获取返回码
|
|
||||||
return_code = process.returncode if process.returncode is not None else -1
|
|
||||||
|
|
||||||
output_queue.put(("line", "-" * 50 + "\n"))
|
|
||||||
|
|
||||||
if timeout_occurred:
|
|
||||||
output_queue.put(("line", f"⏰ 命令执行超时({timeout}秒),返回码: {return_code}\n"))
|
|
||||||
else:
|
|
||||||
output_queue.put(("line", f"✅ 命令执行完成,返回码: {return_code}\n"))
|
|
||||||
|
|
||||||
# 准备完整结果
|
|
||||||
full_output = "\n".join(output_lines)
|
|
||||||
data = {
|
|
||||||
"输出": full_output if full_output else "(无输出)",
|
|
||||||
"返回码": return_code
|
|
||||||
}
|
|
||||||
|
|
||||||
if timeout_occurred:
|
|
||||||
result = self.format_result(False, f"命令执行超时({timeout}秒)", data)
|
|
||||||
else:
|
|
||||||
result = self.format_result(return_code == 0, "命令执行完成", data)
|
|
||||||
|
|
||||||
output_queue.put(("complete", result))
|
|
||||||
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
error_msg = f"❌ 命令执行超时({timeout}秒)"
|
|
||||||
output_queue.put(("line", error_msg + "\n"))
|
|
||||||
output_queue.put(("complete", error_msg))
|
|
||||||
except FileNotFoundError as e:
|
|
||||||
error_msg = f"❌ 命令执行错误: 命令未找到 - {str(e)}"
|
|
||||||
output_queue.put(("line", error_msg + "\n"))
|
|
||||||
output_queue.put(("complete", error_msg))
|
|
||||||
except PermissionError as e:
|
|
||||||
error_msg = f"❌ 命令执行错误: 权限不足 - {str(e)}"
|
|
||||||
output_queue.put(("line", error_msg + "\n"))
|
|
||||||
output_queue.put(("complete", error_msg))
|
|
||||||
except Exception as e:
|
|
||||||
error_msg = f"❌ 执行错误: {str(e)}"
|
|
||||||
output_queue.put(("line", error_msg + "\n"))
|
|
||||||
output_queue.put(("complete", error_msg))
|
|
||||||
finally:
|
|
||||||
# 确保进程被清理
|
|
||||||
if 'process' in locals() and process and process.poll() is None:
|
|
||||||
try:
|
|
||||||
process.terminate()
|
|
||||||
process.wait(timeout=2)
|
|
||||||
except:
|
|
||||||
process.kill()
|
|
||||||
|
|
||||||
def execute(self, command: str, timeout: int = 30,
|
def execute(self, command: str, timeout: int = 30,
|
||||||
work_dir: str = ".", env: Dict = None, **kwargs) -> str:
|
work_dir: str = ".", env: Dict = None, **kwargs) -> str:
|
||||||
|
|||||||
@@ -752,7 +752,9 @@ class MemoTool(Tool):
|
|||||||
|
|
||||||
memo_id = kwargs.get("memo_id")
|
memo_id = kwargs.get("memo_id")
|
||||||
if memo_id:
|
if memo_id:
|
||||||
if self.memo_db.complete_memo(memo_id, force_complete=True):
|
# M5:真实物理删除(此前调 complete_memo 只标记完成,行不删除,
|
||||||
|
# 重复任务被推进而非移除,stats 计数失真)
|
||||||
|
if self.memo_db.delete_memo(memo_id):
|
||||||
return self.format_result(True, f"任务 #{memo_id} 已删除")
|
return self.format_result(True, f"任务 #{memo_id} 已删除")
|
||||||
else:
|
else:
|
||||||
return f"❌ 无法删除任务 #{memo_id}"
|
return f"❌ 无法删除任务 #{memo_id}"
|
||||||
|
|||||||
@@ -39,10 +39,13 @@ class AskUserTool(Tool):
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"required": []
|
# L8:此前 required 为空,LLM 不传 question 时 str + None 抛 TypeError
|
||||||
|
"required": ["question"]
|
||||||
}
|
}
|
||||||
|
|
||||||
def execute(self, question: str, **kwargs) -> str:
|
def execute(self, question: str = None, **kwargs) -> str:
|
||||||
|
if not question:
|
||||||
|
return "❌ 需要提供问题内容(question 参数)"
|
||||||
return (
|
return (
|
||||||
"你遇到了想搞清楚的事情: " + question + "\n你现在可以询问用户。"
|
"你遇到了想搞清楚的事情: " + question + "\n你现在可以询问用户。"
|
||||||
)
|
)
|
||||||
@@ -56,6 +56,11 @@ class UpdateIdentityTool(Tool):
|
|||||||
if not content:
|
if not content:
|
||||||
return "❌ 需要提供身份信息内容"
|
return "❌ 需要提供身份信息内容"
|
||||||
|
|
||||||
|
# N9:持久化注入面约束——内容超限拒绝写入(管理端另有兜底)
|
||||||
|
if len(content) > PromptManager.MAX_CONTENT_LENGTH:
|
||||||
|
return (f"❌ 内容过长({len(content)} > {PromptManager.MAX_CONTENT_LENGTH} 字符),"
|
||||||
|
f"身份文件作为系统提示词加载,请精简后重试")
|
||||||
|
|
||||||
if not confirm:
|
if not confirm:
|
||||||
return "❌ 身份更新已取消,如需更新请设置 confirm=true"
|
return "❌ 身份更新已取消,如需更新请设置 confirm=true"
|
||||||
|
|
||||||
@@ -119,6 +124,11 @@ class UpdateSoulTool(Tool):
|
|||||||
if not content:
|
if not content:
|
||||||
return "❌ 需要提供智能体自我认识定义内容"
|
return "❌ 需要提供智能体自我认识定义内容"
|
||||||
|
|
||||||
|
# N9:持久化注入面约束——内容超限拒绝写入(管理端另有兜底)
|
||||||
|
if len(content) > PromptManager.MAX_CONTENT_LENGTH:
|
||||||
|
return (f"❌ 内容过长({len(content)} > {PromptManager.MAX_CONTENT_LENGTH} 字符),"
|
||||||
|
f"灵魂文件作为系统提示词加载,请精简后重试")
|
||||||
|
|
||||||
if not confirm:
|
if not confirm:
|
||||||
return "❌ 自我认识更新已取消,如需更新请设置 confirm=true"
|
return "❌ 自我认识更新已取消,如需更新请设置 confirm=true"
|
||||||
|
|
||||||
@@ -223,6 +233,11 @@ class SaveMemoryAndEndConversationTool(Tool):
|
|||||||
status_content += f"【下一步计划】\n{next_goals}\n\n"
|
status_content += f"【下一步计划】\n{next_goals}\n\n"
|
||||||
status_content += f"【对话总结】\n{summary}"
|
status_content += f"【对话总结】\n{summary}"
|
||||||
|
|
||||||
|
# N9:状态文件作为系统提示词加载,超长拒绝写入(约束注入规模)
|
||||||
|
if len(status_content) > PromptManager.MAX_CONTENT_LENGTH:
|
||||||
|
return (f"❌ 状态内容过长({len(status_content)} > "
|
||||||
|
f"{PromptManager.MAX_CONTENT_LENGTH} 字符),请精简 current_status/summary 后重试")
|
||||||
|
|
||||||
# 更新状态文件
|
# 更新状态文件
|
||||||
status_updated = self.prompt_manager.update_status(status_content)
|
status_updated = self.prompt_manager.update_status(status_content)
|
||||||
|
|
||||||
|
|||||||
+121
-7
@@ -3,18 +3,26 @@
|
|||||||
tools.proxy_tools - 代理配置工具
|
tools.proxy_tools - 代理配置工具
|
||||||
|
|
||||||
抽离自 aibox.py,包含 2 个工具:
|
抽离自 aibox.py,包含 2 个工具:
|
||||||
- SetProxyTool : 设置 SOCKS5 代理
|
- SetProxyTool : 设置 SOCKS5 代理(设置前验证连通性)
|
||||||
- GetProxyTool : 查看当前代理配置
|
- GetProxyTool : 查看当前代理配置
|
||||||
|
|
||||||
依赖:common (ConfigManager) + tools.base.Tool
|
依赖:common (ConfigManager) + tools.base.Tool
|
||||||
不反向依赖 aibox 任何符号。
|
不反向依赖 aibox 任何符号。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict
|
import os
|
||||||
|
from typing import Dict, Optional, Tuple
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
from common import ConfigManager
|
from common import ConfigManager
|
||||||
from .base import Tool
|
from .base import Tool
|
||||||
|
|
||||||
|
# 探测端点:无内容的轻量响应,验证完整 SOCKS5 握手(含认证)与出网连通性
|
||||||
|
_PROBE_URL = "http://www.gstatic.com/generate_204"
|
||||||
|
# (连接超时, 读超时):验证总耗时上限约 10s,避免阻塞工具调用过久
|
||||||
|
_PROBE_TIMEOUT = (4, 6)
|
||||||
|
|
||||||
|
|
||||||
class SetProxyTool(Tool):
|
class SetProxyTool(Tool):
|
||||||
"""设置 SOCKS5 代理的工具"""
|
"""设置 SOCKS5 代理的工具"""
|
||||||
@@ -25,6 +33,8 @@ class SetProxyTool(Tool):
|
|||||||
def get_description(self) -> str:
|
def get_description(self) -> str:
|
||||||
return """设置智能体的 SOCKS5 代理配置。
|
return """设置智能体的 SOCKS5 代理配置。
|
||||||
调用后所有 HTTP/HTTPS 请求(包括网页访问、API 调用)都会通过该代理。
|
调用后所有 HTTP/HTTPS 请求(包括网页访问、API 调用)都会通过该代理。
|
||||||
|
启用代理时会先验证连通性(完整 SOCKS5 握手 + 探测请求),验证失败不保存配置;
|
||||||
|
代理暂未启动等场景可用 skip_check=true 跳过验证强制保存。
|
||||||
可随时修改,即时生效。"""
|
可随时修改,即时生效。"""
|
||||||
|
|
||||||
def get_parameters(self) -> Dict:
|
def get_parameters(self) -> Dict:
|
||||||
@@ -34,24 +44,128 @@ class SetProxyTool(Tool):
|
|||||||
"host": {"type": "string", "description": "代理服务器地址,如 127.0.0.1"},
|
"host": {"type": "string", "description": "代理服务器地址,如 127.0.0.1"},
|
||||||
"port": {"type": "integer", "description": "代理端口,如 1080"},
|
"port": {"type": "integer", "description": "代理端口,如 1080"},
|
||||||
"username": {"type": "string", "description": "用户名(可选)"},
|
"username": {"type": "string", "description": "用户名(可选)"},
|
||||||
"password": {"type": "string", "description": "密码(可选)"},
|
"password": {"type": "string", "description": "密码(可选,明文仅本次会话有效,不落盘)"},
|
||||||
|
"password_env": {"type": "string", "description": "存有密码的环境变量名(可选,跨重启持久的推荐方式)"},
|
||||||
"enabled": {"type": "boolean", "description": "是否启用代理,默认 true"},
|
"enabled": {"type": "boolean", "description": "是否启用代理,默认 true"},
|
||||||
"save": {"type": "boolean", "description": "是否保存到配置文件(持久化),默认 true"}
|
"save": {"type": "boolean", "description": "是否保存到配置文件(持久化),默认 true"},
|
||||||
|
"skip_check": {"type": "boolean", "description": "跳过连通性验证强制保存(默认 false)", "default": False}
|
||||||
},
|
},
|
||||||
"required": ["host", "port"]
|
"required": ["host", "port"]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _resolve_password(password: Optional[str], password_env: Optional[str]) -> Optional[str]:
|
||||||
|
"""对齐 ConfigManager._resolve_proxy_password:明文优先,否则查环境变量"""
|
||||||
|
if password:
|
||||||
|
return password
|
||||||
|
if password_env:
|
||||||
|
return os.environ.get(password_env) or None
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _check_connectivity(self, host: str, port: int,
|
||||||
|
username: Optional[str], password: Optional[str]) -> Tuple[bool, str, str]:
|
||||||
|
"""验证代理连通性:走一次完整 SOCKS5 握手 + 探测请求。
|
||||||
|
|
||||||
|
只做 TCP connect 验不了用户名/密码,必须发真实请求。
|
||||||
|
返回 (ok, error_kind, detail);error_kind 取值:
|
||||||
|
auth_failed / timeout / connect_failed / probe_failed / error
|
||||||
|
"""
|
||||||
|
if username and password:
|
||||||
|
proxy_url = f"socks5://{username}:{password}@{host}:{port}"
|
||||||
|
else:
|
||||||
|
proxy_url = f"socks5://{host}:{port}"
|
||||||
|
try:
|
||||||
|
# trust_env=False:排除系统环境变量代理(http_proxy 等)干扰,只验证目标代理
|
||||||
|
session = requests.Session()
|
||||||
|
session.trust_env = False
|
||||||
|
resp = session.get(
|
||||||
|
_PROBE_URL,
|
||||||
|
proxies={"http": proxy_url, "https": proxy_url},
|
||||||
|
timeout=_PROBE_TIMEOUT,
|
||||||
|
)
|
||||||
|
# generate_204 正常返回 204;部分代理/目标返回 200 也视为连通
|
||||||
|
if resp.status_code in (200, 204):
|
||||||
|
return True, "", f"探测成功 HTTP {resp.status_code}"
|
||||||
|
return False, "probe_failed", f"探测端点返回 HTTP {resp.status_code}"
|
||||||
|
except requests.exceptions.Timeout as e:
|
||||||
|
# 含 ConnectTimeout(连接阶段超时):异常类型与系统语言无关,优先于消息匹配
|
||||||
|
return False, "timeout", str(e)
|
||||||
|
except requests.exceptions.ConnectionError as e:
|
||||||
|
# 连接层失败(拒绝/不可达/中断)。PySocks 依赖缺失与认证失败
|
||||||
|
# 也会以 ConnectionError 形态冒出,按消息细分。
|
||||||
|
msg = str(e)
|
||||||
|
low = msg.lower()
|
||||||
|
if "missing dependencies for socks" in low:
|
||||||
|
return False, "missing_dependency", msg
|
||||||
|
if "authentication" in low or "auth" in low:
|
||||||
|
return False, "auth_failed", msg
|
||||||
|
return False, "connect_failed", msg
|
||||||
|
except Exception as e:
|
||||||
|
# 兜底:仅按库自身(非本地化)消息关键词分类
|
||||||
|
msg = str(e)
|
||||||
|
low = msg.lower()
|
||||||
|
if "missing dependencies for socks" in low:
|
||||||
|
return False, "missing_dependency", msg
|
||||||
|
if "authentication" in low or "auth" in low:
|
||||||
|
return False, "auth_failed", msg
|
||||||
|
if "timed out" in low or "timeout" in low:
|
||||||
|
return False, "timeout", msg
|
||||||
|
if "refused" in low or "unreachable" in low or "closed" in low:
|
||||||
|
return False, "connect_failed", msg
|
||||||
|
return False, "error", msg
|
||||||
|
|
||||||
def execute(self, host: str, port: int, username: str = None, password: str = None,
|
def execute(self, host: str, port: int, username: str = None, password: str = None,
|
||||||
enabled: bool = True, save: bool = True, **kwargs) -> str:
|
password_env: str = None, enabled: bool = True, save: bool = True,
|
||||||
|
skip_check: bool = False, **kwargs) -> str:
|
||||||
|
# 设置前连通性验证:坏地址/凭据在入口一次性暴露,避免静默落盘后在
|
||||||
|
# 后续 web/API 调用中离场景报错、LLM 难以归因。skip_check 留给
|
||||||
|
# "代理暂未启动但先写配置"的场景;禁用代理无需验证。
|
||||||
|
check_detail = ""
|
||||||
|
if enabled and not skip_check:
|
||||||
|
pw = self._resolve_password(password, password_env)
|
||||||
|
# 指定了 password_env 但环境变量为空:认证凭据缺失,提前给出精准提示
|
||||||
|
if username and not password and password_env and pw is None:
|
||||||
|
return self.format_result(False, "代理验证失败,未保存配置", {
|
||||||
|
"错误类型": "环境变量为空",
|
||||||
|
"详情": f"环境变量 {password_env} 未设置或为空",
|
||||||
|
"提示": "请先设置该环境变量,或改用 password 参数(仅本次会话有效)",
|
||||||
|
})
|
||||||
|
ok, kind, detail = self._check_connectivity(host, port, username, pw)
|
||||||
|
if not ok:
|
||||||
|
hints = {
|
||||||
|
"auth_failed": "用户名/密码错误,请核对凭据",
|
||||||
|
"timeout": "代理不可达(超时),请检查地址与网络",
|
||||||
|
"connect_failed": "连接被拒绝或不可达,请检查 host/port",
|
||||||
|
"missing_dependency": "缺少 SOCKS 支持库,请安装:pip install pysocks",
|
||||||
|
}
|
||||||
|
return self.format_result(False, "代理不可达,未保存配置", {
|
||||||
|
"错误类型": kind,
|
||||||
|
"详情": detail[:300],
|
||||||
|
"提示": hints.get(kind, "如确认配置无误可设置 skip_check=true 跳过验证"),
|
||||||
|
})
|
||||||
|
check_detail = detail
|
||||||
|
|
||||||
config = ConfigManager()
|
config = ConfigManager()
|
||||||
config.set_proxy(host, port, username, password, enabled, save)
|
saved = config.set_proxy(host, port, username, password, password_env, enabled, save)
|
||||||
proxy_info = config.get_proxy_info()
|
proxy_info = config.get_proxy_info()
|
||||||
|
if proxy_info.get("password_env"):
|
||||||
|
pw_source = "环境变量 " + proxy_info["password_env"]
|
||||||
|
elif proxy_info.get("password"):
|
||||||
|
pw_source = "仅本次会话(不落盘)"
|
||||||
|
else:
|
||||||
|
pw_source = "无"
|
||||||
data = {
|
data = {
|
||||||
"代理状态": "启用" if proxy_info["enabled"] else "禁用",
|
"代理状态": "启用" if proxy_info["enabled"] else "禁用",
|
||||||
"服务器": f"{proxy_info['host']}:{proxy_info['port']}",
|
"服务器": f"{proxy_info['host']}:{proxy_info['port']}",
|
||||||
"认证": "有" if proxy_info.get("username") else "无",
|
"认证": "有" if proxy_info.get("username") else "无",
|
||||||
"持久化": "已保存" if save else "仅内存"
|
"密码来源": pw_source,
|
||||||
|
"持久化": "已保存" if saved else "保存失败(仅本次会话有效,重启后丢失)"
|
||||||
}
|
}
|
||||||
|
if check_detail:
|
||||||
|
data["连通性"] = check_detail
|
||||||
|
# L2:落盘失败(磁盘只读等)如实报告,不再静默报"已保存"
|
||||||
|
if save and not saved:
|
||||||
|
return self.format_result(False, "代理已配置但保存失败(仅本次会话有效,重启后丢失)", data)
|
||||||
return self.format_result(True, "代理设置成功", data)
|
return self.format_result(True, "代理设置成功", data)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+72
-11
@@ -11,10 +11,26 @@ import threading
|
|||||||
import time
|
import time
|
||||||
from typing import Dict, List, Optional, Tuple
|
from typing import Dict, List, Optional, Tuple
|
||||||
|
|
||||||
from common import Logger, i18n
|
from common import Logger, constants, i18n
|
||||||
|
|
||||||
from .base import Tool
|
from .base import Tool
|
||||||
|
|
||||||
|
# 工具参数名命中这些关键词时,debug 日志里打码(防密码/密钥进日志文件)
|
||||||
|
# L11(审计 2026-08-18):精确匹配漏掉 apiKey/access_token/passwd 等变体,
|
||||||
|
# 改包含匹配——日志场景下过度打码可接受,漏打码才是问题。
|
||||||
|
_SENSITIVE_PARAM_KEYS = (
|
||||||
|
"password", "passwd", "api_key", "apikey", "token", "secret",
|
||||||
|
"authorization", "credential",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _sanitize_kwargs(kwargs: Dict) -> Dict:
|
||||||
|
"""复制一份参数并把敏感键的值替换为 ***(仅用于日志输出)"""
|
||||||
|
return {
|
||||||
|
k: ("***" if any(s in str(k).lower() for s in _SENSITIVE_PARAM_KEYS) else v)
|
||||||
|
for k, v in kwargs.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
class ToolRegistry:
|
class ToolRegistry:
|
||||||
"""工具注册器 - 支持技能热重载"""
|
"""工具注册器 - 支持技能热重载"""
|
||||||
@@ -25,6 +41,8 @@ class ToolRegistry:
|
|||||||
self.logger = logger or Logger("ToolRegistry")
|
self.logger = logger or Logger("ToolRegistry")
|
||||||
self.current_output_queue = None # 用于流式输出的队列
|
self.current_output_queue = None # 用于流式输出的队列
|
||||||
self.skill_manager: Optional['SkillManager'] = None # 技能管理器引用
|
self.skill_manager: Optional['SkillManager'] = None # 技能管理器引用
|
||||||
|
# 单工具流式执行总上限(工具自身秒级超时之外的兜底;测试可调小)
|
||||||
|
self.stream_tool_timeout = 3600
|
||||||
|
|
||||||
def register(self, tool: Tool, is_builtin: bool = False):
|
def register(self, tool: Tool, is_builtin: bool = False):
|
||||||
"""注册工具,可以标记是否为内置工具"""
|
"""注册工具,可以标记是否为内置工具"""
|
||||||
@@ -63,7 +81,7 @@ class ToolRegistry:
|
|||||||
if not tool:
|
if not tool:
|
||||||
return f"❌ 错误:找不到工具 '{name}'"
|
return f"❌ 错误:找不到工具 '{name}'"
|
||||||
|
|
||||||
self.logger.debug(f"工具调用: {name}({kwargs})")
|
self.logger.debug(f"工具调用: {name}({_sanitize_kwargs(kwargs)})")
|
||||||
start_time = time.time()
|
start_time = time.time()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -84,16 +102,52 @@ class ToolRegistry:
|
|||||||
)
|
)
|
||||||
thread.daemon = True
|
thread.daemon = True
|
||||||
thread.start()
|
thread.start()
|
||||||
|
|
||||||
# 从队列收集输出并发送到当前队列
|
# M2 修复(审计 2026-08-17):转发循环加总超时。
|
||||||
|
# 此前 while thread.is_alive() 在工具内部卡死(如子进程
|
||||||
|
# 等待 stdin)时永不退出;超时后放弃等待,避免本调用
|
||||||
|
# 线程被连带挂起。
|
||||||
|
# N6 修复(审计 2026-08-18):超时路径此前直接 return,
|
||||||
|
# 跳过下方清理——daemon 线程继续持有 tool.output_queue
|
||||||
|
# 实例属性,属性被后续调用覆盖到新会话队列后,僵尸输出
|
||||||
|
# 会混入后续对话流。现在统一在出口清理。
|
||||||
|
tool_timeout = self.stream_tool_timeout
|
||||||
|
deadline = time.time() + tool_timeout
|
||||||
final_result = None
|
final_result = None
|
||||||
|
timed_out = False
|
||||||
|
# N8:流式转发同样受 TOOL_MAX_RESULT_LENGTH 约束——此前仅非流式
|
||||||
|
# 返回值受限,超长内容(如记忆全文)经流式通道可完整推到前端
|
||||||
|
line_budget = constants.TOOL_MAX_RESULT_LENGTH
|
||||||
|
forwarded = 0
|
||||||
|
truncation_noted = False
|
||||||
while thread.is_alive() or not output_queue.empty():
|
while thread.is_alive() or not output_queue.empty():
|
||||||
|
if time.time() > deadline:
|
||||||
|
self.logger.warning(
|
||||||
|
f"工具 {name} 流式执行超过 {tool_timeout}s 总上限,放弃等待")
|
||||||
|
timed_out = True
|
||||||
|
break
|
||||||
try:
|
try:
|
||||||
msg_type, content = output_queue.get(timeout=0.1)
|
msg_type, content = output_queue.get(timeout=0.1)
|
||||||
if msg_type == "line":
|
if msg_type == "line":
|
||||||
|
piece = str(content) if content is not None else ""
|
||||||
|
if forwarded >= line_budget:
|
||||||
|
# 预算耗尽:停止转发,仅提示一次
|
||||||
|
if not truncation_noted:
|
||||||
|
truncation_noted = True
|
||||||
|
if self.current_output_queue:
|
||||||
|
self.current_output_queue.put(
|
||||||
|
("line", f"\n⚠️ 流式输出超过 {line_budget} 字符上限,已停止转发剩余内容\n"))
|
||||||
|
continue
|
||||||
|
remaining = line_budget - forwarded
|
||||||
|
if len(piece) > remaining:
|
||||||
|
piece = piece[:remaining] + \
|
||||||
|
f"\n⚠️ 流式输出超过 {line_budget} 字符上限,已截断"
|
||||||
|
forwarded = line_budget
|
||||||
|
else:
|
||||||
|
forwarded += len(piece)
|
||||||
# 发送到当前队列(会通过WebSocket发送)
|
# 发送到当前队列(会通过WebSocket发送)
|
||||||
if self.current_output_queue:
|
if self.current_output_queue:
|
||||||
self.current_output_queue.put(("line", content))
|
self.current_output_queue.put(("line", piece))
|
||||||
elif msg_type == "complete":
|
elif msg_type == "complete":
|
||||||
final_result = content
|
final_result = content
|
||||||
# 也发送完整结果
|
# 也发送完整结果
|
||||||
@@ -101,14 +155,21 @@ class ToolRegistry:
|
|||||||
self.current_output_queue.put(("complete", content))
|
self.current_output_queue.put(("complete", content))
|
||||||
except queue.Empty:
|
except queue.Empty:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
thread.join(timeout=5)
|
if not timed_out:
|
||||||
if thread.is_alive():
|
thread.join(timeout=5)
|
||||||
return "❌ 工具执行超时"
|
if thread.is_alive():
|
||||||
|
timed_out = True
|
||||||
result = final_result or ""
|
|
||||||
|
# 统一清理(正常与超时路径都必须执行):断开实例属性,
|
||||||
|
# 僵尸 daemon 线程的 send_line 落到 None 上不再外写
|
||||||
tool.streaming_output = False
|
tool.streaming_output = False
|
||||||
tool.output_queue = None
|
tool.output_queue = None
|
||||||
|
|
||||||
|
if timed_out:
|
||||||
|
result = f"❌ 工具执行超时(>{tool_timeout}s,已放弃等待)"
|
||||||
|
else:
|
||||||
|
result = final_result or ""
|
||||||
else:
|
else:
|
||||||
# 普通执行
|
# 普通执行
|
||||||
result = tool.execute(**kwargs)
|
result = tool.execute(**kwargs)
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ tools.skill_tools - 技能相关工具
|
|||||||
import os
|
import os
|
||||||
import queue
|
import queue
|
||||||
import re
|
import re
|
||||||
|
import shlex
|
||||||
from typing import Dict, Optional
|
from typing import Dict, Optional
|
||||||
|
|
||||||
from common import constants, i18n, Logger
|
from common import constants, i18n, Logger
|
||||||
@@ -253,10 +254,14 @@ class InstallSkillTool(Tool):
|
|||||||
skills_dir = constants.SKILLS_DIR
|
skills_dir = constants.SKILLS_DIR
|
||||||
|
|
||||||
# 如果给了具体路径,在说明里带入,否则用占位符
|
# 如果给了具体路径,在说明里带入,否则用占位符
|
||||||
src = skill_path.strip() if skill_path else "/path/to/your_skill.md"
|
raw_src = skill_path.strip() if skill_path else ""
|
||||||
# 去掉扩展名作为技能名称,再统一补 .md 作为目标文件名
|
# 去掉扩展名作为技能名称,再统一补 .md 作为目标文件名
|
||||||
base = os.path.splitext(os.path.basename(src))[0] if src else "your_skill"
|
base = os.path.splitext(os.path.basename(raw_src))[0] if raw_src else "your_skill"
|
||||||
dst = os.path.join(skills_dir, base + ".md")
|
dst = os.path.join(skills_dir, base + ".md")
|
||||||
|
# L7(审计 2026-08-18):src 为 LLM 可控输入,含空格/元字符时
|
||||||
|
# 未经引用的命令会被 shell 拆词/注入(指引由子代理经 run_command 执行)
|
||||||
|
src = shlex.quote(raw_src) if raw_src else "/path/to/your_skill.md"
|
||||||
|
dst_q = shlex.quote(dst)
|
||||||
|
|
||||||
guide = f"""📦 扩展技能安装指引
|
guide = f"""📦 扩展技能安装指引
|
||||||
{"=" * 55}
|
{"=" * 55}
|
||||||
@@ -282,7 +287,7 @@ class InstallSkillTool(Tool):
|
|||||||
|
|
||||||
调用 create_subagent 派生子代理执行复制命令(目标文件名统一补 .md 扩展名):
|
调用 create_subagent 派生子代理执行复制命令(目标文件名统一补 .md 扩展名):
|
||||||
|
|
||||||
cp {src} {dst}
|
cp {src} {dst_q}
|
||||||
|
|
||||||
⚠️ 注意:目标文件名必须带 .md 扩展名,
|
⚠️ 注意:目标文件名必须带 .md 扩展名,
|
||||||
系统以 .md 文件作为技能加载,且技能名称取文件名(不含 .md)。
|
系统以 .md 文件作为技能加载,且技能名称取文件名(不含 .md)。
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
> 权威架构参考。本文以 `core/services.yaml`(实际运行清单)与 `git HEAD` 为事实来源。
|
> 权威架构参考。本文以 `core/services.yaml`(实际运行清单)与 `git HEAD` 为事实来源。
|
||||||
> 与根目录 `README.md` 的偏差见文末「§10 与 README 的偏差」。
|
> 与根目录 `README.md` 的偏差见文末「§10 与 README 的偏差」。
|
||||||
> 最后核对于:2026-08-15。
|
> 最后核对于:2026-08-17。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -85,7 +85,7 @@ Blackbird 是一套 **进程托管式轻量微服务** 系统(自研 UniServe
|
|||||||
└─────────────────────────────────────┘
|
└─────────────────────────────────────┘
|
||||||
|
|
||||||
独立 / 外设(不在主链路):
|
独立 / 外设(不在主链路):
|
||||||
aibox :8765 (Python WebSocket, 独立产品)
|
aibox :8765 (Python WebSocket, 独立产品; --once CLI 单次对话 + models 多模型档案含 fallback 失败转移)
|
||||||
BotOps (aibox 的 TUI 前端)
|
BotOps (aibox 的 TUI 前端)
|
||||||
bot4cj (Cangjie + 树莓派 I2C, 硬件)
|
bot4cj (Cangjie + 树莓派 I2C, 硬件)
|
||||||
pybot (Python, bot4cj 的迁移版: .mt 解析/录制/实时闭环)
|
pybot (Python, bot4cj 的迁移版: .mt 解析/录制/实时闭环)
|
||||||
@@ -148,7 +148,7 @@ Blackbird 是一套 **进程托管式轻量微服务** 系统(自研 UniServe
|
|||||||
| **aibox** | Python | 8765 | 127.0.0.1 | 独立 AI 智能体(WebSocket) | 独立产品 |
|
| **aibox** | Python | 8765 | 127.0.0.1 | 独立 AI 智能体(WebSocket) | 独立产品 |
|
||||||
| **BotOps** | Python | — | — | aibox 的 TUI 前端 | 独立 |
|
| **BotOps** | Python | — | — | aibox 的 TUI 前端 | 独立 |
|
||||||
| **bot4cj** | Cangjie | — | 树莓派 I2C | 机器人脚本解析器 | 独立硬件 |
|
| **bot4cj** | Cangjie | — | 树莓派 I2C | 机器人脚本解析器 | 独立硬件 |
|
||||||
| **pybot** | Python | — | 树莓派 | bot4cj 的 Python 迁移版(.mt 解析、actpred episode 录制、--loop 实时预测闭环) | 独立硬件 |
|
| **pybot** | Python | — | 树莓派 | bot4cj 的 Python 迁移版(.mt 解析、actpred episode 录制、--loop 实时预测闭环;HEF 后端自动异步,推理与执行上一步动作并行) | 独立硬件 |
|
||||||
| **mtc** | Python | — | — | 命令式动作脚本(Python DSL)→ .mt 编译器(非阻塞时间轴 IR + 10ms 栅格化),pybot/bot4cj 的脚本 authoring 工具 | 开发期工具 |
|
| **mtc** | Python | — | — | 命令式动作脚本(Python DSL)→ .mt 编译器(非阻塞时间轴 IR + 10ms 栅格化),pybot/bot4cj 的脚本 authoring 工具 | 开发期工具 |
|
||||||
| **lci** | C++ | — | /dev/fb0 | 嵌入式帧缓冲 GUI | 独立硬件 |
|
| **lci** | C++ | — | /dev/fb0 | 嵌入式帧缓冲 GUI | 独立硬件 |
|
||||||
| **actpred** | Python | — | — | 动作预测(图像+动作历史→下一动作),bot4cj/pybot 的「大脑」配套 | 实验 |
|
| **actpred** | Python | — | — | 动作预测(图像+动作历史→下一动作),bot4cj/pybot 的「大脑」配套 | 实验 |
|
||||||
@@ -236,6 +236,7 @@ Blackbird 是一套 **进程托管式轻量微服务** 系统(自研 UniServe
|
|||||||
- **smtp3 待上线**:C++ 重写已完成并通过安全审计,但 services.yaml 仍指向 Python `smtp`,建议择机平替以消除最后一块 Python 有状态服务。
|
- **smtp3 待上线**:C++ 重写已完成并通过安全审计,但 services.yaml 仍指向 Python `smtp`,建议择机平替以消除最后一块 Python 有状态服务。
|
||||||
- **可观测性缺口**:缺 tracing/metrics/告警,建议后续引入轻量方案(或复用 db2 Logs + 定时聚合)。
|
- **可观测性缺口**:缺 tracing/metrics/告警,建议后续引入轻量方案(或复用 db2 Logs + 定时聚合)。
|
||||||
- **水平扩展限制**:smtp/captcha 内存态为单点,若需多实例需引入 Redis 等共享存储。
|
- **水平扩展限制**:smtp/captcha 内存态为单点,若需多实例需引入 Redis 等共享存储。
|
||||||
|
- **panel 面板微服务(规划中)**:设计稿见 [docs/panel_service.md](panel_service.md)——背景视图 + 各服务自持控件片段(iframe 沙箱 + BFF 转发),解决整页应用互相改前端代码的耦合问题;落地后需三处同步(本文件 / MODULES.md / 根 README)。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+4
-4
@@ -44,7 +44,7 @@
|
|||||||
|
|
||||||
| 模块 | 语言 | 端口 | 职责 |
|
| 模块 | 语言 | 端口 | 职责 |
|
||||||
|------|------|------|------|
|
|------|------|------|------|
|
||||||
| aibox | Python | 8765 | 独立 AI 智能体系统(DeepSeek + WebSocket),自带技能系统/记忆/任务调度/前端与语音客户端 |
|
| aibox | Python | 8765 | 独立 AI 智能体系统(DeepSeek + WebSocket),自带技能系统/记忆/任务调度/前端与语音客户端;`--once` CLI 单次对话(stdout/stderr 分流),`models` 配置节支持多 OpenAI 兼容端点按档案名切换 |
|
||||||
| BotOps | Python/Textual | — | aibox 的终端 TUI 前端,WebSocket 连接 aibox,Markdown 增量流式渲染、思考模式、任务侧栏、自动重连 |
|
| BotOps | Python/Textual | — | aibox 的终端 TUI 前端,WebSocket 连接 aibox,Markdown 增量流式渲染、思考模式、任务侧栏、自动重连 |
|
||||||
| actpred | Python/PyTorch | — | 动作预测(图像 + 动作历史 → 预测下一个动作),bot4cj 的「大脑」配套;支持从 `.mt` 脚本预训练 |
|
| actpred | Python/PyTorch | — | 动作预测(图像 + 动作历史 → 预测下一个动作),bot4cj 的「大脑」配套;支持从 `.mt` 脚本预训练 |
|
||||||
|
|
||||||
@@ -53,7 +53,7 @@
|
|||||||
| 模块 | 语言 | 端口 | 职责 |
|
| 模块 | 语言 | 端口 | 职责 |
|
||||||
|------|------|------|------|
|
|------|------|------|------|
|
||||||
| bot4cj | 仓颉/Cangjie | — (树莓派 I2C) | 机器人硬件脚本解析器,解析 `.mt` 驱动 PCA9685 舵机 / MPU6050 陀螺仪 / 扬声器,面向 Dashun 蜘蛛机器人 |
|
| bot4cj | 仓颉/Cangjie | — (树莓派 I2C) | 机器人硬件脚本解析器,解析 `.mt` 驱动 PCA9685 舵机 / MPU6050 陀螺仪 / 扬声器,面向 Dashun 蜘蛛机器人 |
|
||||||
| pybot | Python | — (树莓派) | bot4cj 的 Python 迁移版:`.mt` 动作脚本解析器(零第三方依赖,与仓颉版逐语义对齐);支持执行 / `--dry-run` / `--record` 录制 actpred episode(含 imu.npy)/ `--loop` 实时预测闭环 |
|
| pybot | Python | — (树莓派) | bot4cj 的 Python 迁移版:`.mt` 动作脚本解析器(零第三方依赖,与仓颉版逐语义对齐);支持执行 / `--dry-run` / `--record` 录制 actpred episode(含 imu.npy)/ `--loop` 实时预测闭环(HEF 后端自动异步:推理与执行上一步动作并行) |
|
||||||
| mtc | Python | — | 命令式动作脚本(Python DSL)→ `.mt` 编译器:`exec` 注入命名空间 → 时间轴 IR(非阻塞语义、重叠截断)→ 10ms 栅格化;产物可由 pybot/bot4cj 直接执行,零第三方依赖 |
|
| mtc | Python | — | 命令式动作脚本(Python DSL)→ `.mt` 编译器:`exec` 注入命名空间 → 时间轴 IR(非阻塞语义、重叠截断)→ 10ms 栅格化;产物可由 pybot/bot4cj 直接执行,零第三方依赖 |
|
||||||
| lci | C++ | — (/dev/fb0) | 嵌入式帧缓冲图形界面(无需 X11/Wayland),矢量 GB2312 中文字库、图片渲染、实时数据可视化 |
|
| lci | C++ | — (/dev/fb0) | 嵌入式帧缓冲图形界面(无需 X11/Wayland),矢量 GB2312 中文字库、图片渲染、实时数据可视化 |
|
||||||
| 3d | — (资产) | — | 机器人 / 机械臂 CAD 模型(STEP/STL/SLDPRT),bot4cj 配套硬件资产 |
|
| 3d | — (资产) | — | 机器人 / 机械臂 CAD 模型(STEP/STL/SLDPRT),bot4cj 配套硬件资产 |
|
||||||
@@ -77,7 +77,7 @@
|
|||||||
| 模块 | 说明 |
|
| 模块 | 说明 |
|
||||||
|------|------|
|
|------|------|
|
||||||
| tests | 黑盒测试套件(`test_db.py`/`test_gateway.py`/`test_waf.py`/`test_dnfs.py`/`test_captcha.py`/`test_smtp.py`/`test_ai_review.py`/`test_static.py`/`test_core_confs.py`),纯标准库 HTTP/TCP 用户旅程模拟,经 `test_runner.py` 统一调度(详见 [tests/blackbox/README.md](../tests/blackbox/README.md)) |
|
| tests | 黑盒测试套件(`test_db.py`/`test_gateway.py`/`test_waf.py`/`test_dnfs.py`/`test_captcha.py`/`test_smtp.py`/`test_ai_review.py`/`test_static.py`/`test_core_confs.py`),纯标准库 HTTP/TCP 用户旅程模拟,经 `test_runner.py` 统一调度(详见 [tests/blackbox/README.md](../tests/blackbox/README.md)) |
|
||||||
| docs | 文档聚合(security/ 审计、reviews/ 审查、reports/ 分析),70 篇 md |
|
| docs | 文档聚合(security/ 审计、reviews/ 审查、reports/ 分析),76 篇 md |
|
||||||
|
|
||||||
## 九、根级构建与部署脚本
|
## 九、根级构建与部署脚本
|
||||||
|
|
||||||
@@ -92,5 +92,5 @@
|
|||||||
- **日志链路**:各服务 `log_client.py`(异步批量)→ `db2:8005` Logs 表(fail-safe,db2 不可用时静默丢弃)
|
- **日志链路**:各服务 `log_client.py`(异步批量)→ `db2:8005` Logs 表(fail-safe,db2 不可用时静默丢弃)
|
||||||
- **配置链路**:`gateway` → `core` 内嵌 confs:8010(内存缓存 + mtime 重载,仅 gateway.ini);其余服务配置纯 CLI 传递
|
- **配置链路**:`gateway` → `core` 内嵌 confs:8010(内存缓存 + mtime 重载,仅 gateway.ini);其余服务配置纯 CLI 传递
|
||||||
- **数据/审核/邮件/缓存**:`dnfs2` → `db2:8005`(pymsg 持久化);`dnfs2` → `ai_review:8007`(审核);注册邮件 `ca` → `smtp:8008`;登录验证码 `ca` → `captcha2:8009`;`memdb:8011`(内存缓存,libmsg RPC,暂无业务调用方,预留热点数据/会话缓存)
|
- **数据/审核/邮件/缓存**:`dnfs2` → `db2:8005`(pymsg 持久化);`dnfs2` → `ai_review:8007`(审核);注册邮件 `ca` → `smtp:8008`;登录验证码 `ca` → `captcha2:8009`;`memdb:8011`(内存缓存,libmsg RPC,暂无业务调用方,预留热点数据/会话缓存)
|
||||||
- **机器人数据链**:`mtc` 编译 Python DSL → `.mt`(或手写 `.mt`)→ `pybot` 执行 / `--record` 录制 episode(images + actions.npy + imu.npy)→ `actpred` 两阶段训练 → ONNX/Hailo HEF 部署 → `pybot --loop` 实时预测闭环(bot4cj 语义对齐)
|
- **机器人数据链**:`mtc` 编译 Python DSL → `.mt`(或手写 `.mt`)→ `pybot` 执行 / `--record` 录制 episode(images + actions.npy + imu.npy)→ `actpred` 两阶段训练 → ONNX/Hailo HEF 部署(`predict_async` 异步流水线)→ `pybot --loop` 实时预测闭环(bot4cj 语义对齐,HEF 后端推理与执行并行)
|
||||||
- **端口一览**:80 redirect80 · 443 waf · 8443 ca · 8445 gateway · 5200/8010 core · 9200 fsys · 8006 dnfs2 · 8005 db2 · 8007 ai_review · 8009 captcha2 · 8008 smtp · 8011 memdb · 8765 aibox
|
- **端口一览**:80 redirect80 · 443 waf · 8443 ca · 8445 gateway · 5200/8010 core · 9200 fsys · 8006 dnfs2 · 8005 db2 · 8007 ai_review · 8009 captcha2 · 8008 smtp · 8011 memdb · 8765 aibox
|
||||||
@@ -0,0 +1,254 @@
|
|||||||
|
# 面板微服务(panel)开发文档
|
||||||
|
|
||||||
|
> 状态:设计稿 v0.1(开发前评审)
|
||||||
|
> 关联:[gateway/README.md](../gateway/README.md)、[docs/dnfs_custom_themes.md](dnfs_custom_themes.md)、[core/services.yaml](../core/services.yaml)、[libhttp/README.md](../libhttp/README.md)
|
||||||
|
|
||||||
|
## 1. 背景与目标
|
||||||
|
|
||||||
|
### 1.1 问题
|
||||||
|
|
||||||
|
现有前端页面(fsys 托管的 `index.html` / `dnfs.html` 等)都是"整页应用":每个页面由单个服务完整提供。当多个服务都想在首页/工作台展示自己的能力时,只能互相改动前端代码或各自开新页面,服务间耦合、发布相互牵连。
|
||||||
|
|
||||||
|
### 1.2 目标
|
||||||
|
|
||||||
|
新增**面板微服务**(panel),提供一张**背景视图**(工作台/仪表盘)。视图内的元素与控件**不由面板自己实现**,而是由各业务微服务以**前端代码片段**的形式提供,面板负责聚合、渲染与托管:
|
||||||
|
|
||||||
|
- 各服务**自持控件代码**(代码维护在各自模块),通过约定协议暴露片段
|
||||||
|
- 面板**拉取并聚合**片段,以 iframe 沙箱插入背景视图
|
||||||
|
- 面板只提供统一外壳(布局 / 主题 / 通信 / 身份透传),**不感知控件业务**
|
||||||
|
|
||||||
|
### 1.3 非目标(v1)
|
||||||
|
|
||||||
|
- 不做控件可视化拖拽编排(v2 候选)
|
||||||
|
- 不做用户自定义控件 / 第三方插件市场(控件仅来自内部服务,白名单)
|
||||||
|
- 不做控件代码的 AI 内容审核——控件是内部服务代码而非用户输入,信任边界与 dnfs2 用户样式节点不同(见 5.3)
|
||||||
|
|
||||||
|
## 2. 架构设计
|
||||||
|
|
||||||
|
### 2.1 拓扑
|
||||||
|
|
||||||
|
```
|
||||||
|
浏览器
|
||||||
|
│ https + q4_token Cookie / Authorization
|
||||||
|
▼
|
||||||
|
waf:443 ──► ca:8443(验签,注入 X-Auth-User / X-Auth-Admin)
|
||||||
|
│ Target-Url: http://127.0.0.1:9201
|
||||||
|
▼
|
||||||
|
gateway:8445 ──► panel:9201(Proxy.opened 白名单)
|
||||||
|
│
|
||||||
|
┌────────────────┼──────────────────────┐
|
||||||
|
│ 内部 HTTP(--providers 白名单) │ BFF 转发
|
||||||
|
▼ ▼ ▼
|
||||||
|
dnfs2:8006 fsys:9200 其他 provider 服务
|
||||||
|
/panel-widget/x /panel-widget/y
|
||||||
|
```
|
||||||
|
|
||||||
|
- 面板对浏览器只暴露**一个 origin**(面板自己);控件 iframe 由面板端点在**同一 origin** 下渲染,不存在跨域、自定义头问题(浏览器无法设置 `Target-Url`,因此控件回源统一走面板 BFF,对齐 fsys→dnfs2 的服务端代理先例)。
|
||||||
|
- 面板加入 `Proxy.opened` 白名单后走 ca 受保护链路:只有登录用户能打开面板;BFF 转发时透传身份头。
|
||||||
|
|
||||||
|
### 2.2 核心概念
|
||||||
|
|
||||||
|
| 概念 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| Panel | 面板服务本身:背景视图 + 聚合 / 渲染 / 托管 |
|
||||||
|
| Widget(控件) | 一个可独立渲染的前端单元,由某微服务提供 |
|
||||||
|
| Provider(提供方) | 拥有控件代码的微服务,实现 `/panel-widget/<id>` 协议 |
|
||||||
|
| Manifest | 控件描述:元数据 + html/css/js 片段 + 布局 / 主题声明 |
|
||||||
|
|
||||||
|
### 2.3 控件生命周期
|
||||||
|
|
||||||
|
1. **声明**:服务在自身实现 `GET /panel-widget/<id>`(内部 HTTP)
|
||||||
|
2. **拉取**:面板按 `--widgets` 清单拉取各 manifest(缓存 + TTL)
|
||||||
|
3. **聚合**:面板把 manifest 渲染为独立子页面 `/widget/<id>`(内联片段 + 白名单 CSP)
|
||||||
|
4. **插入**:面板页用 `<iframe sandbox="allow-scripts">` 加载子页面,完成"背景视图 + 控件"组合
|
||||||
|
5. **通信**:控件与面板用 postMessage 协议交互(就绪 / 调尺寸 / 主题 / 导航)
|
||||||
|
6. **回源**:控件业务数据经面板 BFF 转发到所属服务(透传身份头)
|
||||||
|
|
||||||
|
### 2.4 端口与技术栈
|
||||||
|
|
||||||
|
| 项 | 值 | 理由 |
|
||||||
|
|---|---|---|
|
||||||
|
| 端口 | 9201 | 与 fsys:9200 相邻的 HTTP 对外段;8xxx 保留给 pymsg/TCP 内部服务 |
|
||||||
|
| 语言 | C++20 + libhttp | 与 db2 / dnfs2 / fsys / captcha2 生态一致 |
|
||||||
|
| 日志 | liblogc(`us/LogC.hpp`) | 统一异步推送 db2 Logs 表 |
|
||||||
|
|
||||||
|
## 3. 服务接口
|
||||||
|
|
||||||
|
### 3.1 对外(浏览器,经 gateway)
|
||||||
|
|
||||||
|
| 路由 | 方法 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `/` | GET | 面板页(背景视图 + 控件网格) |
|
||||||
|
| `/widgets` | GET | 控件清单 JSON(面板缓存内的 manifest 元数据 + 状态),供页面 JS 渲染网格 |
|
||||||
|
| `/widget/<id>` | GET | 控件子页面(内联 html/css/js),供 iframe 加载;响应带独立 CSP(见 5.1) |
|
||||||
|
| `/api/w/<id>/<path...>` | 任意 | BFF:转发到 `<id>` 所属 provider,透传身份头 |
|
||||||
|
| `/health` | GET | 健康检查 |
|
||||||
|
|
||||||
|
### 3.2 对 Provider(内部 HTTP 协议)
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /panel-widget/<widget_id>
|
||||||
|
|
||||||
|
200 OK, application/json
|
||||||
|
{
|
||||||
|
"id": "dnfs2-minimap", // 必须与路径一致
|
||||||
|
"title": "树状菜单迷你图", // 面板标题(≤ 64 字符)
|
||||||
|
"version": 3, // 递增版本,作为面板缓存键
|
||||||
|
"size": {"w": 2, "h": 1}, // 网格单位(1 ≤ w,h ≤ 6)
|
||||||
|
"themeable": true, // 是否响应面板主题事件
|
||||||
|
"html": "<div class=\"w-root\">…</div>",
|
||||||
|
"css": ".w-root { … }",
|
||||||
|
"js": "window.wx.ready(); …"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
面板强制校验(失败视为该控件不可用,返回占位):
|
||||||
|
- `html` / `css` / `js` 各 ≤ 256KB;`id` 必须与 URL 一致;所有字段必须存在
|
||||||
|
- 拉取超时 3s;TTL 默认 60s(`--cache-ttl`)
|
||||||
|
- provider 不可达 / 校验失败 → 占位卡片「控件暂不可用」,**fail-soft**:面板是展示性服务,一个控件失败不拖垮整页(区别于 ai_review 审核类的 fail-closed 语义)
|
||||||
|
|
||||||
|
### 3.3 面板配置(core/services.yaml)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: panel
|
||||||
|
cmd: bash /opt/micro_servers/panel/_run.sh
|
||||||
|
args:
|
||||||
|
- "--instance"
|
||||||
|
- "panel"
|
||||||
|
- "--log-level"
|
||||||
|
- "info"
|
||||||
|
- "--host"
|
||||||
|
- "127.0.0.1"
|
||||||
|
- "--port"
|
||||||
|
- "9201"
|
||||||
|
# 控件可见清单(由面板控制;未列出的控件不渲染)
|
||||||
|
- "--widgets"
|
||||||
|
- "dnfs2-minimap;dnfs2-recent;fsys-storage"
|
||||||
|
# 提供方映射(白名单:面板只向这些地址拉取 / 转发)
|
||||||
|
- "--providers"
|
||||||
|
- "dnfs2=http://127.0.0.1:8006;fsys=http://127.0.0.1:9200"
|
||||||
|
- "--cache-ttl"
|
||||||
|
- "60"
|
||||||
|
depends_on: [dnfs2, fsys]
|
||||||
|
sleep_before: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
同时修改 `core/configs/gateway.ini` 白名单:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Proxy]
|
||||||
|
opened = http://127.0.0.1:9200;http://127.0.0.1:8006;http://127.0.0.1:5200;http://127.0.0.1:9201;
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. 控件协议(前端侧)
|
||||||
|
|
||||||
|
### 4.1 运行环境约定
|
||||||
|
|
||||||
|
控件运行在 `<iframe sandbox="allow-scripts">` 中:
|
||||||
|
|
||||||
|
- **无 `allow-same-origin`**:iframe 内为 opaque origin,读不到主页面 DOM / Cookie / localStorage
|
||||||
|
- 控件只能:渲染自己的片段、发 postMessage、`fetch` 面板同源路径(经 BFF)
|
||||||
|
- 控件不得假设自己就是 provider 域:所有业务调用走 `fetch('/api/w/<id>/…')`
|
||||||
|
|
||||||
|
### 4.2 postMessage 事件
|
||||||
|
|
||||||
|
面板 → 控件:
|
||||||
|
|
||||||
|
| 事件 | payload | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `panel.theme` | `{theme:"dark"\|"light"}` | 主题切换(仅 `themeable: true` 的控件) |
|
||||||
|
| `panel.user` | `{user:"user@example.com"}` | 当前登录用户(控件需要个性化时) |
|
||||||
|
|
||||||
|
控件 → 面板:
|
||||||
|
|
||||||
|
| 事件 | payload | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `widget.ready` | `{id}` | 控件渲染完成,面板收起 loading |
|
||||||
|
| `widget.resize` | `{id, w, h}` | 请求调整网格占位(面板限幅在声明 size 的 ±2 内) |
|
||||||
|
| `widget.navigate` | `{id, url}` | 请求面板同源跳转(url 必须同 origin,面板校验) |
|
||||||
|
| `widget.toast` | `{id, message}` | 轻提示 |
|
||||||
|
| `widget.log` | `{id, level, message}` | 前端日志(面板转发 db2 Logs) |
|
||||||
|
|
||||||
|
面板侧校验:优先校验 `e.origin === location.origin`;`payload.id` 必须与绑定的 iframe 一致;必要时以 `e.source === iframe.contentWindow` 兜底校验(opaque origin 场景)。
|
||||||
|
|
||||||
|
### 4.3 布局与主题
|
||||||
|
|
||||||
|
- v1 用 CSS Grid(12 列),按 `manifest.size` 自动流式排布(`grid-auto-flow: dense`),控件间留隙
|
||||||
|
- 面板提供统一控件外壳(标题栏 = title + 版本 + 刷新按钮),内容区为 iframe
|
||||||
|
- 主题变量沿用 dnfs2 风格(`--bg` / `--fg` / `--accent` / …),切换时经 `panel.theme` 同步给 themeable 控件
|
||||||
|
|
||||||
|
## 5. 安全设计
|
||||||
|
|
||||||
|
### 5.1 控件隔离(核心决策)
|
||||||
|
|
||||||
|
**用 iframe 子页面端点 `/widget/<id>`,而非 `srcdoc`。** 原因:
|
||||||
|
|
||||||
|
- `srcdoc` 文档**继承父页面 CSP**;面板页若保持严格 CSP(`script-src 'self'`),会连带封死控件的内联脚本
|
||||||
|
- 独立端点可携带**控件文档自己的 CSP**,实现"父页面严格、控件文档自管"的分层
|
||||||
|
|
||||||
|
```http
|
||||||
|
HTTP/1.1 200 OK
|
||||||
|
Content-Type: text/html
|
||||||
|
Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; script-src 'unsafe-inline'; img-src 'self' data:; connect-src 'self'; sandbox allow-scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
配合 iframe 属性双保险:`<iframe sandbox="allow-scripts" src="/widget/<id>">`。即使 CSP 头缺失,`sandbox` 属性仍保证 opaque origin + 禁止顶层导航。
|
||||||
|
|
||||||
|
### 5.2 信任边界
|
||||||
|
|
||||||
|
| 边界 | 策略 |
|
||||||
|
|---|---|
|
||||||
|
| provider 拉取 | 仅 `--providers` 白名单地址,内部 HTTP;manifest 全字段校验 + 大小上限 |
|
||||||
|
| 浏览器直连 | 面板在 `Proxy.opened`,必须经 ca 验签;身份头存在性由 gateway 不变量保证 |
|
||||||
|
| BFF 转发 | 目标仅限该 widget 所属 provider;透传 `X-Auth-User` / `X-Auth-Admin` / `Authorization`,且 strip-then-set 剥离客户端伪造值(对齐 ca 契约) |
|
||||||
|
| `widget.navigate` | url 必须同 origin(`new URL(u).origin === location.origin`),否则拒绝 |
|
||||||
|
|
||||||
|
### 5.3 日志与审计
|
||||||
|
|
||||||
|
- 拉取失败 / 超时 / 校验失败 → db2 Logs(warn 级别)
|
||||||
|
- manifest `version` 变化 → info 日志(审计"谁升级了控件")
|
||||||
|
- 明确不做内容审核:控件来自内部服务而非用户输入,信任边界在 provider 白名单处收口(与 dnfs2 用户样式节点区分,如需对用户内容把关仍走 dnfs2 的 ai_review 链路)
|
||||||
|
|
||||||
|
## 6. 目录结构(建议)
|
||||||
|
|
||||||
|
```
|
||||||
|
panel/
|
||||||
|
├── Makefile
|
||||||
|
├── _set.sh / _run.sh
|
||||||
|
├── src/
|
||||||
|
│ ├── main.cpp # libhttp HttpServer + 路由注册 + 日志
|
||||||
|
│ ├── widget_store.cpp # provider 拉取 / manifest 校验 / TTL 缓存
|
||||||
|
│ └── bff.cpp # /api/w/<id> 转发(白名单 + 身份头透传)
|
||||||
|
└── public/
|
||||||
|
├── panel.html # 背景视图 + 网格外壳
|
||||||
|
└── panel.js # 拉 /widgets → 建 iframe → postMessage 协议
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. 实现阶段
|
||||||
|
|
||||||
|
| 阶段 | 内容 | 验收 |
|
||||||
|
|---|---|---|
|
||||||
|
| M1 | 骨架:`_set.sh` / `_run.sh`、libhttp 服务、`/health`、静态面板页、widget_store 拉取 + 缓存、`/widgets`、`/widget/<id>` | curl 可见清单与子页面 |
|
||||||
|
| M2 | 页面 JS:网格渲染、iframe sandbox 加载、postMessage 协议、loading / 占位态 | 浏览器两个示例控件可显示 |
|
||||||
|
| M3 | BFF 转发 + 主题同步 + `widget.log` → db2 | 控件调通自己后端接口 |
|
||||||
|
| M4 | 安全加固审计(CSP / 沙箱 / 白名单复核)+ `tests/blackbox/test_panel.py` + 本文档定稿 | 黑盒测试通过 |
|
||||||
|
|
||||||
|
## 8. 测试(tests/blackbox/test_panel.py 思路)
|
||||||
|
|
||||||
|
- 起一个 mock provider(Python 标准库 HTTP server)提供两个 manifest
|
||||||
|
- 断言项:`/widgets` 返回清单;`/widget/<id>` 响应含 CSP `sandbox` 头与内联片段;provider 下线后 `/widgets` 状态为不可用占位;BFF 转发透传 `X-Auth-User`;超大 manifest 被拒;未列入 `--widgets` 的控件不渲染
|
||||||
|
|
||||||
|
## 9. 风险与取舍
|
||||||
|
|
||||||
|
| 取舍 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| iframe 开销 | 每个控件一个 iframe(含独立文档 + CSP),内存与加载略重;换来的是控件间样式/脚本彻底隔离,控件可独立演进 |
|
||||||
|
| 聚合单点 | 面板是控件编排的单点;控件业务能力仍分布在各自服务,面板故障只影响聚合视图,不影响各服务自身 |
|
||||||
|
| 无在线编排 | v1 布局静态声明(manifest.size + 面板清单顺序);拖拽/自定义布局留给 v2 |
|
||||||
|
|
||||||
|
## 10. 参考
|
||||||
|
|
||||||
|
- gateway 转发白名单与身份契约:[gateway/README.md](../gateway/README.md)
|
||||||
|
- 注入机制先例(js 特效 iframe 沙箱 / style 主题):[docs/dnfs_custom_themes.md](dnfs_custom_themes.md)
|
||||||
|
- 服务注册与 CLI 配置约定:[core/services.yaml](../core/services.yaml)、[ca/README.md](../ca/README.md)(strip-then-set 契约)
|
||||||
|
- HTTP 服务端库:[libhttp/README.md](../libhttp/README.md)
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# actpred Hailo 推理速度测试报告:cv2.LUT 预处理优化验收
|
||||||
|
|
||||||
|
> 文档类型:测试报告
|
||||||
|
> 模块:actpred(hailo_infer.py cv2.LUT 快路径)
|
||||||
|
> 作者:自动生成
|
||||||
|
> 日期:2026-08-17
|
||||||
|
> 前置:`actpred_hailo_preprocess_optimization_plan_20260817.md`(优化方案,已实施)、
|
||||||
|
> `actpred_inference_speed_benchmark_20260813.md`(旧基准)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 测试环境
|
||||||
|
|
||||||
|
| 项 | 值 |
|
||||||
|
|---|---|
|
||||||
|
| 设备 | Raspberry Pi 5(4×A76)+ AI HAT 2(Hailo-10H,/dev/h1x-0) |
|
||||||
|
| 软件 | hailo_platform 5.3.0、opencv-python-headless 5.0.0、torch CPU 2.x |
|
||||||
|
| 模型 | actpred_v4.hef(onehot 1864 logits,fixed_range,640×480×3 帧输入) |
|
||||||
|
| 数据 | 新录 640×480 数据集 episode_0000(图像与模型输入同尺寸,免 resize) |
|
||||||
|
| 输入 | 3 帧 640×480 RGB + 5×18 动作历史(stand 姿态,spk 列 0/10 契约) |
|
||||||
|
|
||||||
|
## 2. 测试方法
|
||||||
|
|
||||||
|
- 预热 10 次后连续 200 次取均值(端到端 `predict` 调用,含预处理/绑定/NPU/解码)
|
||||||
|
- 分段计时:预处理(LUT 量化 + 绑定)/ NPU run / 后处理(反量化 + 1864 logits 解码)
|
||||||
|
- 对照:PyTorch CPU 同模型同输入(200 次均值)
|
||||||
|
|
||||||
|
## 3. 结果
|
||||||
|
|
||||||
|
### 3.1 端到端延迟
|
||||||
|
|
||||||
|
| 后端 | 延迟 | 相对优化前 |
|
||||||
|
|---|---|---|
|
||||||
|
| HEF(cv2.LUT 快路径) | **16.1 ms** | 52.4 → 16.1 ms(**3.3×**) |
|
||||||
|
| PyTorch CPU | 107.9 ms | — |
|
||||||
|
| 加速比(HEF vs CPU) | **6.7×** | 优化前 2.1× |
|
||||||
|
|
||||||
|
### 3.2 流水线分段(HEF)
|
||||||
|
|
||||||
|
| 阶段 | 耗时 | 占比 |
|
||||||
|
|---|---|---|
|
||||||
|
| 预处理(cv2.LUT + 输入绑定) | 3.0 ms | 19% |
|
||||||
|
| NPU run(HEF 纯推理) | 12.5 ms | 78% |
|
||||||
|
| 后处理(反量化 + decode) | 0.6 ms | 4% |
|
||||||
|
|
||||||
|
**瓶颈已从 Python 预处理转移到 NPU 本身**——预处理由优化前 26.5ms 降至 3.0ms(8.8×)。
|
||||||
|
|
||||||
|
### 3.3 闭环控制预算
|
||||||
|
|
||||||
|
| 控制率 | 周期预算 | HEF 占用 |
|
||||||
|
|---|---|---|
|
||||||
|
| 10 Hz | 100 ms | 16% |
|
||||||
|
| 20 Hz | 50 ms | 32% |
|
||||||
|
| 30 Hz | 33 ms | 48% |
|
||||||
|
|
||||||
|
10Hz 闭环下 NPU 仅占 16% 预算;上探 30Hz 也只占一半,余量充足。
|
||||||
|
|
||||||
|
## 4. 一致性验收
|
||||||
|
|
||||||
|
### 4.1 cv2.LUT vs float 路径输出(NPU 实测,max_diff)
|
||||||
|
|
||||||
|
| 输入类型 | max_diff |
|
||||||
|
|---|---|
|
||||||
|
| 640×480 RGB(主路径直通) | 0.0000 |
|
||||||
|
| 224×224 RGB(PIL resize) | 0.0000 |
|
||||||
|
| 全零帧 / 全 255 帧 | 0.0000 |
|
||||||
|
| 灰度 (H,W)(PIL 转 RGB) | 0.0000 |
|
||||||
|
| RGBA (H,W,4)(PIL 合成) | 0.0000 |
|
||||||
|
| 奇数尺寸 333×777(PIL resize) | 0.0000 |
|
||||||
|
| 随机帧 + 随机历史 × 8 seed | 0.0000(worst) |
|
||||||
|
|
||||||
|
### 4.2 无 cv2 回退
|
||||||
|
|
||||||
|
`HAS_CV2=False` 时 `_frame_luts=None` → float 路径,输出与 cv2 路径 max_diff = 0.0。
|
||||||
|
|
||||||
|
### 4.3 回归验收
|
||||||
|
|
||||||
|
| 项 | 结果 |
|
||||||
|
|---|---|
|
||||||
|
| `_npu_test.py` 4 角(真实/全零帧 × 真实/全零历史) | PASS(img/hist 敏感度 > 0.1) |
|
||||||
|
| HEF vs torch(原始动作空间) | 0.30°(预算 ≤1°) |
|
||||||
|
| 全套单测(test_actpred.py) | 188 tests OK |
|
||||||
|
|
||||||
|
## 5. 结论
|
||||||
|
|
||||||
|
1. cv2.LUT 快路径端到端 **16.1ms**,相比优化前 52.4ms 提升 3.3×;预处理不再是瓶颈
|
||||||
|
2. 与 float 路径**逐点一致**(7 类输入 + 8 seed 随机全部 0 差异),无隐藏分支
|
||||||
|
3. 无 cv2 环境自动回退,行为不变;量化误差(vs torch 0.30°)未回归
|
||||||
|
4. 10Hz 闭环预算占用 16%,支持控制率上探至 30Hz
|
||||||
|
|
||||||
|
## 6. 附注
|
||||||
|
|
||||||
|
- 优化前数据(52.4ms/26.5ms)来自 2026-08-17 上午 640×480 数据集基准
|
||||||
|
- cv2.LUT 的"归一化+量化"合并表数学上与 float 路径等价(0 LSB 取整差)
|
||||||
|
- 依赖:opencv-python-headless>=4.9(已入 requirements.txt;缺失仅损失加速,功能正常)
|
||||||
@@ -0,0 +1,154 @@
|
|||||||
|
# actpred Hailo 推理预处理优化:cv2.LUT 查表合并归一化与量化(已实施)
|
||||||
|
|
||||||
|
> 文档类型:开发方案(已实施)
|
||||||
|
> 模块:actpred(部署链 hailo_infer.py)
|
||||||
|
> 作者:自动生成
|
||||||
|
> 日期:2026-08-17(原型实测 + 集成实施)
|
||||||
|
> 前置:`actpred_inference_speed_benchmark_20260813.md`(速度基准)、
|
||||||
|
> 2026-08-17 新 640×480 数据集重测(本方案 §2 数据)
|
||||||
|
|
||||||
|
> 📌 **版本历程**
|
||||||
|
> 1. 初版:numpy 查表法为核心,预期端到端 18-20ms
|
||||||
|
> 2. 原型实测(§3.4):**numpy 高级索引查表在 Pi5 上无加速**(fancy
|
||||||
|
> indexing 的 gather 开销与逐元素 float 运算相当),预期不成立;
|
||||||
|
> 合并系数乘加仅 -5ms;**cv2.LUT 实测 15.8× 加速**(§3.4)
|
||||||
|
> 3. **已实施(§5)**:`hailo_infer.py` 集成 cv2.LUT 快路径(可选依赖,
|
||||||
|
> 无 cv2 自动回退 float 路径),端到端 52.4 → **16.0 ms**,输出与
|
||||||
|
> float 路径逐点一致(max_diff = 0)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 背景与动机
|
||||||
|
|
||||||
|
2026-08-17 用新录制的 640×480 数据集(免 resize 训练样本)重测 HEF 推理速度:
|
||||||
|
|
||||||
|
| 阶段 | 640×480 输入 | 224×224 输入(旧数据) |
|
||||||
|
|---|---|---|
|
||||||
|
| 端到端 predict | **52.4 ms** | 60.0 ms |
|
||||||
|
| 预处理 | **35.9 ms** | 43.1 ms |
|
||||||
|
| NPU run | 12.5 ms | 12.9 ms |
|
||||||
|
| 后处理 | 0.6 ms | 0.6 ms |
|
||||||
|
|
||||||
|
输入尺寸已与模型一致(resize 变 no-op),**预处理仍是端到端延迟的主导项**(68%)。
|
||||||
|
旧基准报告(2026-08-13)已记录同类问题:1080p 采集的预处理缩放约 45ms 主导端到端延迟,
|
||||||
|
当时通过把默认分辨率改回 640×480 缓解;本次分析证明 640×480 下 Python 端预处理
|
||||||
|
仍有约 36ms 的优化空间——但空间比初版估计小(见 §3.4 实测)。
|
||||||
|
|
||||||
|
## 2. 瓶颈分析(100 次均值分段计时)
|
||||||
|
|
||||||
|
| 子步骤 | 耗时 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `_frame_float`(3 帧) | **27.2 ms** | PIL fromarray/convert + resize(同尺寸 no-op 仍有开销)+ `(x/255-mean)/std` float 运算(92 万像素×3 通道) |
|
||||||
|
| `create_bindings` + 帧量化绑定 | **9.1 ms** | `_bind_frames`:float 归一化帧 → `round(x/q.scale+q.zp)` 再量化回 uint8 |
|
||||||
|
| history/pred 绑定 | 0.07 ms | 可忽略 |
|
||||||
|
|
||||||
|
**关键冗余**:当前流程先做 float 归一化(`_frame_float`),随后又量化回 uint8(`_bind_frames`)——
|
||||||
|
中间 float 帧是纯中间态,归一化与量化可以**合并为一步**。
|
||||||
|
|
||||||
|
## 3. 优化方案
|
||||||
|
|
||||||
|
### 3.1 主方案:合并系数乘加(纯 numpy,无新依赖)
|
||||||
|
|
||||||
|
HEF 帧输入是 uint8 量化值:`q = round(((x/255 - mean)/std) / q_scale + q_zp)`。
|
||||||
|
把 4 步运算合并为**一次乘加**(系数预计算,按通道):
|
||||||
|
|
||||||
|
```
|
||||||
|
q = round(x * A[c] + B[c])
|
||||||
|
A[c] = 1 / (255 * std[c] * q_scale)
|
||||||
|
B[c] = -mean[c] / (std[c] * q_scale) + q_zp
|
||||||
|
```
|
||||||
|
|
||||||
|
- 现流程 4 次全数组运算(`/255`、`-mean`、`/std`、`/q_scale+q_zp`)→ 1 次乘加
|
||||||
|
- 数值完全等价(实测 max_diff = 0 LSB,见 §3.4)
|
||||||
|
- 直通模式(`_normalize_frames=False`)同理:`A=1/q_scale, B=q_zp`
|
||||||
|
|
||||||
|
### 3.2 附带优化(与主方案正交)
|
||||||
|
|
||||||
|
1. **免 PIL 路径**:输入已是 np.ndarray 时直接运算,跳过 `Image.fromarray/convert`
|
||||||
|
2. **resize no-op 跳过**:`image.size == (W,H)` 时不调用 `resize()`
|
||||||
|
3. **缓冲复用**:`np.multiply/np.add` 带 `out=` 参数,减少临时数组分配
|
||||||
|
|
||||||
|
### 3.3 已实施快路径:cv2.LUT(可选依赖,无 cv2 自动回退)
|
||||||
|
|
||||||
|
`cv2.LUT(img, lut)` 是 C 实现的 uint8→uint8 查表,单次遍历,无 float 中间态——
|
||||||
|
把"归一化+量化"合并为一步。集成要点(`hailo_infer.py`):
|
||||||
|
|
||||||
|
- 惰性导入 cv2(`HAS_CV2`);**无 cv2 或非逐帧输入布局(ncfhw/nhwc_fused)自动回退 float 路径**
|
||||||
|
- 加载时按每帧 `quant_info` 构建 256 项表(归一化/直通两种,`_normalize_frames` 判定)
|
||||||
|
- **OpenCV 5.0 的 cv2.LUT 只接受单通道表**((256,3) 会断言失败),逐通道应用后 `cv2.merge`
|
||||||
|
- 输入尺寸与模型不一致时先 PIL resize(仅此情况走 PIL)
|
||||||
|
- 依赖安装:`opencv-python-headless`(5.0.0,清华镜像;官方源过慢)
|
||||||
|
- 注:本方案实测后发现**合并系数乘加(§3.1)未实施**——cv2.LUT 收益已远超,
|
||||||
|
避免冗余代码;§3.1 保留作为无 cv2 环境下的潜在备选
|
||||||
|
|
||||||
|
### 3.4 原型实测评估(2026-08-17,100-200 次均值,640×480×3 帧)
|
||||||
|
|
||||||
|
| 方案 | 3 帧耗时 | 对比现 float 28.5ms | 数值差异 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 现 float 路径 | 28.5 ms | 基线 | — |
|
||||||
|
| numpy 高级索引查表 | 26.6 ms | **无加速(1.0x)** | 0 LSB |
|
||||||
|
| 合并系数乘加 | 23.2 ms | 快 19%(-5ms) | 0 LSB |
|
||||||
|
| **cv2.LUT(逐通道)** | **1.68 ms** | **快 15.8×** | **0 LSB** |
|
||||||
|
|
||||||
|
**结论**:
|
||||||
|
- numpy fancy indexing 的 gather 开销与逐元素 float 运算相当,查表法**无收益**
|
||||||
|
- 纯 numpy 可达的最佳是合并乘加(-5ms),受限于 92 万像素 float32 数组的内存吞吐
|
||||||
|
- cv2.LUT 的 C 查表把帧预处理降一个数量级,且数值完全等价(0 LSB)
|
||||||
|
|
||||||
|
### 3.5 实测收益(已实施,集成后 NPU 实测)
|
||||||
|
|
||||||
|
| 指标 | 优化前 | 优化后(cv2.LUT) |
|
||||||
|
|---|---|---|
|
||||||
|
| 帧预处理(3 帧) | 26.5 ms | **3.0 ms(分段实测含绑定)** |
|
||||||
|
| 端到端 predict | 52.4 ms | **16.1 ms(3.3×)**(200 次均值,见测试报告) |
|
||||||
|
| 输出一致性 | — | 与 float 路径 max_diff = **0.0000**(7 类输入 + 8 seed 随机) |
|
||||||
|
| HEF vs torch | 0.50° | 0.30°(4 角复验 PASS) |
|
||||||
|
| vs PyTorch CPU | 2.1× | **6.7×**(16.1 vs 107.9 ms) |
|
||||||
|
|
||||||
|
闭环 10Hz(100ms 预算)占用率 52% → **16%**,20Hz 32%,30Hz(33ms)48%。
|
||||||
|
|
||||||
|
> 完整验收数据见 `actpred_cv2_lut_benchmark_report_20260817.md`(测试报告)。
|
||||||
|
|
||||||
|
## 4. 一致性约束(关键)
|
||||||
|
|
||||||
|
- **训练侧不动**:`dataset.py` 的 `Resize+ToTensor+Normalize` 保持 float 路径——
|
||||||
|
合并乘加只是推理侧对"归一化+量化"的合并实现,数值等价(实测 0 LSB)
|
||||||
|
- **三后端口径**:`predict.py`(PyTorch,float 输入不量化)、`onnx_infer.py`(ORT,float)
|
||||||
|
不受影响;优化仅作用于 `hailo_infer.py` 的 uint8 量化输入路径
|
||||||
|
- **帧输入判定不变**:`_normalize_frames`(量化网格负零偏判定)决定走
|
||||||
|
归一化还是直通,逻辑与现状一致
|
||||||
|
- 旧 5 输入 / 旧版单图像 HEF 布局同样适用(按各自 quant_info 算系数)
|
||||||
|
|
||||||
|
## 5. 实施步骤(已完成 ✅)
|
||||||
|
|
||||||
|
| 步骤 | 内容 | 状态 |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | `hailo_infer.py`:加载时按每帧 quant_info 构建 256 项 LUT(归一化/直通两种) | ✅ `_build_frame_luts` |
|
||||||
|
| 2 | `predict` 的 nhwc_frames 分支走 cv2.LUT 快路径(尺寸不匹配时 PIL resize);无 cv2 回退 float | ✅ `_frame_lut_quant` + `HAS_CV2` |
|
||||||
|
| 3 | NPU 实测:640×480 数据集 100 次均值 | ✅ 16.0 ms(52.4 → 16.0) |
|
||||||
|
| 4 | `_npu_test.py` 4 角 + HEF vs torch 对比复验 | ✅ PASS,0.30°(≤1° 预算) |
|
||||||
|
| 5 | 全套单测(TestHailoPredictor 等) | ✅ 188 tests OK |
|
||||||
|
| 6 | 依赖安装:opencv-python-headless 5.0.0(清华镜像) | ✅ venv 已装 |
|
||||||
|
|
||||||
|
## 6. 风险与对策
|
||||||
|
|
||||||
|
| 风险 | 对策 |
|
||||||
|
|---|---|
|
||||||
|
| 查表取整与 float 路径存在差异 | 实测 0 LSB(表项 float32 与现路径同精度);NPU 输出 vs float 路径 max_diff=0,vs torch 0.30° |
|
||||||
|
| 无 cv2 环境 | 惰性导入 + 自动回退 float 路径,行为与优化前完全一致 |
|
||||||
|
| OpenCV 5.0 多通道 LUT 断言失败 | 逐通道单表 + `cv2.merge`(实测无性能损失) |
|
||||||
|
| 非 nhwc_frames 布局(旧 HEF) | 仅该布局启用 cv2 路径,其余走原 float 路径 |
|
||||||
|
| 未来输入尺寸变更 | LUT 与尺寸无关(逐像素映射);尺寸不一致时才走 PIL resize |
|
||||||
|
|
||||||
|
## 7. 决策记录
|
||||||
|
|
||||||
|
- **查表法(初版核心)实测证伪**:numpy 高级索引 gather 在 Pi5 上与逐元素 float
|
||||||
|
运算同速(26.6 vs 28.5ms),"查表降为内存索引即快"的假设不成立——教训:
|
||||||
|
性能方案须原型实测,不能凭直觉估计
|
||||||
|
- **最终方案选 cv2.LUT**:C 实现查表把帧预处理从 26.5ms 降到 1.7ms(15.8×),
|
||||||
|
端到端 52.4 → 16.0ms;数值 0 LSB 等价。合并系数乘加(-5ms)收益被碾压,
|
||||||
|
不再实施(§3.1 保留为无 cv2 环境的备选)
|
||||||
|
- **依赖取舍**:opencv-python-headless 作为可选依赖(惰性导入 + 自动回退),
|
||||||
|
有 cv2 环境收益 3.3×,无 cv2 环境行为不变——收益远大于依赖成本
|
||||||
|
- **训练侧不改**:训练用 float 归一化(增强/数值精度),推理侧查表实现——两侧
|
||||||
|
口径在"归一化值"层面一致,量化差异由部署链固有预算吸收(HEF vs torch 0.30° 余量充足)
|
||||||
@@ -0,0 +1,437 @@
|
|||||||
|
# aibox 模块安全与质量审计报告
|
||||||
|
|
||||||
|
- **日期**:2026-08-17(第一轮)/ 2026-08-18(第二轮复审)/ 2026-08-18(第三轮:遗留复核 + v3.4 专审)/ 2026-08-18(第四轮:3 路并行审读 + Batch H 修复)
|
||||||
|
- **审计基线**:第一轮 HEAD `ce142fe`(Batch A–D 后)→ 第二轮 HEAD `177a9ac`(Batch E 修复后)→ 第三轮 HEAD `aa15841`(L4a/b/c 落地后)→ 第四轮 HEAD `d1f780e`(Batch G + SOCKS5 验证后)
|
||||||
|
- **审计范围**:`aibox.py`(~3310 行)+ `llm_client.py` + `tools/`(7 文件)+ `subsystems/`(6 文件)+ `common/`(5 文件)
|
||||||
|
- **方法**:第一轮逐文件代码审读 + PoC 动态验证;第二轮 Batch E 修复复审 + 全量重审 + 6 处关键发现实证抽查;第三轮遗留项复核 + v3.4 专审;第四轮 3 路并行审读 + 主审逐条复核 + Linux PoC 实证(6/6)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一、总体结论
|
||||||
|
|
||||||
|
**第一轮**(基线 ce142fe):架构设计(分层、参数化 SQL、密钥隔离、消息回滚)整体健康,但**命令执行链存在一条系统性断裂**:实际运行路径绕过了精心设计的执行器,导致 1MB 硬约束、超时、中断三项保护同时失效。
|
||||||
|
|
||||||
|
**第二轮**(基线 177a9ac):Batch E 五项修复(H1/H2/H3/M1/M2)中四项确认修复到位(PoC 5/5 验证),**M2 修复不完整**(超时路径跳过工具属性清理,僵尸输出问题仍存)。第二轮全量重审新发现 **1 高危 + 9 中危 + 若干低危**,其中三项与 Batch E 修复直接相关(reader 线程泄漏 / 子代理超时盲区 / M2 残留),其余为存量问题(代理密码明文落盘、drain 线程泄漏、事件循环阻塞等)。详见"八、第二轮复审"。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 二、高危发现(3 项)
|
||||||
|
|
||||||
|
### H1:`run_command` 实际路径绕过注入的 executor——三重保护同时失效
|
||||||
|
|
||||||
|
证据链:
|
||||||
|
- `aibox.py:1012` 将 `command_executor` 注入 `CommandLineTool`
|
||||||
|
- 但 `tools/command_tool.py:86-219` 的 `execute_streaming` 自带一套 **150 行的重复 Popen 实现**,`self.executor` 全程未使用(死注入)
|
||||||
|
- `tools/registry.py:71-86` 检测到 `execute_streaming` 即走该路径——**生产 100% 命中**
|
||||||
|
|
||||||
|
后果:
|
||||||
|
|
||||||
|
| 保护 | 设计位置 | 实际状态 |
|
||||||
|
|---|---|---|
|
||||||
|
| 1MB 输出上限(硬约束) | `executors.py:69` | **失效**——command_tool 的 `output_lines` 无限累积 |
|
||||||
|
| `cancel()` 中断 | `executors.py:76-86` 只 kill `_current_process` | **失效**——实际进程引用只在 command_tool 局部变量,三层中断的第三层对 run_command 不生效 |
|
||||||
|
| 超时/截断统一语义 | executors | 旁路(见 H2) |
|
||||||
|
|
||||||
|
附带问题:`msvcrt.kbhit()` 检查的是**当前进程**键盘而非子管道(`command_tool.py:123-125` 无意义代码);循环内重复 `import select`。
|
||||||
|
|
||||||
|
### H2:超时机制对无输出命令完全失效(PoC 已证)
|
||||||
|
|
||||||
|
`readline()` 是阻塞调用,超时检查只在**两次 readline 之间**执行。无输出命令会让 readline 阻塞到进程自然退出:
|
||||||
|
|
||||||
|
```
|
||||||
|
PoC:timeout=3s,命令 powershell Start-Sleep 15(无 stdout 输出)
|
||||||
|
结果:readline blocked 15.2s (timeout was 3s)
|
||||||
|
VERDICT: timeout mechanism ineffective for silent commands: True
|
||||||
|
```
|
||||||
|
|
||||||
|
两处同病:`command_tool.py:106-165` 与 `executors.py:97-113`。`--once` 模式有 `--once-timeout` 兜底,**WebSocket 模式无兜底**。
|
||||||
|
|
||||||
|
### H3:`stdin=subprocess.PIPE` 从不写入也不关闭——死锁放大器
|
||||||
|
|
||||||
|
`executors.py:156`、`command_tool.py:93`:等待 stdin 的命令(`cat` 无参、交互式 `python`、`ssh`)永不退出 → 与 H2 叠加:**readline 永久阻塞 + 超时检测永不执行 + cancel 杀不到进程** = 工具线程永久挂起。应为 `stdin=subprocess.DEVNULL`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 三、中危发现(4 项)
|
||||||
|
|
||||||
|
| # | 问题 | 位置 | 说明 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| M1 | **heredoc 定界符注入** | `basic_tools.py:179` | `content` 含独立行 `AIBOX_EOF` 时 heredoc 提前终止,**后续行成为 shell 命令执行**。AI 写文件内容是不可信输入(可被网页内容/用户消息诱导)。修复:定界符唯一化(检测冲突则换 `AIBOX_EOF_2` 递增) |
|
||||||
|
| M2 | 工具流式转发循环可永久挂起 | `registry.py:90-107` | `while thread.is_alive() or not empty()` + `get(0.1)`——H3 场景下永不退出;`join(5)` 超时后返回但 daemon 线程继续向输出队列写(僵尸输出混入后续对话流) |
|
||||||
|
| M3 | web 工具无协议/目标校验 | `basic_tools.py:315-338` | `url` 未限制 scheme,`curl file:///etc/shadow`、内网地址均可达(SSRF)。本地单用户产品风险有限,建议白名单 http/https |
|
||||||
|
| M4 | shlex.quote POSIX 转义在 Windows 失效 | `basic_tools.py:99-112` 等全部指引命令 | 单引号在 cmd.exe 无效,Windows 下 `head/sed/grep` 指引命令执行异常。Linux 部署(树莓派)不受影响 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 四、低危发现(3 项)
|
||||||
|
|
||||||
|
| # | 问题 | 位置 |
|
||||||
|
|---|---|---|
|
||||||
|
| L1 | `get_pending_memos` 的 `LIMIT {limit}` f-string 拼接 | `db.py:417`——当前调用链 limit 均为内部 int,无外部直达;防御性参数化即可 |
|
||||||
|
| L2 | `check_path_safety` 仅查大小,无路径边界 | `base.py:96-110`——by design(AI 可全盘读写,本地单用户),已确认设计意图 |
|
||||||
|
| L3 | `executors.execute()` finally kill 后未 `wait()` | `executors.py:205-207`——Windows 句柄短暂滞留 |
|
||||||
|
|
||||||
|
**做得好的**:SQL 全面参数化(LIKE 通配符正确转义 `db.py:200-209`)、API key 零日志泄漏、WAL 已启用、消息回滚/tool 配对修复、`_msg_lock`/`_process_lock` 使用正确、工具实例按连接隔离。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 五、架构合规矩阵
|
||||||
|
|
||||||
|
| 硬约束 | 状态 |
|
||||||
|
|---|---|
|
||||||
|
| WebSocket :8765 | ✅ |
|
||||||
|
| 依赖单向 `common←subsystems←tools←aibox` | ✅ |
|
||||||
|
| Worker 仅 7 执行工具 | ✅ |
|
||||||
|
| 工具输出 ≤1MB + 超时 + 中断 | ❌ **H1/H2/H3 三项全破** |
|
||||||
|
| prompt 三级排序加载 / sqlite 线程安全 | ✅ |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 六、修复建议(一批搞定 H1/H2/H3,约 ±120 行)
|
||||||
|
|
||||||
|
1. **删除 command_tool.py 的重复 Popen 实现**,`execute_streaming` 改为一行委托 `self.executor.execute_streaming(output_queue, ...)` → 1MB、cancel、统一语义全部自动恢复
|
||||||
|
2. **executors.py 读取循环改线程化**:后台 reader 线程做阻塞 `readline` 入队列,主循环带超时轮询队列——跨平台解决 H2
|
||||||
|
3. `stdin=subprocess.DEVNULL`(H3)
|
||||||
|
4. M1 定界符唯一化 + M2 转发循环加总超时,顺手可修
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 七、修复落地(Batch E,2026-08-17)
|
||||||
|
|
||||||
|
五项修复(H1/H2/H3/M1/M2)已全部实施并通过验证:
|
||||||
|
|
||||||
|
| # | 修复 | 位置 | 状态 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| H1 | `CommandLineTool.execute_streaming` 改为纯委托 `executor.execute_streaming`,删除 150 行重复 Popen(-165 行) | `tools/command_tool.py` | ✅ |
|
||||||
|
| H2 | `_read_with_limit` 线程化:后台 reader 线程阻塞 `readline` 入队,主循环 0.1s 轮询 + 超时/1MB 检查;shell 已死但 reader 无 EOF 时 1s 宽限排空(Windows shell=True 孙进程持管道场景) | `subsystems/executors.py` | ✅ |
|
||||||
|
| H3 | 两处 Popen 均改 `stdin=subprocess.DEVNULL` | `subsystems/executors.py` | ✅ |
|
||||||
|
| M1 | heredoc 定界符唯一化:content 含独立行与 `AIBOX_EOF` 相同时递增 `_X` 后缀直到唯一 | `tools/basic_tools.py` | ✅ |
|
||||||
|
| M2 | registry 转发循环加 3600s 总超时兜底,超时后放弃等待并返回错误信息 | `tools/registry.py` | ✅ |
|
||||||
|
|
||||||
|
### PoC 回归验证(`aibox/tests/_poc_audit_regression.py`,5/5 PASS)
|
||||||
|
|
||||||
|
```
|
||||||
|
[PASS] PoC-1 无输出命令超时(H2): 3.0s 返回(修复前阻塞15s+),结果含超时: True
|
||||||
|
[PASS] PoC-2 stdin等待命令不挂起(H3): 0.2s 返回(修复前永久挂起)
|
||||||
|
[PASS] PoC-3 1MB输出截断: 0.2s,结果 1.00MB,含截断标记: True
|
||||||
|
[PASS] PoC-4 cancel中断: cancel 后 3.0s 结束(修复前杀不到进程需等满120s)
|
||||||
|
[PASS] PoC-5 工具层1MB贯通(H1): 0.2s,工具层结果 1.00MB(修复前无限累积)
|
||||||
|
```
|
||||||
|
|
||||||
|
PoC-2 说明:`stdin=DEVNULL` 下等待输入的命令立即收到 EOF 快速退出(0.2s),优于按超时返回;判定标准为"不永久挂起"。
|
||||||
|
|
||||||
|
### 单元测试
|
||||||
|
|
||||||
|
aibox 全量单测 **264 passed**(零回归)。
|
||||||
|
|
||||||
|
### 遗留项(本轮未处理)
|
||||||
|
|
||||||
|
- M3(web 工具 scheme/目标校验)、M4(Windows 下 shlex.quote 转义失效)、L1/L3(防御性加固)——低优先级,待后续批次
|
||||||
|
- 架构合规矩阵"工具输出 ≤1MB + 超时 + 中断"一项修复后为 ✅
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 八、第二轮复审(2026-08-18,HEAD `177a9ac`)
|
||||||
|
|
||||||
|
方法:Batch E 修复质量复审 + 全量重审(3 路并行审读 + 主审对 6 处关键发现逐一实证抽查)。
|
||||||
|
|
||||||
|
### 8.1 Batch E 修复复审结论
|
||||||
|
|
||||||
|
| # | 修复 | 复审结论 |
|
||||||
|
|---|---|---|
|
||||||
|
| H1 | command_tool 纯委托 executor | ✅ 到位(PoC-5 验证 1MB 贯通) |
|
||||||
|
| H2 | reader 线程 + 超时轮询 | ✅ 到位(PoC-1 验证),**但有两个残余**:N4 reader 线程泄漏、N5 子代理超时盲区 |
|
||||||
|
| H3 | stdin=DEVNULL | ✅ 到位(PoC-2 验证) |
|
||||||
|
| M1 | heredoc 定界符唯一化 | ✅ 到位 |
|
||||||
|
| M2 | registry 转发总超时 | ⚠️ **不完整**(见 N6) |
|
||||||
|
|
||||||
|
### 8.2 第一轮遗留项状态
|
||||||
|
|
||||||
|
| # | 问题 | 状态 |
|
||||||
|
|---|---|---|
|
||||||
|
| M3 | web 工具无 scheme 校验 | 未修复(实证:`basic_tools.py:319+` url 未限制,`file://` 等可达) |
|
||||||
|
| M4 | shlex.quote POSIX 转义 Windows 失效 | 未修复(实证:`sed -r`/`grep -oP` 指引,Linux 部署不受影响) |
|
||||||
|
| L1 | `LIMIT {limit}` f-string | 未修复,且**新发现同型第二处**(`db.py:417` 与 `db.py:459` `get_immediate_memos`) |
|
||||||
|
| L2 | check_path_safety 无路径边界 | 维持 by design(本地单用户,AI 可全盘读写为设计意图) |
|
||||||
|
| L3 | execute() finally kill 后未 wait() | 未修复(实证:`executors.py:236-241` 无 wait;`cancel()` 同样无 wait) |
|
||||||
|
|
||||||
|
### 8.3 新发现——高危(1 项)
|
||||||
|
|
||||||
|
#### N1:SOCKS5 代理密码明文持久化到配置文件
|
||||||
|
|
||||||
|
`common/config.py:238-251`(实证):`SetProxyTool.execute` 默认 `save=True` → `set_proxy` 将 `password` 明文写入 `_config["proxy"]["socks5"]` → `_save_config` 用 `json.dump` 全量落盘 `~/.aibox/config.json`。与 api_key 走环境变量的隔离设计形成反差,任何能读该文件的进程可获取代理凭据。
|
||||||
|
|
||||||
|
修复建议:密码不落盘(仅内存)或走系统凭据库;必须落盘时对称加密 + `os.chmod(0o600)`;`save` 默认改 `False`。
|
||||||
|
|
||||||
|
### 8.4 新发现——中危(9 项)
|
||||||
|
|
||||||
|
| # | 问题 | 位置 | 说明 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| N2 | drain_thread 每次工具调用泄漏一个永不退出的线程 | `aibox.py:1232-1243`(实证) | 循环条件只查 cancel 事件不查队列空,注释与实现不符;闭包持 output_callback(含 websocket)阻碍 GC;长期运行累积线程/内存 |
|
||||||
|
| N3 | 同步 threading.RLock 在 async 协程中阻塞事件循环 | `aibox.py:1855-1902, 2275, 2376` | `/new` 的 2s 轮询竞态窗口后仍可能 `with self._msg_lock:` 冻结整个事件循环 → 服务假死 |
|
||||||
|
| N4 | reader 线程泄漏(H2 残余) | `subsystems/executors.py:130-144` | 宽限 1s break 后 reader 仍阻塞 readline(Windows shell=True 孙进程持管道),stdout 未 close、reader 未 join,常驻服务累积 fd 与线程 |
|
||||||
|
| N5 | --once 超时对子代理完全无效(H2 盲区) | `aibox.py:3052-3058, 638-649` | Timer 到点只 `manager.cancel()` 父代理;子代理独立 `_cancel_event`/独立 executor/独立 HTTP client,cancel 不传播,进程挂起至子代理自行结束 |
|
||||||
|
| N6 | registry 超时路径跳过工具属性清理(M2 残余) | `tools/registry.py:99, 116`(实证 L114-120) | 两个超时 return 均跳过 `streaming_output=False`/`output_queue=None`;daemon 线程继续写实例属性 `tool.output_queue`,该属性被后续调用覆盖到新会话队列 → 僵尸输出串流 |
|
||||||
|
| N7 | FTS5 关键词注入 | `subsystems/db.py:179` | 关键词内 `"` 未转义(应翻倍),可破坏短语边界改变 AND/OR 语义;语法错误时回退 LIKE,影响有限 |
|
||||||
|
| N8 | 流式输出旁路 TOOL_MAX_RESULT_LENGTH | `tools/db_tools.py:97-98` + `registry.py:104-110` | 超长记忆 content 经流式通道完整推到前端,100000 字符上限仅约束非流式返回值 |
|
||||||
|
| N9 | prompt 注入存储 | `tools/prompt_tools.py:64, 127, 227` | Agent 可控内容原样写入 `00_IDENTITY.md`/`99_SOUL.md`/`100_STATUS.md` 并在后续会话作为系统提示词加载,形成持久化注入;update_soul 可自我强化 |
|
||||||
|
| N10 | mark_as_triggered TOCTOU 竞态 | `subsystems/db.py:509-570` | 跨连接先读后写,并发触发 trigger_count 失真;建议 BEGIN IMMEDIATE 同事务读写 |
|
||||||
|
|
||||||
|
### 8.5 新发现——低危(7 项)
|
||||||
|
|
||||||
|
| # | 问题 | 位置 | 说明 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| N11 | debug 日志打印工具敏感参数 | `tools/registry.py:66, 126` | debug=3 时 set_proxy 的 password 明文进 `~/.aibox/ai.log`(日志无 0600) |
|
||||||
|
| N12 | _load_persona 路径遍历(限 .md) | `aibox.py:527-543` | persona 参数 LLM 可控未净化,可读 SKILLS_DIR 外 .md 文件并注入子代理 |
|
||||||
|
| N13 | broadcast 裸 except + unregister KeyError 逃逸 | `aibox.py:2076-2082, 1971-1973` | 吞 CancelledError;并发注销时 `remove` 抛 KeyError;建议 `discard` + `except Exception` |
|
||||||
|
| N14 | _processing_tasks 按 ws id 覆盖,旧任务失联 | `aibox.py:2728-2742` | 连接关闭只能 cancel 最后一个任务;handle_command 不受 is_processing 保护 |
|
||||||
|
| N15 | llm_client 异常含完整响应体 | `llm_client.py:172-174` | 某些网关 4xx 页回显 Authorization 头 → 密钥进异常栈;建议截断 500 字符 |
|
||||||
|
| N16 | 历史索引裸 except + 归档文件名撞名覆盖 | `subsystems/conversation_history.py:60-66, 135-137` | Ctrl+C 被吞后空索引覆盖原文件;时间戳粒度不足同秒覆盖 |
|
||||||
|
| N17 | InstallSkillTool skill_path 无校验 | `tools/skill_tools.py:252-304` | 可引导复制任意敏感文件进技能目录,经 search_skills show_content 预览泄露 |
|
||||||
|
|
||||||
|
杂项(低危,随批处理):`executors.py:209/301/338` 裸 except 应收窄为 TimeoutExpired;`db.py` 多处 `with sqlite3.connect` 不显式 close(CPython 下无害,PyPy 下句柄延迟);`config.py:31-34` AI_CONFIG_PATH 可重定向配置加载(本地单用户风险有限);skill_manager 技能目录代码执行面维持 by design(建议文档化可信边界)。
|
||||||
|
|
||||||
|
### 8.6 更新后的架构合规矩阵
|
||||||
|
|
||||||
|
| 硬约束 | 第一轮 | 第二轮 |
|
||||||
|
|---|---|---|
|
||||||
|
| WebSocket :8765 | ✅ | ✅ |
|
||||||
|
| 依赖单向 `common←subsystems←tools←aibox` | ✅ | ✅ |
|
||||||
|
| Worker 仅 7 执行工具 | ✅ | ✅ |
|
||||||
|
| 工具输出 ≤1MB + 超时 + 中断 | ❌ H1/H2/H3 全破 | ⚠️ **executor 链已恢复**;残余:流式 db_tools 旁路长度上限(N8)、子代理超时无效(N5)、reader 泄漏(N4) |
|
||||||
|
| prompt 三级排序加载 / sqlite 线程安全 | ✅ | ⚠️ sqlite 主流程安全;mark_as_triggered 竞态(N10)、FTS5 注入(N7)待修 |
|
||||||
|
|
||||||
|
### 8.7 Batch F 修复建议(按优先级)
|
||||||
|
|
||||||
|
1. **F1(高)**:N1 代理密码脱敏——`set_proxy` 落盘前抹除 password 字段 + `save` 默认 False
|
||||||
|
2. **F2(中)**:N6 registry 超时路径补清理(两个 return 前置 `tool.streaming_output=False; tool.output_queue=None`)+ N4 reader 线程收尾(break 后 `stdout.close()` + `reader.join(2)`)
|
||||||
|
3. **F3(中)**:N5 子代理 cancel 传播——`run_full_subagent` 传递父 `_cancel_event`,每轮 think 前检查
|
||||||
|
4. **F4(中)**:N2 drain_thread 加 sentinel 退出 + N3 reset_conversation 移出事件循环(run_in_executor)
|
||||||
|
5. **F5(中)**:N7 FTS5 转义 + N8 流式截断 + N10 同事务读写 + 顺带修 L1(两处 LIMIT 参数化)+ L3(finally wait)
|
||||||
|
6. **F6(低)**:N9 prompt 内容过滤/确认门槛 + N11 日志脱敏 + N12 persona 路径校验 + N13/N14/N15/N16/N17 择机处理 + M3/M4 视部署平台决定
|
||||||
|
|
||||||
|
### 8.8 Batch F 修复进展
|
||||||
|
|
||||||
|
**F1 ✅ 已落地(2026-08-18)**——N1 + N11 一并修复:
|
||||||
|
|
||||||
|
- `common/config.py`:`set_proxy` 新增 `password_env` 参数;`_save_config` 落盘前 deepcopy 脱敏(明文密码强制置 None,纵深防御,任何保存路径均拦截);`get_proxy` 按"内存明文优先 → password_env 查环境变量"解析,下游(llm_client / web 工具 `_proxy_curl_args`)无感知;加载时检测旧版明文自动迁移(内存保留供本会话 + 立即重写脱敏文件,一次性清除存量)
|
||||||
|
- `tools/proxy_tools.py`:`SetProxyTool` 新增 `password_env` 参数,结果显示"密码来源"(环境变量 / 仅本次会话 / 无)
|
||||||
|
- `tools/registry.py`(N11):debug 日志对 `password/api_key/token/secret/authorization` 参数打码
|
||||||
|
- 测试:新增 `tests/test_config.py` 3 例(落盘无明文 / password_env 解析 / 旧配置迁移),全量 267 passed
|
||||||
|
- 行为变化:已有明文密码的配置重启后代理需重设密码或改用环境变量(迁移日志有提示)——清除存量明文的必然代价
|
||||||
|
|
||||||
|
**F2 ✅ 已落地(2026-08-18)**——N6 + N4 一并修复:
|
||||||
|
|
||||||
|
- N6(registry 超时路径补清理):两个超时 return(deadline 命中、join(5) 超时)统一改为 `timed_out` 标记 + 出口统一执行 `tool.streaming_output=False` / `tool.output_queue=None`;僵尸 daemon 线程的 `send_line` 落到 None 上不再外写,消除僵尸输出串流路径。`stream_tool_timeout` 提为实例属性(默认 3600s,测试可调)
|
||||||
|
- N4(reader 线程收尾,**含二次修正**):初版按审计建议"break 后 `stdout.close()` + `reader.join(2)`"实现,PoC 回归实证 **close 会阻塞**——CPython 流对象内部有锁,readline 持锁阻塞时 `close()` 一同挂起直到 readline 返回(PoC-1 从 3s 退化到 15.2s,恰为命令 Start-Sleep 时长)。修正后策略:reader 已正常退出(EOF)→ 显式 close 释放管道 fd;reader 仍阻塞 → 只做 `join(timeout=2)` 后放弃(daemon 兜底,孙进程退出收 EOF 后自然结束,fd 随管道消亡回收),主流程及时返回
|
||||||
|
- **审计建议修正记录**:8.3 节 N4 与 8.7 节 F2 中"close stdout 可中断阻塞 readline(CPython 行为)"的表述**不成立**,已按实测修正——跨线程中断阻塞中的管道 read 无便携方案,正确做法是控制泄漏窗口而非强行中断
|
||||||
|
- **测试**:新增 `tests/test_f2_fixes.py` 5 例(N6 超时/正常清理 ×2、N4 及时返回 ×2 + EOF 后 close ×1),PoC 回归 5/5(PoC-1/4 恢复 ~5s:超时 3s + join 宽限 2s),全量 272 passed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 九、第三轮审计(2026-08-18,HEAD `aa15841`)
|
||||||
|
|
||||||
|
方法:前两轮遗留项逐条实证复核 + v3.4 新增代码(多模型档案 / L4a/b/c 失败转移,`llm_client.py`)全量审读 + 测试基线确认(全量 **355 passed**)。
|
||||||
|
|
||||||
|
### 9.1 遗留项修复状态复核
|
||||||
|
|
||||||
|
| # | 问题 | 状态 | 实证 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| L3 | execute() kill 后未 wait() | ✅ **已修** | `executors.py:222-230`:kill/terminate → `wait(timeout=2)` → 超时再 kill 兜底 |
|
||||||
|
| N17 | InstallSkillTool 任意文件复制 | ✅ **已消解(设计变更)** | 工具重构为纯指引(只返回安装说明文本,不再执行 shell/复制);`CreateSkillTool` 增加 skill_name 正则校验(`^[A-Za-z0-9_-]+$`) |
|
||||||
|
| N1/N11 | 代理密码脱敏 / 日志打码 | ✅ 在位 | F1 修复无回归 |
|
||||||
|
| N4/N6 | reader 收尾 / registry 超时清理 | ✅ 在位 | F2 修复无回归 |
|
||||||
|
| N2 | drain 线程泄漏 | ❌ 未修 | `aibox.py:1280-1303`:循环条件仍只查 cancel 事件,注释"自动退出"与实现不符,每次工具调用泄漏一个常驻线程 |
|
||||||
|
| N3 | `/new` 阻塞事件循环 | ❌ 未修 | `aibox.py:2325`:2s 轮询竞态窗口后仍 `with self._msg_lock:`(同步 RLock acquire 于事件循环线程) |
|
||||||
|
| N5 | 子代理 cancel 不传播 | ❌ 未修 | `aibox.py:643-654`:子代理独立 `_cancel_event`/executor/client;`--once` Timer(3110-3113)只 cancel 父代理 |
|
||||||
|
| N7 | FTS5 引号未翻倍 | ❌ 未修 | `db.py:178`:`f'"{kw}"'` 直接拼,关键词含 `"` 破坏短语边界 |
|
||||||
|
| N8 | 流式旁路长度上限 | ❌ 未修 | `registry.py:120-129`:`("line", content)` 无限转发;`aibox.py:1326` 的 `MAX_RESULT_LENGTH` 只管回喂 LLM 的最终 result |
|
||||||
|
| N9 | prompt 注入存储 | ❌ 未修 | `prompt_tools.py`:confirm 默认 True,Agent 可控内容仍直写 00/99/100 文件 |
|
||||||
|
| N10 | mark_as_triggered TOCTOU | ❌ 未修 | `db.py:509-525`:跨连接先 `get_memo` 后 UPDATE,无 BEGIN IMMEDIATE |
|
||||||
|
| N12 | _load_persona 路径遍历 | ❌ 未修 | `aibox.py:529-545`:persona LLM 可控,`os.path.join(SKILLS_DIR, name + ".md")` 无边界校验 |
|
||||||
|
| N13 | broadcast 裸 except | ❌ 未修 | `aibox.py:2131`:`except:` 吞 CancelledError |
|
||||||
|
| N14 | _processing_tasks 覆盖 | ❌ 未修 | `aibox.py:2788`:按 ws id 覆盖登记;先完成任务的 done_callback 会误 pop 后续任务的登记 |
|
||||||
|
| N15 | 异常含完整响应体 | ⚠️ 部分改善 | 瞬时路径已截断 `[:200]`(163/213 行);**非瞬时路径仍全量**(`llm_client.py:215-216`) |
|
||||||
|
| N16 | 历史索引裸 except / 同秒覆盖 | ❌ 未修 | `conversation_history.py:64` 裸 `except:`;136 行归档文件名秒级时间戳 |
|
||||||
|
| M3/M4 | web scheme 校验 / shlex Windows | ❌ 未修 | 维持原判(Linux 部署不受影响) |
|
||||||
|
| L1 | LIMIT f-string | ❌ 未修 | `db.py:417, 459` 两处在位(内部 int 直达,防御性待修) |
|
||||||
|
|
||||||
|
### 9.2 v3.4 新代码审计(多模型档案 + L4 失败转移)
|
||||||
|
|
||||||
|
**设计评价:整体高质量**。免密端点与密钥隔离(`allow_key_file=False` 防跨厂商凭据泄漏)、用户中断绝不转移(cancel 广播全链 + `_cancelled` 门控)、流式仅首事件前重放(`produced_any` 门控)、fallback 链一层不递归 + 自引用/悬空引用硬性报错、构造失败跳过不阻断主档案——关键决策均正确且有测试锚定(`test_models_profile.py` 约 60 个 L4 用例)。
|
||||||
|
|
||||||
|
新发现(P 系列):
|
||||||
|
|
||||||
|
| # | 级别 | 问题 | 位置 | 说明 |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| P1 | 中 | 非瞬时 HTTP 错误响应体无截断(N15 残留) | `llm_client.py:215-216` | `RuntimeError(f"HTTP {code}: {response.text}")` 全量响应体进异常栈;部分网关 4xx 页回显 Authorization 头 → 密钥进日志。与瞬时路径 `[:200]` 不一致,建议同样截断 |
|
||||||
|
| P2 | 中 | Retry-After 无上限 | `llm_client.py:290` | `max(1.0, float(v))` 无上界——异常/恶意服务端 `Retry-After: 999999` 使档案冷却 11.5 天。建议 cap(如 `min(v, max(cooldown_seconds, 3600))`) |
|
||||||
|
| P3 | 低中 | fallback 档案缺 model 不校验 | `llm_client.py:671-673` | 主档案经 `_require_model` 硬性报错,fallback 档案直接 `fb_profile.get("model")` → model=None → API 400 → 表现为神秘 None 而非清晰配置错误。建议构造链时同样校验 |
|
||||||
|
| P4 | 低 | FailoverClient 非线程安全 | `llm_client.py:321-334` | `_cancelled`/`_call_started`/`_stats` 无锁。当前调用面无实际并发(compressor 在 `_msg_lock` 内同步调用、子代理独立 client),但 cancel(WebSocket 线程)与 generator 启动时 `_cancelled = False` 复位存在理论竞态窗口。建议注释声明"非并发契约"或加锁 |
|
||||||
|
| P5 | 低 | `LLMClient._cancelled` 死状态 | `llm_client.py:52, 76` | 只写不读:cancel() 置位后无消费方也从不清除,真正判断靠 FailoverClient 自身标志。建议删除或在底层 client 消费 |
|
||||||
|
| P6 | 低 | fallback 列表重复项不查重 | `llm_client.py:660-673` | `["kimi","kimi"]` 会在链内构造两个同名 client,冷却键同名冲突、重复撞同端点;影响有限 |
|
||||||
|
| P7 | 信息 | timeout_budget 仅在尝试间检查 | `llm_client.py:363-366, 404, 443` | 预算不截断单档案进行中的阻塞读(单次仍可超时到 timeout)。与文档"限制整条链尝试的总秒数"有细微出入,可接受,建议文档注明"尝试间预算" |
|
||||||
|
|
||||||
|
### 9.3 架构合规矩阵(第三轮)
|
||||||
|
|
||||||
|
| 硬约束 | 状态 |
|
||||||
|
|---|---|
|
||||||
|
| WebSocket :8765 | ✅ |
|
||||||
|
| 依赖单向 `common←subsystems←tools←aibox` | ✅ |
|
||||||
|
| Worker 仅 7 执行工具 | ✅ |
|
||||||
|
| 工具输出 ≤1MB + 超时 + 中断 | ⚠️ executor 链 ✅;残余:N8 流式旁路、N5 子代理超时、N2 drain 泄漏 |
|
||||||
|
| prompt 三级排序 / sqlite 线程安全 | ⚠️ N7/N10 待修 |
|
||||||
|
| 多模型档案凭据隔离(v3.4 新增维度) | ✅ 跨厂商 Key 不互发、免密端点显式声明 |
|
||||||
|
|
||||||
|
### 9.4 Batch G 修复建议(按优先级)
|
||||||
|
|
||||||
|
1. **G1(中)**:P1 非瞬时路径响应体截断 `[:200]` + P2 Retry-After 上限 cap——两处各一行改动
|
||||||
|
2. **G2(中)**:P3 fallback 档案补 `_require_model` 校验(缺 model 的 fallback 跳过并告警,与缺 Key 行为对齐)
|
||||||
|
3. **G3(中)**:N5 子代理 cancel 传播(`run_full_subagent` 传递父 `_cancel_event`,每轮 think 前检查)+ N2 drain 线程加 sentinel/空队列退出
|
||||||
|
4. **G4(中低)**:N3 `/new` 的 `reset_conversation` 移入 `run_in_executor` + N8 registry 转发行截断 + N12 persona 路径边界校验(`os.path.realpath` 前缀检查)
|
||||||
|
5. **G5(低)**:N7 FTS 引号翻倍 + N10 BEGIN IMMEDIATE + L1 LIMIT 参数化 + N13 `except Exception`/`discard` + N14 任务登记改按 task id 集合 + N16 收窄 except + 归档文件名加毫秒/序号
|
||||||
|
6. **G6(低/择机)**:P4 并发契约注释或加锁 + P5 死状态清理 + P6 fallback 去重 + N9 prompt 更新确认门槛 + M3/M4 视部署平台决定
|
||||||
|
|
||||||
|
### 9.5 测试基线
|
||||||
|
|
||||||
|
全量 **355 passed**(21.8s,Windows 开发机 `py -3`),含 L4 失败转移约 60 用例、Batch E PoC 回归 5/5、Batch F1/F2 修复测试 8 例。零回归。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 十、Batch G 修复落地(2026-08-18,第三轮审计修复)
|
||||||
|
|
||||||
|
按 §9.4 建议分 5 波实施,每波全量测试验证后进入下一波。
|
||||||
|
|
||||||
|
### 10.1 修复清单
|
||||||
|
|
||||||
|
| 波次 | # | 修复内容 | 文件/位置 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| W1 | P1 | 非瞬时 HTTP 错误响应体截断 `[:200]`(与瞬时路径对齐,密钥/错误页不进异常栈) | `llm_client.py:216` |
|
||||||
|
| W1 | P2 | Retry-After 解析上限 `min(max(1.0, v), 3600.0)`——恶意/异常头最多冷却 1h | `llm_client.py:284-296` |
|
||||||
|
| W1 | P3 | fallback 档案构造前补 `_validate_profile` + `_require_model`,配置缺陷跳过并告警(对齐缺 Key 行为) | `llm_client.py:681-695` |
|
||||||
|
| W1 | P5 | 删除 `LLMClient._cancelled` 死状态(只写不读;真实机制在 FailoverClient),cancel() 注释说明职责边界 | `llm_client.py:48-49, 71-76` |
|
||||||
|
| W1 | P6 | fallback 列表 `seen` 去重(自引用硬性报错保持不变,测试锚定) | `llm_client.py:669-679` |
|
||||||
|
| W1 | P4 | FailoverClient 类文档补并发契约声明(非线程安全 + 当前调用面为何安全) | `llm_client.py:324-327` |
|
||||||
|
| W2 | N8 | registry 流式转发受 `TOOL_MAX_RESULT_LENGTH` 约束:累计预算 + 超限截断 + 停止转发提示(一次性) | `tools/registry.py:113-145` |
|
||||||
|
| W2 | N12 | persona 路径边界校验:normcase+abspath 前缀检查,越界/绝对路径拒绝 | `aibox.py:536-544` |
|
||||||
|
| W3 | N2 | drain 线程重构:批次级单线程 + 独立 stop 事件 + try/finally 统一回收(正常/异常/中断路径均 join≤2s);`set_output_queue(None)` 移入 finally | `aibox.py:1254-1386` |
|
||||||
|
| W3 | N5 | 子代理取消传播:`run_full_subagent` 新增 `cancel_event` 参数→子代理共享父事件(think 前置/工具间/流式事件间既有检查点联动)+ 0.2s 观察者线程转发深度中断(强关在途 HTTP、kill 子进程),finally 通知退出;`SubAgentTool` 构造注入父 `_cancel_event` | `aibox.py:621-624, 669-684, 773-775, 786-790, 897-902, 1145-1147` |
|
||||||
|
| W4 | N3 | `/new` 的 `reset_conversation` + `prompt_manager.reload()` 移入 `run_in_executor`(不再同步获取 `_msg_lock` 冻结事件循环) | `aibox.py:2386-2400` |
|
||||||
|
| W4 | N13 | broadcast 裸 except → `except Exception`(不再吞 CancelledError);unregister `remove` → `discard`(幂等) | `aibox.py:2080-2084, 2185-2192` |
|
||||||
|
| W4 | N14 | `_processing_tasks` 改按 websocket 键的任务集合登记:done_callback 从集合 discard;连接关闭取消该连接全部在途任务(此前仅最后一个) | `aibox.py:2022, 2847-2870` |
|
||||||
|
| W5 | N7 | FTS5 关键词内双引号翻倍转义(`"` → `""`),MATCH 短语语法合法 | `subsystems/db.py:178-182` |
|
||||||
|
| W5 | L1 | 两处 `LIMIT {limit}` f-string → `LIMIT ?` 参数化(`get_pending_memos`/`get_immediate_memos`) | `subsystems/db.py:419-421, 462-465` |
|
||||||
|
| W5 | N10 | `mark_as_triggered` 读写收进单连接 `BEGIN IMMEDIATE` 事务(消灭 check-then-act 竞态:并发触发不再双计数/双推进 repeat) | `subsystems/db.py:515-590` |
|
||||||
|
| W5 | N16 | 历史索引加载裸 except → `except Exception` + 告警;归档文件名冲突时加 `_N` 序号(同秒归档不覆盖) | `subsystems/conversation_history.py:58-69, 138-146` |
|
||||||
|
|
||||||
|
未实施(按 §9.4 G6 择机项):N9(prompt 更新确认门槛,涉及交互设计)、M3/M4(web scheme 校验 / shlex Windows,Linux 部署不受影响)、P7(timeout_budget 语义为文档注记,已在 FailoverClient 文档说明)。
|
||||||
|
|
||||||
|
### 10.2 新增测试
|
||||||
|
|
||||||
|
`tests/test_g_fixes.py` **20 例**(全部通过):
|
||||||
|
- P1/P2 ×4:非瞬时错误体截断、Retry-After cap/正常值/非法值
|
||||||
|
- P3/P5/P6 ×2:fallback 缺 model 跳过 + 重复去重(链 = [main, dupe, ok2])、死状态删除
|
||||||
|
- N2 ×2:drain 线程批次结束/工具异常路径均回收
|
||||||
|
- N5 ×3:子代理共享父取消事件 / 未传时保留自身 / SubAgentTool 注册注入链路
|
||||||
|
- N7 ×2:含引号关键词不破坏检索、常规检索不回归
|
||||||
|
- N8 ×1:120000 字符流式输出被截断至预算 + 提示标记
|
||||||
|
- N10 ×2:缺失 id 事务路径返回 None、一次性任务完成语义
|
||||||
|
- N12 ×3:`../` 越界/绝对路径拒绝、合法 persona 正常加载
|
||||||
|
- N16 ×1:同秒两次归档文件名不冲突
|
||||||
|
|
||||||
|
N3/N13/N14 为 async WebSocket 路径,依赖事件循环与真实连接对象,由现有 WebSocket 测试面回归(全量通过)。
|
||||||
|
|
||||||
|
### 10.3 测试基线
|
||||||
|
|
||||||
|
全量 **375 passed**(22.5s,Windows 开发机 `py -3`)= 原 355 + Batch G 新增 20。零回归。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 十一、第四轮审计(2026-08-18,HEAD `d1f780e`)
|
||||||
|
|
||||||
|
方法:3 路并行审读(aibox.py 全文 / llm_client+common+tools / subsystems+tests)+ 主审对全部候选发现逐条代码复核,关键项 Linux 动态实证(PoC 6/6)。
|
||||||
|
|
||||||
|
### 11.1 新发现与修复(Batch H,13 项全落地)
|
||||||
|
|
||||||
|
| # | 级别 | 问题 | 位置 | 修复 |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| H1 | **高危** | **N5 修复引入回归**:子代理共享父取消事件后,首轮 `_think_and_respond_impl` 无条件 `clear()` 吞掉父级已置位的取消信号(用户中断 / --once 超时恰落在子代理生成窗口时被清,观察者线程随之失效)——取消传播被绕过,子代理跑完全程 | `aibox.py:675, 1470` | 共享事件时置 `_shared_cancel_event` 跳过 clear;赋事件后立即检查 is_set 直接返回取消 |
|
||||||
|
| M1 | 中 | `/reload_prompts` 在事件循环线程同步获取 `_msg_lock` 冻结整个事件循环(N3 同类遗漏,只修了 `/new`) | `aibox.py:2492` | 移入 `run_in_executor`(与 /new 一致) |
|
||||||
|
| M2 | 中 | SkillManager 懒加载在事件循环线程串行执行 `--help` 子进程(每技能 ≤10s),首条消息/首个任务触发时冻结服务 | `aibox.py:2885-2891` | 启动前预热 WS 与调度器两个共享实例 |
|
||||||
|
| M3 | 中 | 调度器任务无超时:AI think 卡死 → `await ai_future` 永久阻塞 → 调度循环停摆、后续任务全部丢失(已 mark_as_triggered 不重试) | `aibox.py:333` | `call_later` 任务级超时(`scheduler.task_timeout`,默认 600s)→ `ai.cancel()`,循环继续 |
|
||||||
|
| M4 | 中 | `get_recent_memories(limit=None)` 无 LIMIT 全表加载(db_tools 分页传 None),记忆库增长后单次调用全量入内存 | `db.py:234` | None → 默认上限(对齐 search_memories 语义) |
|
||||||
|
| M5 | 中 | `memo delete` 实为 `complete_memo(force)`——只标记完成不删行,重复任务被推进而非移除,stats 计数失真 | `db_tools.py:747` | db 层新增 `delete_memo` 物理删除,工具改调 |
|
||||||
|
| M6 | 中 | `PromptManager.reload()` 先清空 `_prompts` 再逐文件重读:并发连接构建 system 消息时读到空/半截列表(N3 移线程池后放大) | `prompt_manager.py:298` | 本地构建新列表后原子替换 |
|
||||||
|
| L1 | 低 | read_file 的 `start_line/end_line` 未强转直拼 sed 单引号模板——LLM 传含引号字符串可闭合引号注入生成的指引命令 | `basic_tools.py:97` | int() 强转 + 范围夹取 |
|
||||||
|
| L2 | 低 | `_save_config` 吞异常仅 print:磁盘只读时 set_proxy 仍报"已保存",重启后配置丢失无感知 | `config.py:284` | `_save_config` 返回落盘结果,set_proxy 透传,工具如实报告 |
|
||||||
|
| L3 | 低 | SSE 行 `decode("utf-8")` 遇非 UTF-8 字节抛 UnicodeDecodeError,整条流中断且不触发 failover | `llm_client.py:222` | `errors="replace"` |
|
||||||
|
| L4 | 低 | persona 路径校验用 abspath 未解析符号链接:skills 目录内软链指向任意文件可绕过前缀检查被读取注入 | `aibox.py:536` | realpath 后再做前缀校验 |
|
||||||
|
| L5 | 低 | 1MB 截断按字节切后 `errors='replace'` 解码,行尾出现半个字符 � | `executors.py:150` | 改 `errors='ignore'` |
|
||||||
|
| L6 | 低 | compressor 压缩块全空内容时 `original_tokens=0` → 除零崩溃 | `conversation_compressor.py:517` | `max(original_tokens, 1)` |
|
||||||
|
| L7 | 低 | InstallSkillTool 指引命令 `ls -l {src}`/`cp {src} {dst}` 未引用:LLM 可控路径含空格/元字符时 shell 拆词/注入 | `skill_tools.py:269-285` | shlex.quote 源与目标路径 |
|
||||||
|
| L8 | 低 | ask_user 的 required 为空,LLM 不传 question 时 `str + None` 抛 TypeError | `misc_tools.py:42` | required 补 question + 缺参明确报错 |
|
||||||
|
| L9 | 低 | Linux 孙进程持管道泄漏(N4 残余变体):`(sleep 300 &)` 形态 shell 已死但孙进程持写端,reader 阻塞无 EOF;**实证修复前孙进程残留** | `executors.py` | POSIX `start_new_session` + 进程组信号(组 ID 在 Popen 后立即捕获,shell 死亡后 getpgid 不可用);宽限窗口到期强杀进程组 |
|
||||||
|
| L10 | 低 | web 工具 url 无 scheme 校验(M3 遗留):`file://` 等经 curl 指引可达(SSRF) | `basic_tools.py:319` | http/https 白名单 |
|
||||||
|
| L11 | 低 | registry 日志打码精确匹配:apiKey/passwd 等变体漏网 | `registry.py:19` | 子串包含匹配(含 apiKey/passwd/credential) |
|
||||||
|
| L12 | 低 | FailoverClient 异常消息经 `_log` 落 ai.log:网关错误页若在 200 字符内回显 Authorization 头则密钥进日志 | `llm_client.py` | `_log`/on_failover 前 Bearer 令牌正则脱敏 |
|
||||||
|
|
||||||
|
### 11.2 修复实证(Linux PoC 6/6 PASS)
|
||||||
|
|
||||||
|
```
|
||||||
|
[PASS] L-1 无输出命令超时: 3.0s 返回,含超时(原 PoC-1 为 Windows powershell 命令)
|
||||||
|
[PASS] L-3 1MB截断: 结果 1.00MB(Linux 等价命令)
|
||||||
|
[PASS] L-4 cancel中断: cancel 后 2.0s 结束
|
||||||
|
[PASS] L-5 工具层1MB贯通: 结果 1.00MB
|
||||||
|
[PASS] L-9 孙进程持管道不悬挂: 1.1s 返回,reader 正常收尾
|
||||||
|
[PASS] L-9 孙进程已被清理: 残留无(修复前 `(sleep 300 &)` 孙进程残留至自然退出)
|
||||||
|
```
|
||||||
|
|
||||||
|
L-9 实证过程说明:首版实现按"break 后 killpg"处理,但 shell 已死时 `os.getpgid(pid)` 抛 ProcessLookupError,组 ID 无法反查,孙进程仍残留(实证 FAIL 抓出)。修正为 **Popen 成功后立即捕获组 ID 并保存**,killpg 用保存值——6/6 通过。
|
||||||
|
|
||||||
|
### 11.3 遗留项(维持原判 / 择机)
|
||||||
|
|
||||||
|
- **N9**(prompt 注入存储):confirm 默认 True、100_STATUS 直写无门槛——涉及交互设计(用户确认门槛),维持 G6 择机项
|
||||||
|
- **M4/M3(旧编号)**:shlex Windows 转义(Linux 部署不受影响)、Q9 代理认证 curl 指引(修复需把密码嵌入命令串,反而引入密码进日志/会话史的新泄漏,维持现状)
|
||||||
|
- **F5** express_emotion 不可中断、**F6** /new 占用默认线程池、**F8** WS 限速/空闲回收、**F9** --once 软超时(H1 修复后取消可传播,残余窗口受 timeout 有界)、**Q7a** stream_chat 取消空窗(受 timeout 有界)、Q4 busy_timeout 重试、Q5 提醒格式异常静默、Q7b load_conversation 无调用面、Q8 对话文件原子写、compressor 裸 print / token 双计、单例 __new__ 无锁、db.py `if limit:` limit=0 边界——均为低危/信息级,下批处理
|
||||||
|
- **测试环境注记**:`test_executors.py::test_execute_timeout` 与 `test_models_profile.py::test_flat_api_still_reads_key_file` 为 Windows 向用例(ping -n 语法 / 依赖 ~/.cert/deepseek.key),Linux 环境预存失败,与本轮改动无关(git stash 实证)
|
||||||
|
|
||||||
|
### 11.4 测试基线(Linux,`aibox/venv`)
|
||||||
|
|
||||||
|
全量 **408 passed** = 原 382(Linux 环境)+ Batch H 新增 26。零回归;2 个预存环境性失败同前。PoC 回归脚本为 Windows 命令集,Linux 用等价命令实证(见 11.2)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 十二、Batch I 修复落地(2026-08-18,遗留项收尾)
|
||||||
|
|
||||||
|
按 §11.3 遗留清单分 3 波实施:安全项 → 稳定性项 → 信息级顺手项。
|
||||||
|
|
||||||
|
### 12.1 修复清单
|
||||||
|
|
||||||
|
| # | 级别 | 修复内容 | 文件/位置 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| N9 | 中 | prompt 写入内容长度上限(持久化注入面约束):`PromptManager.MAX_CONTENT_LENGTH=10000`,update_identity/update_soul/update_status 超长拒绝(manager 兜底 + 工具友好提示);save_memory_and_end_conversation 状态内容超长拒绝 | `prompt_manager.py`、`prompt_tools.py` |
|
||||||
|
| Q7b | 低 | `load_conversation` realpath 前缀校验,`../` 越界/软链逃逸返回 None(当前无 LLM 调用面,防御性加固) | `conversation_history.py:348` |
|
||||||
|
| F5 | 低 | 情绪子代理取消传播:`run_light_subagent` 新增 `cancel_event`,改走 stream_chat(可被 cancel 强关在途 HTTP)+ 观察者线程;父已取消早退;`ExpressEmotionTool` 构造注入父 `_cancel_event` | `aibox.py:513, 608-660, 1177` |
|
||||||
|
| Q4 | 低 | `mark_as_triggered` BEGIN IMMEDIATE 前 `PRAGMA busy_timeout = 10000`——多进程同库(CLI+服务)写竞争等待锁释放而非 5s 后直接报 locked | `db.py:527` |
|
||||||
|
| Q5 | 低 | 提醒时间格式异常任务不再静默丢弃:去掉 SQL 层 `reminder_time <= ?` 预过滤(字符串比较会静默排除格式异常行),时间过滤改由 Python 二次验证权威判断;格式异常按"到期"返回并告警 | `db.py:405-411, 442-444` |
|
||||||
|
| Q8 | 低 | 对话文件原子写:`_atomic_write_json`(tmp + os.replace)覆盖历史索引/当前对话/归档三处写路径,崩溃不留半截 JSON | `conversation_history.py` |
|
||||||
|
| F6 | 低 | WS 命令专用线程池:`WebSocketHandler._cmd_executor`(max_workers=2),`/new`、`/reload_prompts` 不再占用默认池(繁忙 AI 时多连接并发 /new 不再耗尽线程池);服务退出时 shutdown | `aibox.py` |
|
||||||
|
| F9 | 低 | --once 硬超时兜底:软超时 cancel 后若 AI 卡死在不可中断路径(HTTP 空窗、忽略信号的子进程),宽限一个超时周期后 `os._exit(4)` 强杀,保证硬性结束语义 | `aibox.py:3283-3299` |
|
||||||
|
| — | 信息 | PromptManager 单例 `__new__` 加锁(并发首例化竞态防护) | `prompt_manager.py:49-53` |
|
||||||
|
| — | 信息 | `get_pending_memos`/`get_immediate_memos` 的 `if limit:` → `if limit is not None`(limit=0 语义修正,None 才不加 LIMIT) | `db.py:420, 464` |
|
||||||
|
|
||||||
|
### 12.2 维持原判(未改动)
|
||||||
|
|
||||||
|
- **Q7a**(stream_chat 取消空窗):POST 返回前 `_current_response` 未赋值,cancel 无法中断——受 timeout 有界;根治需线程化 POST + 轮询取消,复杂度高,维持记录
|
||||||
|
- **M4(旧编号)**(shlex Windows 转义):Linux 部署不受影响
|
||||||
|
- **Q9(代理认证 curl 指引)**:修复会把密码嵌入命令串(进日志/会话史),反而引入新泄漏,维持现状
|
||||||
|
- **F8**(WS 限速/空闲回收):本地单用户产品设计项
|
||||||
|
- compressor 裸 print(终端过程展示,保留)、`estimate_tokens` 英文双计(测试锚定既有算法,仅影响压缩阈值)
|
||||||
|
|
||||||
|
### 12.3 新增测试(`tests/test_i_fixes.py` 16 例,全部通过)
|
||||||
|
|
||||||
|
N9 ×5(manager 兜底/身份/灵魂/状态超长拒绝 + 正常写入)、Q7b ×2(越界拒绝 + 正常加载)、F5 ×3(父已取消早退/流式中断/注册注入链路)、Q5 ×2(格式异常按到期返回 + 正常过滤不回归)、Q8 ×1(无 .tmp 残留 + JSON 合法)、F6 ×1(专用线程池存在与释放)、limit=0 ×2。
|
||||||
|
|
||||||
|
### 12.4 测试基线(Linux,`aibox/venv`)
|
||||||
|
|
||||||
|
全量 **424 passed** = 原 408 + Batch I 新增 16。零回归;2 个预存环境性失败同前。
|
||||||
@@ -0,0 +1,358 @@
|
|||||||
|
# aibox CLI 单次调用模式 · 开发文档
|
||||||
|
|
||||||
|
> 文档类型:功能开发计划
|
||||||
|
> 模块:aibox(单 AI 智能体系统)
|
||||||
|
> 日期:2026-08-17
|
||||||
|
> 状态:方案研究(待评审)
|
||||||
|
> 影响范围:`aibox/aibox.py`(仅增量改动,WebSocket 模式零回归)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 概述
|
||||||
|
|
||||||
|
### 1.1 背景与动机
|
||||||
|
|
||||||
|
aibox 目前只有两种使用方式:
|
||||||
|
|
||||||
|
| 使用方式 | 入口 | 特点 |
|
||||||
|
|----------|------|------|
|
||||||
|
| WebSocket 服务器 | `python aibox.py` → `ws://localhost:8765` | 长连接、多轮对话、前端/BotOps 交互(核心场景) |
|
||||||
|
| 备忘录 CLI | `--check-memo` / `--memo-add` / `--memo-list` / `--memo-complete` | 只做任务 CRUD,不触发 AI 对话 |
|
||||||
|
|
||||||
|
缺少一种**"问一次就退"**的 CLI 一次性对话模式。这导致以下场景无法覆盖:
|
||||||
|
|
||||||
|
1. **脚本化调用**:无法在 shell 脚本 / cron / CI 流水线中把 AI 当命令行工具用(如 `aibox --once "统计当前目录 py 文件行数" | grep ...`)
|
||||||
|
2. **资源受限环境**:树莓派 / NPU 盒子上不想常驻一个 WebSocket 服务进程,只想按需拉起、用完即退
|
||||||
|
3. **调试与测试**:复现某个 prompt 行为必须启动服务器 + 前端连接,链路太重
|
||||||
|
4. **进程级隔离执行**:Worker 子代理当前是**同进程线程**(`run_full_subagent` → `DeepSeekChat`),子代理崩溃/死循环会影响主服务;有了 CLI 单次模式后,可演进为"独立子进程执行",OS 级隔离
|
||||||
|
|
||||||
|
### 1.2 设计目标
|
||||||
|
|
||||||
|
1. **新增 `--once QUERY` 参数**:执行一次性 AI 对话(含完整工具调用链),结果流式输出到 stdout,完成后进程退出
|
||||||
|
2. **纯增量、零回归**:不改动 WebSocket 模式任何现有行为;不加参数时行为与现状完全一致
|
||||||
|
3. **复用现有架构**:复用 `EnhancedAIManager` / `DeepSeekChat` / Orchestrator-Worker 工具分层 / 三层中断机制,不重写对话逻辑
|
||||||
|
4. **管道友好**:stdout 只出 AI 回答正文,过程行(🔧 工具调用等)走 stderr,退出码语义化,可被 shell 脚本安全组合
|
||||||
|
5. **状态隔离明确**:单次模式默认不加载历史对话、不写对话归档,记忆写入显式可控
|
||||||
|
|
||||||
|
### 1.3 非目标
|
||||||
|
|
||||||
|
- 不做多轮对话 CLI(交互式 REPL 已由 `client.py` 覆盖)
|
||||||
|
- 不做 HTTP/REST 单次接口(如需可在后续批次评估,复用同一内核)
|
||||||
|
- 不改变 Worker 子代理的线程内执行方式(进程化是后续可选演进,不在本期范围)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 现状分析
|
||||||
|
|
||||||
|
### 2.1 入口结构(aibox.py)
|
||||||
|
|
||||||
|
```
|
||||||
|
parse_args() # L2786: argparse 定义
|
||||||
|
main() # L2927: 主入口
|
||||||
|
├── 语言/配置/调试覆盖 # L2931-2959
|
||||||
|
├── 目录创建 + 日志初始化 # L2961-2973
|
||||||
|
├── SubprocessCommandExecutor # L2976
|
||||||
|
├── EnhancedAIManager # L2978-2981
|
||||||
|
├── handle_check_commands() # L2983: --check-memo 处理后 return
|
||||||
|
├── handle_memo_commands() # L2986: --memo-* 处理后 return
|
||||||
|
└── manager.run() # L2990: WebSocket 服务器(唯一终点)
|
||||||
|
```
|
||||||
|
|
||||||
|
关键观察:
|
||||||
|
|
||||||
|
- `handle_check_commands` / `handle_memo_commands` 已经是**"处理后即退出"的 CLI 模式先例**,单次对话模式可完全沿用该模式:在 `manager.run()` 之前插入 `handle_once_mode(manager)`,命中则执行并 return
|
||||||
|
- `EnhancedAIManager.add_ai()`(L2584)已封装完整的 AI 实例组装(memory_db / memo_db / history_manager / prompt_manager / command_executor 注入),单次模式可直接复用
|
||||||
|
- `DeepSeekChat.think_and_respond(input_text, output_callback)`(L1308)是同步阻塞函数,天然适合 CLI 直接调用——不需要 asyncio 事件循环
|
||||||
|
|
||||||
|
### 2.2 output_callback 消息类型(流式输出契约)
|
||||||
|
|
||||||
|
`think_and_respond` 通过 `output_callback(msg_type, content)` 实时推送:
|
||||||
|
|
||||||
|
| msg_type | 语义 | CLI 模式去向 |
|
||||||
|
|----------|------|--------------|
|
||||||
|
| `chunk` | AI 回答增量片段(**权威内容来源**) | **stdout**(流式 write + flush) |
|
||||||
|
| `complete` | 收尾信号(内容与 chunk 累积重复) | 仅作收尾标记,不重复输出 |
|
||||||
|
| `reasoning_start/reasoning/reasoning_end` | 思考模式流式 | stderr(可用 `--show-reasoning` 控制) |
|
||||||
|
| `line` | 过程行:🔧 工具调用、工具输出、💬 前缀 | stderr |
|
||||||
|
| `ai_break` | 工具调用分段信号 | 忽略(CLI 无 widget 概念) |
|
||||||
|
| `error` | API/解析错误 | stderr + 退出码 2 |
|
||||||
|
| `dialogue_ended` | AI 主动结束对话 | 正常收尾 |
|
||||||
|
|
||||||
|
### 2.3 需要处理的技术要点
|
||||||
|
|
||||||
|
| 要点 | 现状 | 风险与对策 |
|
||||||
|
|------|------|-----------|
|
||||||
|
| **压缩输出污染** | `_think_and_respond_impl` L1346 中 `print("📦 对话压缩...")` 直接打 stdout | 单次模式默认不加载历史 → 消息极少 → 几乎不触发压缩;另需确认该 print 走 stderr 或抑制(Batch C 处理) |
|
||||||
|
| **记忆写入并发** | 多个 CLI 进程并发写 `memories.db`(SQLite) | 默认 `--once` 模式 AI 侧仍可调用 `memory` 工具(orchestrator profile);SQLite 有文件锁兜底,低频并发无碍;提供 `--no-save-memory` 开关时通过提示词引导(见 §3.4) |
|
||||||
|
| **Ctrl+C 中断** | 三层取消机制(Event + HTTP close + 子进程 kill)依赖 `cancel()` 被调用 | CLI 模式注册 `signal.SIGINT` handler 调 `manager.cancel()`,与 WebSocket `handle_interrupt` 路径等价 |
|
||||||
|
| **子代理输出转发** | `SubAgentTool._subagent_output` 经 `send_line` → `output_queue` → 回调链 | CLI 模式的 `output_callback` 会自然收到子代理的 `line`/`reasoning`,全部导流 stderr,无需特殊处理 |
|
||||||
|
| **对话归档写入** | `global_exit_handler` 会调 `history_manager.end_current_conversation()` | 单次模式传 `exit_callback=None`(与 TaskScheduler 的任务实例一致),AI 主动 `save_memory_and_end_conversation` 时不触发全局归档 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 设计方案
|
||||||
|
|
||||||
|
### 3.1 命令行接口
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 基本用法:一次性对话,结果输出到 stdout
|
||||||
|
python aibox.py --once "帮我统计当前目录下所有 py 文件的行数"
|
||||||
|
|
||||||
|
# 静默模式:只输出最终回答正文(工具过程完全静默,适合管道)
|
||||||
|
python aibox.py --once "把 README.md 翻译成英文摘要" --quiet
|
||||||
|
|
||||||
|
# 显式控制
|
||||||
|
python aibox.py --once "..." --no-save-memory # 不写入记忆数据库
|
||||||
|
python aibox.py --once "..." --timeout 120 # 全局超时 120 秒(默认 300)
|
||||||
|
python aibox.py --once "..." --show-reasoning # stderr 显示思考过程(默认不显示)
|
||||||
|
```
|
||||||
|
|
||||||
|
新增参数(全部可选,`--once` 之外均有默认值):
|
||||||
|
|
||||||
|
| 参数 | 类型 | 默认 | 说明 |
|
||||||
|
|------|------|------|------|
|
||||||
|
| `--once QUERY` | str | — | 一次性对话模式:执行 QUERY 后退出,不启动 WebSocket |
|
||||||
|
| `--quiet` | flag | false | 仅 stdout 输出回答正文;过程行/thinking 全静默 |
|
||||||
|
| `--no-save-memory` | flag | false | 禁止本次对话写入记忆数据库 |
|
||||||
|
| `--timeout SEC` | int | 300 | 全局执行超时,超时 kill 并退出码 4 |
|
||||||
|
| `--show-reasoning` | flag | false | 在 stderr 显示思考过程流 |
|
||||||
|
|
||||||
|
### 3.2 退出码语义
|
||||||
|
|
||||||
|
| 退出码 | 含义 |
|
||||||
|
|--------|------|
|
||||||
|
| 0 | 成功(AI 正常完成回答) |
|
||||||
|
| 1 | 工具执行链路异常(如 run_command 失败且 AI 报告失败) |
|
||||||
|
| 2 | API 请求失败(网络/HTTP/JSON 解析,服务端已回滚消息) |
|
||||||
|
| 3 | 用户中断(Ctrl+C) |
|
||||||
|
| 4 | 全局超时(--timeout 到期) |
|
||||||
|
|
||||||
|
> Batch A 只保证 0 / 2 / 3 / 4;退出码 1 的判定(从 AI 文本中识别失败)在 Batch B 实现,初版可统一归 0。
|
||||||
|
|
||||||
|
### 3.3 核心实现(Batch A,约 60~80 行增量)
|
||||||
|
|
||||||
|
**改动点 1:`parse_args()`(aibox.py L2786)** 追加 §3.1 的 5 个参数定义。
|
||||||
|
|
||||||
|
**改动点 2:新增 `handle_once_mode()`**,插入 `main()` 中 `handle_memo_commands()` 之后、`manager.run()` 之前:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def handle_once_mode(manager, args) -> bool:
|
||||||
|
"""一次性对话模式:命中 --once 则执行并返回 True(进程退出)。"""
|
||||||
|
if not args.once:
|
||||||
|
return False
|
||||||
|
|
||||||
|
ai = manager.add_ai()
|
||||||
|
ai.history_manager = None # 单次模式不加载/不写对话历史
|
||||||
|
if args.no_save_memory:
|
||||||
|
ai.memory_db = None # 工具注册时已注入;此处断引用防止运行期写入
|
||||||
|
# 注:Batch A 中 memory 工具已在 _register_tools 注册;
|
||||||
|
# 完整禁用见 §3.4 说明
|
||||||
|
|
||||||
|
exit_code = [0]
|
||||||
|
|
||||||
|
def cli_callback(msg_type, content):
|
||||||
|
if msg_type == "chunk":
|
||||||
|
sys.stdout.write(content)
|
||||||
|
sys.stdout.flush()
|
||||||
|
elif msg_type == "complete":
|
||||||
|
sys.stdout.write("\n")
|
||||||
|
sys.stdout.flush()
|
||||||
|
elif msg_type == "error":
|
||||||
|
print(content, file=sys.stderr)
|
||||||
|
exit_code[0] = 2
|
||||||
|
elif msg_type == "line" and not args.quiet:
|
||||||
|
sys.stderr.write(content)
|
||||||
|
sys.stderr.flush()
|
||||||
|
elif msg_type.startswith("reasoning") and args.show_reasoning:
|
||||||
|
sys.stderr.write(content)
|
||||||
|
sys.stderr.flush()
|
||||||
|
# ai_break / dialogue_ended 忽略
|
||||||
|
|
||||||
|
# Ctrl+C → 三层取消(与 WebSocket interrupt 路径等价)
|
||||||
|
def _sigint(sig, frame):
|
||||||
|
manager.cancel()
|
||||||
|
signal.signal(signal.SIGINT, _sigint)
|
||||||
|
|
||||||
|
# 全局超时:守护线程到点 cancel
|
||||||
|
timer = threading.Timer(args.timeout, manager.cancel)
|
||||||
|
timer.daemon = True
|
||||||
|
timer.start()
|
||||||
|
try:
|
||||||
|
response = ai.think_and_respond(args.once, cli_callback)
|
||||||
|
if response is None or (isinstance(response, str)
|
||||||
|
and response.startswith(f"\n{i18n.get('error_prefix')}")):
|
||||||
|
exit_code[0] = max(exit_code[0], 2)
|
||||||
|
# 被 cancel 打断时 think_and_respond 返回 interrupt 文案
|
||||||
|
if ai._is_cancelled() and timer.is_alive() is False:
|
||||||
|
exit_code[0] = 4
|
||||||
|
finally:
|
||||||
|
timer.cancel()
|
||||||
|
ai.cancel() # 兜底清理子进程
|
||||||
|
|
||||||
|
sys.exit(exit_code[0])
|
||||||
|
```
|
||||||
|
|
||||||
|
**改动点 3:`main()` 一行插入**:
|
||||||
|
|
||||||
|
```python
|
||||||
|
if handle_memo_commands(manager):
|
||||||
|
return
|
||||||
|
|
||||||
|
if handle_once_mode(manager, args): # 新增
|
||||||
|
return
|
||||||
|
|
||||||
|
manager.run()
|
||||||
|
```
|
||||||
|
|
||||||
|
> 注:`handle_once_mode` 内部 `sys.exit()`,return 分支实际不可达,保留 return 仅为与 `handle_check_commands` / `handle_memo_commands` 模式一致。
|
||||||
|
|
||||||
|
### 3.4 记忆写入的处理
|
||||||
|
|
||||||
|
`--no-save-memory` 的**完整实现**需要在工具注册层禁用 `MemoryTool`,但 `_register_tools` 依赖构造期注入的 `memory_db`。Batch A 采用轻量方案:
|
||||||
|
|
||||||
|
- **Batch A**:仅断 `ai.memory_db = None` 引用 + 文档声明"`--once` 场景不建议依赖记忆";AI 若调用 `memory` 工具会因注入的 db 为 None 由工具自身报错兜底(`MemoryTool.execute` 内部访问 db 返回错误字符串,不会崩溃)
|
||||||
|
- **Batch B**:`DeepSeekChat.__init__` 增加 `enable_memory: bool = True` 形参,为 False 时 `_register_tools` 跳过 `MemoryTool` 注册,工具 schema 不出现在请求中,LLM 无从调用——这才是干净方案
|
||||||
|
|
||||||
|
### 3.5 与既有架构约束的对齐
|
||||||
|
|
||||||
|
| 项目硬约束 | 本方案如何满足 |
|
||||||
|
|-----------|---------------|
|
||||||
|
| WebSocket localhost:8765 通信 | 不受影响;`--once` 不启动 WebSocket,互斥使用 |
|
||||||
|
| Worker 仅 7 种执行工具 | 不受影响;单次模式沿用 orchestrator profile,工具分层不变 |
|
||||||
|
| 工具输出 ≤1MB + 超时中断 | 不受影响;复用 `SubprocessCommandExecutor` |
|
||||||
|
| 单向依赖 common←subsystems←tools←aibox | 满足;改动全部在 aibox.py 聚合层 |
|
||||||
|
| 新增/变更模块三处同步(根README/docs/MODULES/ARCHITECTURE) | 本功能非新模块,但落地时需同步更新 `aibox/README.md` 的"命令行参数"章节 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 分批次实施计划
|
||||||
|
|
||||||
|
遵循项目"增量重构、低风险先行"惯例:
|
||||||
|
|
||||||
|
### Batch A:最小可用版(核心,建议先做)
|
||||||
|
|
||||||
|
- [ ] `parse_args()` 新增 `--once` / `--quiet` / `--timeout` 三个参数
|
||||||
|
- [ ] 新增 `handle_once_mode()`(§3.3 实现)
|
||||||
|
- [ ] `main()` 插入调用点
|
||||||
|
- [ ] SIGINT handler + 全局超时 Timer
|
||||||
|
- [ ] 退出码 0/2/3/4
|
||||||
|
- [ ] **验收**:`python aibox.py --once "现在几点了"` 正常输出并退出;`echo $?` 为 0;WebSocket 模式回归(启动服务器 + client.py 对话一轮)
|
||||||
|
|
||||||
|
### Batch B:体验与边界补齐
|
||||||
|
|
||||||
|
- [ ] `--no-save-memory` / `--show-reasoning` 参数落地
|
||||||
|
- [ ] `DeepSeekChat.__init__` 增加 `enable_memory` 开关,干净禁用 MemoryTool
|
||||||
|
- [ ] `_think_and_respond_impl` L1346 压缩提示 `print` 改走 stderr(或注入 print 回调)
|
||||||
|
- [ ] 退出码 1(工具失败识别)
|
||||||
|
- [ ] 补充 `tests/test_once_mode.py`:mock LLM 客户端,验证 stdout/stderr 分流、退出码、超时路径
|
||||||
|
- [ ] 更新 `aibox/README.md` 命令行参数章节
|
||||||
|
|
||||||
|
### Batch C:安全加固(可选,按需)
|
||||||
|
|
||||||
|
- [ ] SQLite WAL 模式(`PRAGMA journal_mode=WAL`),缓解多 CLI 进程并发写锁等待
|
||||||
|
- [ ] `--worker-only` 开关:`tool_profile=worker` 纯执行面模式,供其他模块以子进程方式调用 AI 执行器(为"Worker 进程级隔离"演进铺路)
|
||||||
|
- [ ] `--model` / `--temperature` 覆盖参数
|
||||||
|
|
||||||
|
### 明确不做(本期)
|
||||||
|
|
||||||
|
- 多轮交互式 CLI REPL(`client.py` 已覆盖)
|
||||||
|
- HTTP 单次接口
|
||||||
|
- Worker 子进程化(架构演进议题,单独立项)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 风险清单
|
||||||
|
|
||||||
|
| 风险 | 等级 | 缓解 |
|
||||||
|
|------|------|------|
|
||||||
|
| 冷启动开销(PromptManager 加载 + SQLite 初始化 + LLM 握手 ≈1~3s) | 低 | 可接受取舍;批量场景文档引导用 WebSocket 长连接 |
|
||||||
|
| CLI 模式与 WebSocket 模式代码 drift | 中 | 复用 `EnhancedAIManager.add_ai()`,单次模式新增代码控制在 80 行内,核心对话逻辑零复制 |
|
||||||
|
| `print` 污染 stdout 破坏管道 | 中 | Batch A 用 `--quiet` 兜底;Batch B 根治压缩提示 print |
|
||||||
|
| 并发进程 SQLite 写锁 | 低 | 低频无碍;Batch C WAL 加固 |
|
||||||
|
| Ctrl+C 时子进程泄漏 | 低 | 复用现成三层取消;`finally` 中 `ai.cancel()` 兜底 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 测试方案
|
||||||
|
|
||||||
|
### 6.1 单元测试(tests/test_once_mode.py,Batch B)
|
||||||
|
|
||||||
|
| 用例 | 验证点 |
|
||||||
|
|------|--------|
|
||||||
|
| 正常问答 | mock LLM 返回纯文本 → stdout 含回答、退出码 0 |
|
||||||
|
| 工具调用链 | mock LLM 返回 tool_calls → line 走 stderr、chunk 走 stdout |
|
||||||
|
| API 失败 | mock LLM 抛 RequestException → 退出码 2、stderr 含错误 |
|
||||||
|
| 超时 | mock LLM 阻塞 → Timer 触发 cancel → 退出码 4 |
|
||||||
|
| SIGINT | 发送 SIGINT → 三层取消触发 → 退出码 3 |
|
||||||
|
| `--quiet` | 过程行不输出到任何流 |
|
||||||
|
| `--no-save-memory` | memories.db 无新增行 |
|
||||||
|
|
||||||
|
### 6.2 回归测试
|
||||||
|
|
||||||
|
- [ ] `py -m pytest aibox/tests -q` 现有 232 项全过
|
||||||
|
- [ ] WebSocket 模式手测:启动服务器 + `client.py` 完成一轮对话 + Ctrl+C 中断
|
||||||
|
|
||||||
|
### 6.3 手动验收脚本
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 基础问答(无工具)
|
||||||
|
python aibox.py --once "用一句话介绍你自己" ; echo "exit=$?"
|
||||||
|
|
||||||
|
# 工具调用链(会派生 worker 子代理执行命令)
|
||||||
|
python aibox.py --once "在 ~/.aibox/workspace 下创建一个 hello.txt 写入 hello" ; echo "exit=$?"
|
||||||
|
|
||||||
|
# 管道组合
|
||||||
|
python aibox.py --once --quiet "列出当前目录 py 文件" | grep ".py" | wc -l
|
||||||
|
|
||||||
|
# 超时路径
|
||||||
|
python aibox.py --once --timeout 5 "运行 sleep 60" ; echo "exit=$?" # 期望 4
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. 实施状态(2026-08-17 落地记录)
|
||||||
|
|
||||||
|
> 本节为实施后回填,与第 3/4 节计划存在少量偏差,以本节为准。
|
||||||
|
|
||||||
|
### 7.1 提交与批次
|
||||||
|
|
||||||
|
| 批次 | 提交 | 内容 |
|
||||||
|
|---|---|---|
|
||||||
|
| Batch A | `ce3fb7f` | `--once` / `--quiet` / `--once-timeout`(300s 默认)+ `handle_once_mode` + SIGINT/超时复用三层取消 |
|
||||||
|
| Batch B | `9e0d39c` | `--no-save-memory`(`enable_memory` 构造开关干净禁用 MemoryTool)、`--show-reasoning`(思考流默认静默走 stderr)、压缩提示改走 output_callback、退出码 1(orchestrator 层工具异常)+ `test_once_mode.py` 10 项 |
|
||||||
|
| Batch C | `a5af4bb` | SQLite WAL、`--worker-only`(worker profile + 精简 system prompt 防人格污染)、`--model` / `--temperature`、子代理失败标记透传(`_SUBAGENT_TOOL_FAILED_MARKER` → orchestrator `stats.tool_failed`)+ 5 项测试 |
|
||||||
|
| Batch D | `d03dbc7` | 多模型档案 `models` 配置节(本档案为实施中新增需求,超出原计划范围)+ `test_models_profile.py` 17 项 + README 更新 |
|
||||||
|
|
||||||
|
### 7.2 与计划的偏差
|
||||||
|
|
||||||
|
1. **超时参数名**:计划 `--timeout`,落地为 `--once-timeout`(避免与 `api.timeout` 语义混淆)
|
||||||
|
2. **`models` 默认留空 `{}`**:计划未涉及;实施中发现默认档案硬编码 `DEEPSEEK_API_KEY` / 官方 URL 会覆盖用户自定义配置(本机为 `ARK_API_KEY` + 火山方舟端点,曾导致 401),故默认空 dict,未配置档案时零破坏回退旧 `api` 扁平节
|
||||||
|
3. **`--worker-only` 需要精简 system prompt**:原计划仅切 tool_profile;实测 orchestrator 的人格/身份提示词会污染 worker 语义(自称"老板"、主动写文件、多轮不收敛),已对齐 `run_full_subagent` 的子代理语义修复
|
||||||
|
4. **退出码 1 的边界**:仅覆盖 orchestrator 直接感知的工具异常;worker 子代理内的命令失败经 Batch C 的标记透传后同样计入。子命令失败但 AI 正常汇报时退出码为 0(AI 视角任务完成)
|
||||||
|
|
||||||
|
### 7.3 验证结果
|
||||||
|
|
||||||
|
- 单测 **264 项全过**(once 模式 15 + 多模型档案 17 新增)
|
||||||
|
- 冒烟:基础问答 exit 0 / 工具调用链(子代理执行 echo)exit 0 / 超时路径 exit 4 / `--no-save-memory --quiet` stdout 干净 / `--show-reasoning` 思考流走 stderr / `--temperature` 透传 / 不存在的模型名 exit 2
|
||||||
|
|
||||||
|
### 7.4 明确不做(维持原判)
|
||||||
|
|
||||||
|
WebSocket 会话恢复、`--once` 结果缓存、批处理多 query——见第 4 节"明确不做"。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 后续演进方向(不在本期)
|
||||||
|
|
||||||
|
1. **Worker 进程级隔离**:`run_full_subagent` 的线程内 `DeepSeekChat` → `subprocess.run(["python", "aibox.py", "--once", "--worker-only", ...])`,子代理崩溃不波及主服务,可用 OS 用户/chroot 做权限隔离
|
||||||
|
2. **HTTP 单次接口**:`POST /chat` 无状态接口,与 `--once` 共享内核
|
||||||
|
3. **结构化输出**:`--json` 输出 `{answer, tool_calls, exit_reason}` JSON,供程序消费
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 决策记录
|
||||||
|
|
||||||
|
| 决策点 | 结论 | 理由 |
|
||||||
|
|--------|------|------|
|
||||||
|
| 新入口文件还是复用 aibox.py | 复用 aibox.py + `--once` 参数 | 沿用 `--check-memo` 先例;避免维护两份入口 drift |
|
||||||
|
| stdout 输出协议 | chunk 流式 + 末尾换行 | 与 WebSocket 的 chunk 权威语义一致;管道可逐段消费 |
|
||||||
|
| 过程行去向 | stderr | Unix 管道惯例:stdout 是数据、stderr 是元信息 |
|
||||||
|
| 单次模式是否加载历史 | 否 | "一次性"语义明确;避免历史污染导致行为不可复现 |
|
||||||
|
| 是否默认写记忆 | 默认与 orchestrator 一致(可写),`--no-save-memory` 显式关闭 | 与 WebSocket 模式行为对齐,减少心智负担 |
|
||||||
@@ -416,9 +416,9 @@ function renderMarkdown(element, content) {
|
|||||||
html: false,
|
html: false,
|
||||||
highlight: function(code, lang) {
|
highlight: function(code, lang) {
|
||||||
if (lang && hljs.getLanguage(lang)) {
|
if (lang && hljs.getLanguage(lang)) {
|
||||||
return `<pre><code class="hljs language-${lang}">${hljs.highlight(code, {language: lang}).value}</code></pre>`
|
return `<pre><code class="hljs language-${lang}"><div style="font-size: 11px;">${lang}</div>${hljs.highlight(code, {language: lang}).value}</code></pre>`
|
||||||
} else {
|
} else {
|
||||||
return `<pre><code class="hljs">${md.utils.escapeHtml(code)}</code></pre>`
|
return `<pre><code class="hljs"><div style="font-size: 11px;">${lang}</div>\n${md.utils.escapeHtml(code)}</code></pre>`
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
#include <set>
|
#include <set>
|
||||||
#include <optional>
|
#include <optional>
|
||||||
#include <algorithm>
|
#include <algorithm>
|
||||||
|
#include <utility>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -72,7 +73,35 @@ public:
|
|||||||
|
|
||||||
explicit Memdb() = default;
|
explicit Memdb() = default;
|
||||||
explicit Memdb(int max_time) : MAX_TIME(std::max(0, max_time)) {}
|
explicit Memdb(int max_time) : MAX_TIME(std::max(0, max_time)) {}
|
||||||
Memdb& operator=(const Memdb& to_be_cp_memdb) = default;
|
|
||||||
|
// 深拷贝:重建索引,保证 keys/time_touch 的迭代器指向副本自身的 list
|
||||||
|
Memdb(const Memdb& other)
|
||||||
|
: __size(other.__size), MAX_TIME(other.MAX_TIME) {
|
||||||
|
this -> db = other.db;
|
||||||
|
for (auto it = this -> db.begin(); it != this -> db.end(); ++it) {
|
||||||
|
Memdb::index_keys new_key;
|
||||||
|
new_key.item = it;
|
||||||
|
this -> keys.emplace(it -> key, new_key);
|
||||||
|
|
||||||
|
Memdb::index_timestamp new_time_touch;
|
||||||
|
new_time_touch.item = it;
|
||||||
|
this -> time_touch.insert(new_time_touch);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Memdb& operator=(const Memdb& other) {
|
||||||
|
if (this == &other) {
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
Memdb tmp(other);
|
||||||
|
std::swap(this -> db, tmp.db);
|
||||||
|
std::swap(this -> keys, tmp.keys);
|
||||||
|
std::swap(this -> time_touch, tmp.time_touch);
|
||||||
|
std::swap(this -> __size, tmp.__size);
|
||||||
|
std::swap(this -> MAX_TIME, tmp.MAX_TIME);
|
||||||
|
return *this;
|
||||||
|
}
|
||||||
|
|
||||||
~Memdb() = default;
|
~Memdb() = default;
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -331,10 +331,11 @@ private:
|
|||||||
try {
|
try {
|
||||||
task.message = pmsg_recv(task.fd, _config.recv_timeout);
|
task.message = pmsg_recv(task.fd, _config.recv_timeout);
|
||||||
} catch (const MsgException&) {
|
} catch (const MsgException&) {
|
||||||
|
// 不在此清除 busy:fd 保持 busy 直到 _reap_failed 关闭连接。
|
||||||
|
// 旧实现提前清除 busy,EOF 可读性会让事件循环为同一 fd 重复
|
||||||
|
// 投递任务,迟到的失败标记经 fd 复用可能落在新连接上误杀之。
|
||||||
std::lock_guard<std::mutex> flk(_failed_mtx);
|
std::lock_guard<std::mutex> flk(_failed_mtx);
|
||||||
_failed.insert(task.fd);
|
_failed.insert(task.fd);
|
||||||
std::lock_guard<std::mutex> blk(_busy_mtx);
|
|
||||||
_busy.erase(task.fd);
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -354,11 +355,12 @@ private:
|
|||||||
failed = true;
|
failed = true;
|
||||||
}
|
}
|
||||||
if (failed) {
|
if (failed) {
|
||||||
|
// 失败连接交由 _reap_failed 统一清除 busy 并关闭,
|
||||||
|
// 不提前恢复 select 监听(同 recv 失败路径,防 fd 复用竞态)。
|
||||||
std::lock_guard<std::mutex> flk(_failed_mtx);
|
std::lock_guard<std::mutex> flk(_failed_mtx);
|
||||||
_failed.insert(task.fd);
|
_failed.insert(task.fd);
|
||||||
}
|
} else {
|
||||||
// 清除 busy 标记,恢复 select 监听
|
// 清除 busy 标记,恢复 select 监听
|
||||||
{
|
|
||||||
std::lock_guard<std::mutex> lock(_busy_mtx);
|
std::lock_guard<std::mutex> lock(_busy_mtx);
|
||||||
_busy.erase(task.fd);
|
_busy.erase(task.fd);
|
||||||
}
|
}
|
||||||
@@ -528,6 +530,7 @@ private:
|
|||||||
// 事件循环线程不阻塞接收,防 Slowloris DoS CWE-400)
|
// 事件循环线程不阻塞接收,防 Slowloris DoS CWE-400)
|
||||||
{
|
{
|
||||||
std::lock_guard<std::mutex> blk(_busy_mtx);
|
std::lock_guard<std::mutex> blk(_busy_mtx);
|
||||||
|
if (_busy.count(fd)) return; // 已有在途任务,防同一 fd 重复投递(fd 复用竞态)
|
||||||
_busy.insert(fd);
|
_busy.insert(fd);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
|
|||||||
+4
-1
@@ -69,4 +69,7 @@ Testing/
|
|||||||
.cache/
|
.cache/
|
||||||
|
|
||||||
# Databases
|
# Databases
|
||||||
*.db*
|
*.db*
|
||||||
|
# unit test binary
|
||||||
|
test_libmemdb
|
||||||
|
.memdb.pid
|
||||||
+7
-5
@@ -8,6 +8,9 @@ BUILD = build
|
|||||||
TARGET = $(BUILD)/memdb
|
TARGET = $(BUILD)/memdb
|
||||||
SRC = src/memdb.cpp
|
SRC = src/memdb.cpp
|
||||||
|
|
||||||
|
HOST ?= 127.0.0.1
|
||||||
|
PORT ?= 8011
|
||||||
|
|
||||||
all: $(TARGET)
|
all: $(TARGET)
|
||||||
|
|
||||||
$(BUILD):
|
$(BUILD):
|
||||||
@@ -37,13 +40,12 @@ test-libmemdb: $(TEST_LIBMEMDB)
|
|||||||
|
|
||||||
test-memdb: $(TARGET)
|
test-memdb: $(TARGET)
|
||||||
@echo "Starting memdb server..."
|
@echo "Starting memdb server..."
|
||||||
./$(TARGET) --host $(HOST) --port $(PORT) --workers 2 --max-time 10 &
|
@./$(TARGET) --host $(HOST) --port $(PORT) --workers 2 --max-time 10 >/dev/null 2>&1 & echo $$! > .memdb.pid
|
||||||
@sleep 1
|
@sleep 1
|
||||||
-python3 test_memdb.py $(HOST) $(PORT); EXIT=$$?
|
@python3 test_memdb.py $(HOST) $(PORT); EXIT=$$?; \
|
||||||
@kill %1 2>/dev/null; wait %1 2>/dev/null
|
kill $$(cat .memdb.pid) 2>/dev/null; rm -f .memdb.pid; exit $$EXIT
|
||||||
@exit $$EXIT
|
|
||||||
|
|
||||||
clean:
|
clean:
|
||||||
rm -f $(TARGET) $(TEST_LIBMEMDB)
|
rm -f $(TARGET) $(TEST_LIBMEMDB) .memdb.pid
|
||||||
|
|
||||||
.PHONY: all clean test test-libmemdb test-memdb
|
.PHONY: all clean test test-libmemdb test-memdb
|
||||||
+2
-1
@@ -21,7 +21,7 @@ Memdb<std::string, std::string> (libmemdb 模板:LRU 链表 + 哈希索
|
|||||||
| `update` | `key`, `value`, `refresh_ttl?` | `true/false` | 更新值,可选刷新 TTL |
|
| `update` | `key`, `value`, `refresh_ttl?` | `true/false` | 更新值,可选刷新 TTL |
|
||||||
| `exists` | `key` | `true/false` | 键是否存在 |
|
| `exists` | `key` | `true/false` | 键是否存在 |
|
||||||
| `size` | — | `int` | 当前条目数 |
|
| `size` | — | `int` | 当前条目数 |
|
||||||
| `expire` | `key`, `value` | `true/false` | 更新值并重置 TTL(del + set) |
|
| `expire` | `key`, `value` | `true/false` | 更新已有键的值并重置 TTL(键不存在返回 false) |
|
||||||
| `show` | — | `[{key,value},...]` | 导出所有条目 |
|
| `show` | — | `[{key,value},...]` | 导出所有条目 |
|
||||||
|
|
||||||
## 配置
|
## 配置
|
||||||
@@ -34,6 +34,7 @@ Memdb<std::string, std::string> (libmemdb 模板:LRU 链表 + 哈希索
|
|||||||
| `listen_port` | `8011` | 监听端口 |
|
| `listen_port` | `8011` | 监听端口 |
|
||||||
| `workers` | `4` | 线程池大小 |
|
| `workers` | `4` | 线程池大小 |
|
||||||
| `max_time` | `600` | 默认 TTL(秒),10 分钟 |
|
| `max_time` | `600` | 默认 TTL(秒),10 分钟 |
|
||||||
|
| `backlog` | `128` | listen() backlog(libmsg 默认 10,短连接突刺易丢连接) |
|
||||||
|
|
||||||
## 构建与运行
|
## 构建与运行
|
||||||
|
|
||||||
|
|||||||
+10
-7
@@ -20,6 +20,7 @@ struct ServerConfig {
|
|||||||
int listen_port = 8011;
|
int listen_port = 8011;
|
||||||
int workers = 4;
|
int workers = 4;
|
||||||
int max_time = 600; // 默认 TTL 10 分钟
|
int max_time = 600; // 默认 TTL 10 分钟
|
||||||
|
int backlog = 128; // 短连接并发突刺时防丢连接(libmsg 默认仅 10)
|
||||||
};
|
};
|
||||||
|
|
||||||
static std::atomic<bool> g_running{true};
|
static std::atomic<bool> g_running{true};
|
||||||
@@ -35,7 +36,7 @@ private:
|
|||||||
pmc::net::RpcServer __rpc;
|
pmc::net::RpcServer __rpc;
|
||||||
std::mutex __mtx;
|
std::mutex __mtx;
|
||||||
|
|
||||||
static const std::string& _require_string(const json::object& req, const std::string& field) {
|
static std::string _require_string(const json::object& req, const std::string& field) {
|
||||||
auto it = req.find(field);
|
auto it = req.find(field);
|
||||||
if (it == req.end()) {
|
if (it == req.end()) {
|
||||||
throw std::runtime_error("missing required field: " + field);
|
throw std::runtime_error("missing required field: " + field);
|
||||||
@@ -43,7 +44,8 @@ private:
|
|||||||
if (!it->value().is_string()) {
|
if (!it->value().is_string()) {
|
||||||
throw std::runtime_error("field '" + field + "' must be a string");
|
throw std::runtime_error("field '" + field + "' must be a string");
|
||||||
}
|
}
|
||||||
return it->value().as_string();
|
const auto& s = it->value().as_string();
|
||||||
|
return std::string(s.data(), s.size());
|
||||||
}
|
}
|
||||||
|
|
||||||
public:
|
public:
|
||||||
@@ -51,7 +53,7 @@ public:
|
|||||||
: __cfg(cfg)
|
: __cfg(cfg)
|
||||||
, __db(cfg.max_time)
|
, __db(cfg.max_time)
|
||||||
, __rpc(cfg.listen_host, cfg.listen_port,
|
, __rpc(cfg.listen_host, cfg.listen_port,
|
||||||
{.workers = cfg.workers})
|
{.workers = cfg.workers, .listen_backlog = cfg.backlog})
|
||||||
{
|
{
|
||||||
_register_routes();
|
_register_routes();
|
||||||
}
|
}
|
||||||
@@ -130,13 +132,12 @@ private:
|
|||||||
return result;
|
return result;
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---- expire: 更新值并重置 TTL(等价于 del + set)----
|
// ---- expire: 更新值并重置 TTL(仅已有键;等价于 update + refresh_ttl)----
|
||||||
__rpc.add_route("expire", [this](const json::object& req) -> json::value {
|
__rpc.add_route("expire", [this](const json::object& req) -> json::value {
|
||||||
const auto& key = _require_string(req, "key");
|
const auto& key = _require_string(req, "key");
|
||||||
const auto& val = _require_string(req, "value");
|
const auto& val = _require_string(req, "value");
|
||||||
std::lock_guard<std::mutex> lock(__mtx);
|
std::lock_guard<std::mutex> lock(__mtx);
|
||||||
__db.remove(key);
|
return __db.update(key, val, true);
|
||||||
return __db.insert(key, val);
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@@ -156,7 +157,8 @@ int main(int argc, char** argv) {
|
|||||||
("host", po::value<std::string>()->default_value("127.0.0.1"), "listen host")
|
("host", po::value<std::string>()->default_value("127.0.0.1"), "listen host")
|
||||||
("port", po::value<int>()->default_value(8011), "listen port")
|
("port", po::value<int>()->default_value(8011), "listen port")
|
||||||
("workers", po::value<int>()->default_value(4), "worker threads")
|
("workers", po::value<int>()->default_value(4), "worker threads")
|
||||||
("max-time", po::value<int>()->default_value(600), "default TTL (seconds)");
|
("max-time", po::value<int>()->default_value(600), "default TTL (seconds)")
|
||||||
|
("backlog", po::value<int>()->default_value(128), "listen() backlog");
|
||||||
|
|
||||||
po::variables_map vm;
|
po::variables_map vm;
|
||||||
po::store(po::parse_command_line(argc, argv, desc), vm);
|
po::store(po::parse_command_line(argc, argv, desc), vm);
|
||||||
@@ -171,6 +173,7 @@ int main(int argc, char** argv) {
|
|||||||
cfg.listen_port = vm["port"].as<int>();
|
cfg.listen_port = vm["port"].as<int>();
|
||||||
cfg.workers = vm["workers"].as<int>();
|
cfg.workers = vm["workers"].as<int>();
|
||||||
cfg.max_time = vm["max-time"].as<int>();
|
cfg.max_time = vm["max-time"].as<int>();
|
||||||
|
cfg.backlog = vm["backlog"].as<int>();
|
||||||
|
|
||||||
auto instance = vm["instance"].as<std::string>();
|
auto instance = vm["instance"].as<std::string>();
|
||||||
auto log_level = vm["log-level"].as<std::string>();
|
auto log_level = vm["log-level"].as<std::string>();
|
||||||
|
|||||||
+50
-9
@@ -10,6 +10,7 @@
|
|||||||
#include <cassert>
|
#include <cassert>
|
||||||
#include <chrono>
|
#include <chrono>
|
||||||
#include <iostream>
|
#include <iostream>
|
||||||
|
#include <optional>
|
||||||
#include <string>
|
#include <string>
|
||||||
#include <thread>
|
#include <thread>
|
||||||
#include <vector>
|
#include <vector>
|
||||||
@@ -266,19 +267,57 @@ static void test_get_time_touch() {
|
|||||||
PASS();
|
PASS();
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── copy assignment ─────────────────────────────────────────────────────
|
// ── copy semantics ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
static void test_copy_assignment() {
|
static void test_copy_assignment() {
|
||||||
TEST("copy assignment");
|
TEST("copy assignment");
|
||||||
Memdb<std::string, std::string> db1;
|
|
||||||
db1.insert("a", "1");
|
|
||||||
Memdb<std::string, std::string> db2;
|
Memdb<std::string, std::string> db2;
|
||||||
db2 = db1;
|
{
|
||||||
CHECK(db2.search("a").value() == "1", "copied db should have same data");
|
Memdb<std::string, std::string> db1;
|
||||||
// verify independence
|
db1.insert("a", "1");
|
||||||
db2.insert("b", "2");
|
db2 = db1;
|
||||||
CHECK(db1.size() == 1, "original should be unchanged");
|
CHECK(db2.search("a").value() == "1", "copied db should have same data");
|
||||||
CHECK(db2.size() == 2, "copy should be independent");
|
// verify independence
|
||||||
|
db2.insert("b", "2");
|
||||||
|
CHECK(db1.size() == 1, "original should be unchanged");
|
||||||
|
CHECK(db2.size() == 2, "copy should be independent");
|
||||||
|
}
|
||||||
|
// db1 destroyed — db2 must still work (indices must not dangle)
|
||||||
|
CHECK(db2.search("a").value() == "1", "copy must survive source destruction");
|
||||||
|
CHECK(db2.remove("a"), "copy remove should work after source destroyed");
|
||||||
|
CHECK(db2.size() == 1, "copy size after remove");
|
||||||
|
PASS();
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_copy_constructor() {
|
||||||
|
TEST("copy constructor");
|
||||||
|
std::optional<Memdb<std::string, std::string>> cpy;
|
||||||
|
{
|
||||||
|
Memdb<std::string, std::string> src;
|
||||||
|
src.insert("a", "1");
|
||||||
|
src.insert("b", "2");
|
||||||
|
cpy.emplace(src); // copy constructor
|
||||||
|
CHECK(cpy->size() == 2, "copy should have 2 entries");
|
||||||
|
CHECK(src.size() == 2, "source unchanged");
|
||||||
|
}
|
||||||
|
CHECK(cpy->search("a").value() == "1", "copy value a after source destroyed");
|
||||||
|
CHECK(cpy->search("b").value() == "2", "copy value b after source destroyed");
|
||||||
|
cpy->remove("a");
|
||||||
|
CHECK(cpy->size() == 1, "copy mutation after source destroyed");
|
||||||
|
// flush on copy must not touch freed memory
|
||||||
|
CHECK(cpy->search("b").value() == "2", "flush path safe on copy");
|
||||||
|
PASS();
|
||||||
|
}
|
||||||
|
|
||||||
|
static void test_update_refresh_extends_ttl() {
|
||||||
|
TEST("update with refresh extends TTL");
|
||||||
|
Memdb<std::string, std::string> db(1);
|
||||||
|
db.insert("a", "1");
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(700));
|
||||||
|
CHECK(db.update("a", "2", true), "update with refresh should succeed");
|
||||||
|
std::this_thread::sleep_for(std::chrono::milliseconds(600));
|
||||||
|
CHECK(db.search("a").value() == "2",
|
||||||
|
"entry alive past original TTL thanks to refresh");
|
||||||
PASS();
|
PASS();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -344,6 +383,8 @@ int main() {
|
|||||||
test_get_keys();
|
test_get_keys();
|
||||||
test_get_time_touch();
|
test_get_time_touch();
|
||||||
test_copy_assignment();
|
test_copy_assignment();
|
||||||
|
test_copy_constructor();
|
||||||
|
test_update_refresh_extends_ttl();
|
||||||
test_large_number_of_keys();
|
test_large_number_of_keys();
|
||||||
test_reinsert_after_remove();
|
test_reinsert_after_remove();
|
||||||
test_update_without_refresh_keeps_old_ttl();
|
test_update_without_refresh_keeps_old_ttl();
|
||||||
|
|||||||
+9
-3
@@ -197,6 +197,12 @@ def test_expire(c, cnt):
|
|||||||
ok, data = c.call("get", key="test:exp")
|
ok, data = c.call("get", key="test:exp")
|
||||||
cnt.check(ok and data == "new", "value after expire", f"got {data}")
|
cnt.check(ok and data == "new", "value after expire", f"got {data}")
|
||||||
|
|
||||||
|
ok, data = c.call("expire", key="test:exp:nonexistent", value="x")
|
||||||
|
cnt.check(ok and data is False, "expire nonexistent key returns false", f"got {data}")
|
||||||
|
|
||||||
|
ok, data = c.call("get", key="test:exp:nonexistent")
|
||||||
|
cnt.check(ok and data is None, "expire nonexistent does not insert", f"got {data}")
|
||||||
|
|
||||||
|
|
||||||
def test_missing_params(c, cnt):
|
def test_missing_params(c, cnt):
|
||||||
ok, data = c.call("set", key="only_key")
|
ok, data = c.call("set", key="only_key")
|
||||||
@@ -206,13 +212,13 @@ def test_missing_params(c, cnt):
|
|||||||
cnt.check(not ok, "get missing 'key' returns error", f"got {data}")
|
cnt.check(not ok, "get missing 'key' returns error", f"got {data}")
|
||||||
|
|
||||||
|
|
||||||
def test_concurrent(c, cnt):
|
def test_concurrent(c, cnt, host="127.0.0.1", port=8011):
|
||||||
"""并发 set/get:验证 mutex 保护不崩溃"""
|
"""并发 set/get:验证 mutex 保护不崩溃"""
|
||||||
errors = []
|
errors = []
|
||||||
results = []
|
results = []
|
||||||
|
|
||||||
def worker(worker_id):
|
def worker(worker_id):
|
||||||
client = MemdbClient()
|
client = MemdbClient(host, port)
|
||||||
for i in range(20):
|
for i in range(20):
|
||||||
key = f"concurrent:w{worker_id}:k{i}"
|
key = f"concurrent:w{worker_id}:k{i}"
|
||||||
ok, _ = client.call("set", key=key, value=str(worker_id))
|
ok, _ = client.call("set", key=key, value=str(worker_id))
|
||||||
@@ -266,7 +272,7 @@ def main():
|
|||||||
test_show(c, cnt)
|
test_show(c, cnt)
|
||||||
test_expire(c, cnt)
|
test_expire(c, cnt)
|
||||||
test_missing_params(c, cnt)
|
test_missing_params(c, cnt)
|
||||||
test_concurrent(c, cnt)
|
test_concurrent(c, cnt, host, port)
|
||||||
|
|
||||||
return 0 if cnt.summary() else 1
|
return 0 if cnt.summary() else 1
|
||||||
|
|
||||||
|
|||||||
@@ -50,6 +50,11 @@ cat scripts/std.mt | ./_run.sh # 与 bot4cj 相同的管道用法
|
|||||||
|
|
||||||
- checkpoint 后缀 `.hef` → Hailo NPU 后端,否则 → PyTorch 后端
|
- checkpoint 后缀 `.hef` → Hailo NPU 后端,否则 → PyTorch 后端
|
||||||
- `hz` 须 ≥ 推理 + 执行耗时(ONNX ~11ms / Hailo ~12ms,hz ≤ 50 安全)
|
- `hz` 须 ≥ 推理 + 执行耗时(ONNX ~11ms / Hailo ~12ms,hz ≤ 50 安全)
|
||||||
|
- **异步推理路径**:predictor 提供 `predict_async`(Hailo NPU 后端,见
|
||||||
|
actpred `hailo_infer.py`)时自动启用——提交推理后**并行执行上一步动作**
|
||||||
|
(滞后一拍执行,动作与状态错位一拍,10Hz 下 = 100ms,对微动作步态影响
|
||||||
|
可忽略),隐藏真实 I2C 执行耗时(实测 HEF 步耗时 ~16ms → ~12ms);
|
||||||
|
PyTorch/ONNX 同步路径不变
|
||||||
- `@Action` 表头列数须 ≤ `action_dim`(默认 18),闭环中跳过 `spk.delay` 阻塞
|
- `@Action` 表头列数须 ≤ `action_dim`(默认 18),闭环中跳过 `spk.delay` 阻塞
|
||||||
- **帧源**:闭环从 camera 守护线程队列取最近 F 帧(无阻塞),`@Camera` 的
|
- **帧源**:闭环从 camera 守护线程队列取最近 F 帧(无阻塞),`@Camera` 的
|
||||||
`framerate` 建议 ≥ 2×`hz` 以保证帧新鲜度;`capacity` 应 ≥ F(帧窗长度)。
|
`framerate` 建议 ≥ 2×`hz` 以保证帧新鲜度;`capacity` 应 ≥ F(帧窗长度)。
|
||||||
|
|||||||
+49
-12
@@ -163,6 +163,7 @@ class RobotStruct:
|
|||||||
raise ValueError("no @Action matrix in script")
|
raise ValueError("no @Action matrix in script")
|
||||||
columns = self._validate_action(action) # Q05: 执行前整表校验
|
columns = self._validate_action(action) # Q05: 执行前整表校验
|
||||||
delay_idx = self._find_delay_idx(columns)
|
delay_idx = self._find_delay_idx(columns)
|
||||||
|
freq_idx = self._find_freq_idx(columns)
|
||||||
|
|
||||||
camera_var = None
|
camera_var = None
|
||||||
imu_vars = []
|
imu_vars = []
|
||||||
@@ -198,8 +199,8 @@ class RobotStruct:
|
|||||||
if recorder is not None:
|
if recorder is not None:
|
||||||
recorder.record(
|
recorder.record(
|
||||||
self._grab_frame(camera_var),
|
self._grab_frame(camera_var),
|
||||||
self._build_action_vec(
|
self._record_vec(step_vals, recorder.action_dim,
|
||||||
step_vals, recorder.action_dim),
|
freq_idx),
|
||||||
imu_vec=self._read_imus(imu_vars))
|
imu_vec=self._read_imus(imu_vars))
|
||||||
self._execute_step(step_vals, columns)
|
self._execute_step(step_vals, columns)
|
||||||
if self.dry_run:
|
if self.dry_run:
|
||||||
@@ -209,7 +210,7 @@ class RobotStruct:
|
|||||||
if recorder is not None:
|
if recorder is not None:
|
||||||
recorder.record(
|
recorder.record(
|
||||||
self._grab_frame(camera_var),
|
self._grab_frame(camera_var),
|
||||||
self._build_action_vec(vals, recorder.action_dim),
|
self._record_vec(vals, recorder.action_dim, freq_idx),
|
||||||
imu_vec=self._read_imus(imu_vars))
|
imu_vec=self._read_imus(imu_vars))
|
||||||
self._execute_step(vals, columns)
|
self._execute_step(vals, columns)
|
||||||
|
|
||||||
@@ -286,21 +287,41 @@ class RobotStruct:
|
|||||||
dt = 1.0 / hz
|
dt = 1.0 / hz
|
||||||
step = 0
|
step = 0
|
||||||
overrun = 0
|
overrun = 0
|
||||||
log.info("closed loop start: hz=%.1f F=%d H=%d action_dim=%d",
|
# 异步模式:predictor 声明 predict_async 时启用——推理期间并行执行
|
||||||
hz, F, H, action_dim)
|
# 上一步动作(滞后一拍执行,动作与状态错位一拍,10Hz 下 = 100ms,
|
||||||
|
# 对微动作步态影响可忽略),省去执行耗时(实测 HEF 步耗时 ~16ms → ~12ms)
|
||||||
|
use_async = hasattr(predictor, "predict_async")
|
||||||
|
prev_act = None
|
||||||
|
log.info("closed loop start: hz=%.1f F=%d H=%d action_dim=%d%s",
|
||||||
|
hz, F, H, action_dim,
|
||||||
|
" async" if use_async else "")
|
||||||
try:
|
try:
|
||||||
while max_steps is None or step < max_steps:
|
while max_steps is None or step < max_steps:
|
||||||
t0 = time.monotonic()
|
t0 = time.monotonic()
|
||||||
frames.extend(self._loop_recent_frames(camera_var, F)) # 1 采样
|
frames.extend(self._loop_recent_frames(camera_var, F)) # 1 采样
|
||||||
imu = self._read_imus(imu_vars)
|
imu = self._read_imus(imu_vars)
|
||||||
if use_imu: # 2 推理
|
if use_async: # 2 推理(异步)
|
||||||
act = predictor.predict(list(frames), np.array(history),
|
if use_imu:
|
||||||
imu=self._pad_imu(
|
job = predictor.predict_async(
|
||||||
imu, predictor.imu_dim))
|
list(frames), np.array(history),
|
||||||
|
imu=self._pad_imu(imu, predictor.imu_dim))
|
||||||
|
else:
|
||||||
|
job = predictor.predict_async(
|
||||||
|
list(frames), np.array(history))
|
||||||
|
if prev_act is not None:
|
||||||
|
self._execute_action_vec(prev_act, exec_cols) # 3 执行(∥ NPU)
|
||||||
|
act = np.asarray(job.result(timeout=10.0),
|
||||||
|
dtype=np.float32).reshape(-1)
|
||||||
|
prev_act = act.copy()
|
||||||
else:
|
else:
|
||||||
act = predictor.predict(list(frames), np.array(history))
|
if use_imu:
|
||||||
act = np.asarray(act, dtype=np.float32).reshape(-1)
|
act = predictor.predict(list(frames), np.array(history),
|
||||||
self._execute_action_vec(act, exec_cols) # 3 执行
|
imu=self._pad_imu(
|
||||||
|
imu, predictor.imu_dim))
|
||||||
|
else:
|
||||||
|
act = predictor.predict(list(frames), np.array(history))
|
||||||
|
act = np.asarray(act, dtype=np.float32).reshape(-1)
|
||||||
|
self._execute_action_vec(act, exec_cols) # 3 执行
|
||||||
history.append(act.copy()) # 4 回写
|
history.append(act.copy()) # 4 回写
|
||||||
if recorder is not None:
|
if recorder is not None:
|
||||||
recorder.record(
|
recorder.record(
|
||||||
@@ -416,6 +437,22 @@ class RobotStruct:
|
|||||||
return j
|
return j
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
def _find_freq_idx(self, columns):
|
||||||
|
"""找到 spk.F(frequency)列索引;无 Speaker 返回 None"""
|
||||||
|
for j, (dev, param) in enumerate(columns):
|
||||||
|
if isinstance(self.M.get(dev), SpeakerVar) and \
|
||||||
|
self._resolve_param(param) == "frequency":
|
||||||
|
return j
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _record_vec(self, vals, action_dim, freq_idx=None):
|
||||||
|
"""构建录制动作向量;spk.F 列固定写 0(只影响声音不影响运动,
|
||||||
|
与模型固定输出契约一致——音频元数据不进入学习数据)"""
|
||||||
|
vec = self._build_action_vec(vals, action_dim)
|
||||||
|
if freq_idx is not None:
|
||||||
|
vec[freq_idx] = 0.0
|
||||||
|
return vec
|
||||||
|
|
||||||
def _build_step_vals(self, prev_vals, target_vals, columns, k, n):
|
def _build_step_vals(self, prev_vals, target_vals, columns, k, n):
|
||||||
"""为微步 k 构建插值后的值列表"""
|
"""为微步 k 构建插值后的值列表"""
|
||||||
vals = []
|
vals = []
|
||||||
|
|||||||
@@ -104,6 +104,14 @@ def _make_robot(mt_text, mock_i2c):
|
|||||||
return RobotStruct(io.StringIO(mt_text), dry_run=True)
|
return RobotStruct(io.StringIO(mt_text), dry_run=True)
|
||||||
|
|
||||||
|
|
||||||
|
class AsyncMockPredictor(MockPredictor):
|
||||||
|
"""带 predict_async 的鸭子类型桩:异步返回与同步相同的可辨识动作"""
|
||||||
|
|
||||||
|
def predict_async(self, images, action_history):
|
||||||
|
act = self.predict(images, action_history)
|
||||||
|
return type("Job", (), {"result": lambda self, timeout=None: act})()
|
||||||
|
|
||||||
|
|
||||||
# ---------------- 冷启动 ----------------
|
# ---------------- 冷启动 ----------------
|
||||||
|
|
||||||
def test_cold_start_fills_windows(mock_i2c):
|
def test_cold_start_fills_windows(mock_i2c):
|
||||||
@@ -230,6 +238,44 @@ def test_loop_pacing(mock_i2c):
|
|||||||
assert elapsed >= 0.09 # 允许少量计时误差
|
assert elapsed >= 0.09 # 允许少量计时误差
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------- 异步推理路径 ----------------
|
||||||
|
|
||||||
|
def test_loop_async_uses_predict_async(mock_i2c):
|
||||||
|
"""predictor 声明 predict_async 时闭环走异步路径(执行滞后一拍)"""
|
||||||
|
robot = _make_robot(LOOP_MT, mock_i2c)
|
||||||
|
pred = AsyncMockPredictor(frame_len=3, history_len=2, action_dim=4)
|
||||||
|
robot.active()
|
||||||
|
robot.run_loop(pred, hz=1000.0, max_steps=3)
|
||||||
|
robot.deactive()
|
||||||
|
# 3 步闭环 → 3 次异步推理(calls 由桩内 predict 记录)
|
||||||
|
assert len(pred.calls) == 3
|
||||||
|
# history 回写与同步一致:第 3 步历史末步 = 第 2 步预测动作 [2,3,4,5]
|
||||||
|
_, hist2 = pred.calls[2]
|
||||||
|
np.testing.assert_allclose(hist2[-1], [2, 3, 4, 5])
|
||||||
|
|
||||||
|
|
||||||
|
def test_loop_async_executes_prev_action(mock_i2c):
|
||||||
|
"""异步模式执行滞后一拍:第 k 步执行第 k-1 步预测的动作"""
|
||||||
|
robot = _make_robot(LOOP_MT, mock_i2c)
|
||||||
|
fd = robot.M["i2c-1"].play({})["fd"]
|
||||||
|
robot.active()
|
||||||
|
mock_i2c.journal.clear()
|
||||||
|
pred = AsyncMockPredictor(frame_len=3, history_len=2, action_dim=4)
|
||||||
|
robot.run_loop(pred, hz=1000.0, max_steps=3)
|
||||||
|
robot.deactive()
|
||||||
|
|
||||||
|
writes = [(e[2], e[3]) for e in mock_i2c.journal
|
||||||
|
if e[0] == "write_reg" and e[1] == fd]
|
||||||
|
# 预测序列:step0→[1,2,3,4], step1→[2,3,4,5], step2→[3,4,5,6]
|
||||||
|
# 执行序列(滞后一拍):step1 执行 [1,2,3,4],step2 执行 [2,3,4,5]
|
||||||
|
off = lambda deg: int((0.5 + 2 * deg / 180) * 4096 / 20) # noqa: E731
|
||||||
|
off_reg = lambda ch: 0x06 + 4 * ch + 2 # noqa: E731 —— LEDx_OFF_L
|
||||||
|
s1_vals = [v for r, v in writes if r == off_reg(0)] # s1 ch0
|
||||||
|
s2_vals = [v for r, v in writes if r == off_reg(1)] # s2 ch1
|
||||||
|
assert s1_vals == [off(1) & 0xFF, off(2) & 0xFF] # step0/step1 预测的 s1
|
||||||
|
assert s2_vals == [off(2) & 0xFF, off(3) & 0xFF] # step0/step1 预测的 s2
|
||||||
|
|
||||||
|
|
||||||
# ---------------- 帧源:守护线程队列 ----------------
|
# ---------------- 帧源:守护线程队列 ----------------
|
||||||
|
|
||||||
class _SeqGrabber:
|
class _SeqGrabber:
|
||||||
|
|||||||
Reference in new issue
Block a user