- ReadFileTool: replace readlines() with on-demand range reading; add reverse block reading for large file tail; limit single line to 10KB; hard-cap max_lines to 500
- SubprocessCommandExecutor: replace capture_output with line-by-line Popen reading; enforce 1MB output limit with immediate process termination; merge stdout/stderr counting for both sync and streaming modes
- Raise TOOL_MAX_RESULT_LENGTH default from 10KB to 1MB as final safety net
- Add SECURITY_AUDIT.md documenting vulnerabilities and remediation roadmap
- Update README with memory safety features and v2.3 changelog
P1 fixes:
- P1-1 _handle_reload: empty string/0/false/falsy name no longer clears all
caches; only explicit null/omitted name triggers full cache clear; non-string
name raises ConfigError instead of being silently str()-converted
- P1-2 ConfigStore.get(): return dict(entry) shallow copy instead of direct
cache reference to prevent callers from mutating the internal cache
- P1-3 list(): resolve symlinks via os.path.realpath() and skip entries whose
real path escapes configs/ root; prevents leaking external file metadata
(size, mtime) through directory-traversing symlinks
- P1-4 _handle_get key: validate key is a string (isinstance check); reject
non-string/empty key with clear ConfigError instead of misleading str()
conversion leading to "key not found" errors
P2 fixes:
- P2-1 ConfigParser: use interpolation=None to disable %(key)s interpolation,
preventing InterpolationError on config values containing bare % characters
- P2-2 argparse: update description from "pymsg/TCP" to "pymsg RpcServer/TCP"
to match module docstring
Also update SECURITY_AUDIT_20260801.md to mark issues 9-14 as fixed and
refresh the security mechanisms summary table; all P0/P1 issues are now resolved.
- Split _resolve() into _validate_name() (pure string checks, lock-free
fast-fail) and _resolve_locked() (filesystem checks, must be called
under lock)
- Perform all filesystem operations (realpath → isfile → stat → open)
inside self._lock in get(), eliminating the window between path
validation and file open where a symlink swap could escape the
configs directory
- Apply same fix to reload(): validate name outside lock, resolve path
inside lock to prevent TOCTOU during targeted reload
- Hold lock during list() directory traversal so the `cached` flag
(which reads self._cache) is consistent with directory state; wrap
os.stat() in try/except to tolerate files disappearing mid-traversal
- Update docstring and README:
- Protocol description corrected to pymsg RpcServer/TCP
- Python client examples updated from legacy MsgClient to RpcClient
- Security section now documents TOCTOU mitigation and Slowloris
protection (added by earlier pymsg fix)
- Replace legacy MsgClient (low-level TCP frames) with RpcClient (high-level
RPC) to match dbv2's RpcServer interface; the old MsgClient-based version
was incompatible with the current RPC routing protocol
- Apply all hardening fixes from the service copies:
- Outer try/except in _flush_loop to prevent any exception from killing
the background thread
- _safe_json() helper for unserializable extra fields (fallback to str/repr)
- Timestamp generated at push() (enqueue time) instead of _send() time
- Event-based flush() synchronization instead of unreliable sleep(0.6)
- Queue maxsize=10000 with drop-oldest policy to prevent unbounded memory
- put_nowait with Full handling ensures push() never blocks business threads
- RpcClient construction wrapped in try/except
- Add explicit service_name parameter example for all 4 services in comments
- This is the template file referenced by db/README.md and copied by new
services; keeping it in sync prevents propagating old bugs
- Wrap _flush_loop main body in outer try/except to prevent any
unexpected exception (network errors, serialization errors, etc.)
from killing the background thread (which would cause unbounded
queue growth and eventual OOM)
- Wrap _send() call in inner try/except as final safety net
- Wrap RpcClient() construction in try/except (connection failures
during client creation were previously unhandled)
- Add _safe_json() helper: unserializable extra fields fall back to
repr()/str() instead of raising TypeError that could kill thread
- Generate timestamp at push() (enqueue time) instead of _send() time,
so timestamps reflect actual event time even when queue is backed up
- Replace unreliable time.sleep(0.6) in flush() with _flush_done Event
for proper synchronization between caller and background thread
- Add _QUEUE_MAXSIZE=10000 with drop-oldest policy to prevent unbounded
memory growth even in pathological cases
- Use put_nowait with Full handling to ensure push() never blocks
business threads
- Sync fix to all 4 copies: confs/static/smtp/dnfs
Prompt system bugs and improvements:
1. **SKILL_PREFIX bug fixed**: SKILL_PREFIX = "1_" only matched files starting
with "1_", causing 2_ through 10_ skill prompt files to be misclassified
as "other" files and sorted in wrong order (e.g., 10_task_handling.md
sorted before 2_memory_save.md due to lexicographic ordering). Replaced
with regex _SKILL_FILE_RE = re.compile(r'^(\d+)_') and _skill_sort_key()
that extracts the numeric prefix for proper numeric sorting.
2. **_init_system_prompts used unsorted list**: get_prompts_list() returned
prompts in os.listdir() lexicographic order, which put 100_STATUS.md
before 1_memory_search.md and 10_task_handling.md before 1_memory_search.md.
Now uses get_sorted_prompts_list() for correct ordering:
identity → skill prompts (1-10 by number) → soul → status.
3. **Verbose pre_prompt injected every turn (~30 lines)**: The pre_prompt
contained detailed workflow instructions (search_skills first, memory
search, ask_user, etc.) that duplicated what system skill prompts
already say, wasting ~300+ tokens per user message. Worse, the same
verbose prompt was injected on recursive think_and_respond calls after
tool execution, confusing the model with repeated "第一步搜索技能"
instructions when it was already mid-conversation with tool results.
- Reduced pre_prompt to 5 concise bullet points as a reminder
- Added is_continuation parameter: recursive calls after tool execution
skip pre_prompt entirely and just pass "请基于以上工具执行结果继续回答。"
- Removed post_prompt (save_memory/update_soul instructions already in
2_memory_save.md and 8_identity_management.md)
4. **Unreasonable large-file thresholds**: 3_file_operations.md default
said >50KB or >200 lines is "too large" — too conservative for code
files. Raised to 500KB/2000 lines; added tail=true and binary file
guidance.
5. **Overly rigid memory search instruction**: 1_memory_search.md said
"每次都要做！" (must do every time), causing unnecessary memory searches
for simple greetings/acknowledgments. Softened to "根据话题相关性判断",
with explicit examples of when to skip (greetings, confirmations).
New/changed methods:
- PromptManager._skill_sort_key(cls, filename) - extract numeric prefix
- PromptManager._prompt_sort_key(self, p) - tuple-based sort key
- PromptManager.get_sorted_prompts_list() - returns correctly ordered list
- PromptManager.get_skill_prompts() - uses regex instead of startswith("1_")
- DeepSeekChat.think_and_respond() - added is_continuation parameter
- Add MAX_CONFIG_SIZE constant (1 MiB) to limit single config file size
- Check file size via os.stat() before reading into memory
- Oversized files raise ConfigError with descriptive message instead of
being fully read (which could cause OOM with multi-GB files)
- Use single os.stat() call to get both mtime and size atomically
- Update README: document size limit in security mechanisms and notes
- Existing small configs (ini/json/yaml, typically <10KB) are unaffected
- Add configurable recv_timeout (default 10s) to MsgServer
- _handle_client now uses recv(s, timeout=...) instead of infinite blocking recv(s)
- Catch TimeoutError and close the slow client connection
- This prevents a single stalled connection from blocking the entire main select loop
- Update README with recv_timeout parameter and Slowloris mitigation docs
- Backward compatible: keyword-only arg with sensible default
- Fix all references from main.py to aibox.py (actual entry file)
- Update installation to use pip install -r requirements.txt
- Expand WebSocket message types table: add line/task_error/task_added/
notification/connection_ack/interrupt_ack, correct chunk/complete/error semantics
- Fix message flow example: tool calls are reported via 'line' messages,
not a non-existent 'tool_call' type
- Document _sanitize_tool_result() and error recovery (_rollback_messages)
in development guide
- Add FAQ entry for tool-call-induced API failures
- Add v2.1 changelog covering all robustness fixes (reasoning_content
filtering, tool_call id fallback, control character sanitization,
message rollback on API failure)
Root causes of API request failures after tool calls:
1. reasoning_content (an API response field) was echoed back in assistant
messages, which could cause API format errors
2. tool_call id could be empty string when model didn't return one in a
streaming chunk, producing invalid tool_calls that caused 400 errors
3. Tool execute() could return None or non-string types; direct string
operations (len(), + "\n", [:200]) raised TypeError and produced
non-string content in tool messages, breaking JSON serialization
4. Tool output (e.g. from run_command) could contain control characters
(NULL/BEL/BS) that corrupted JSON payloads
5. When API request failed (network/4xx/5xx/parse error), messages added
in that turn (user msg, assistant msg with tool_calls, tool responses)
remained in self.messages, permanently poisoning conversation history
so all subsequent requests would also fail
Fixes:
- Stop sending reasoning_content back to API in message construction;
normalize assistant message content: null when tool_calls present, ""
when neither content nor tool_calls exist, str otherwise
- Sanitize tool messages: ensure tool_call_id is non-empty (fallback id)
and content is always a string
- Generate fallback tool_call id when model omits it; handle streaming id
delivery where id arrives in a later chunk
- Add _sanitize_tool_result(): coerce None/non-string to str, strip
control characters (preserve \n \t \r) to prevent JSON/API corruption
- Wrap execute_tool call in try/except in _handle_tool_calls for double
safety; ensure history logging never crashes
- Truncate oversized results with an explicit note instead of replacing
with a generic error that lost all useful output
- Add _rollback_messages(to_length) and record msgs_before_turn at start
of think_and_respond; on any exception (RequestException, JSONDecodeError,
or unexpected Exception), rollback self.messages to the pre-turn state
so the conversation is not left in a broken state
- In _handle_tool_calls, wrap execute_tool in try/except and always
produce a safe string result
Root cause: aibox.py think_and_respond recursively calls itself after tool
execution, and each recursive call's complete callback carries only the
final-round full_response. The old finish_ai logic treated complete as a
full-text alignment signal — when full_text != self._ai_text (which always
happened after recursive tool calls because _ai_text accumulated all chunks
while complete only held the last round), it triggered remove+remount,
causing visible duplicate/flash of AI responses. A secondary issue was
dialogue_ended arriving before complete, which made a second finish_ai call
remount the message a second time.
Changes:
- Remove the full-text replacement branch in finish_ai; chunk streaming is
now the authoritative content source, complete is a finalization signal only
- Add _ai_finalized flag: start_ai resets it, finish_ai sets it; repeated
finish_ai calls (dialogue_ended + complete) return early to prevent remount
- append_ai drops late chunks after finalization to avoid dirty appends
- Empty pre-built widget without chunks still gets filled via w.update(full_text)
- Full lost-frame fallback (no widget + full_text) is preserved
- Update BotOps/README.md protocol section to reflect new complete semantics
- New token_manager.js: silent refresh timer (80% lifetime) + global
fetch 401 interceptor with one-shot retry.
- dnfs.py /whoami now returns iat, created_at, keep_time for expiry
estimation.
- Dnfs.html loads token_manager.js before dnfs.js (zero change to dnfs.js).
- Graceful fallback: stale tokens without new fields, refresh failures,
and /refresh_token 401 itself all degrade to the existing 401→login flow.
- POST /refresh_token: accept current Bearer token (or Cookie fallback),
verify RSA signature, check grace window (iat + keep_time * multiplier)
and refresh count limit, then issue a new token.
- Token payload extended with iat, refresh_count, max_refresh fields.
- iat preserves original issue time; created_at updates on each refresh.
- Rate limited: 5 req / 5 min / IP via RateLimiter.
- Config: Token.refresh_grace_multiplier (default 2), Token.max_refresh (default 10).
- Backward compatible: old tokens without new fields still pass proxy checks.
qing