Merge WebSocket client functionality into HttpClient using Boost.Beast,
enabling a single client instance to handle both HTTP (libcurl) and
WebSocket (ws:// and wss://) protocols. Backward-compatible type aliases
(WebSocketClient, WebSocketClientBase) preserve existing code.
Changes:
- Add synchronous WebSocket client: ws_connect/ws_send/ws_recv/ws_close
- Support ws:// (plain TCP) and wss:// (TLS with SNI) connections
- Layered stream architecture with beast::tcp_stream for timeout support
- Binary message support via ws_send_binary()
- Update Makefile to link libcurl (-lcurl)
- Add README.md documenting unified HttpServer + HttpClient API
- Add _ai_prefab flag to track pre-created empty AI widget
- start_reasoning() now reorders widgets: remove empty AI widget, mount reasoning first, then remount AI widget
- This ensures reasoning always appears BEFORE the actual answer
- Filter out 'Conversation started'/'对话开始' info notifications (implicit event, no need to show)
- Increase reasoning display limit from 2000 to 3000 chars
Merge WebSocketServer functionality into HttpServer so a single port can
serve both HTTP(S) and WebSocket(S) endpoints while preserving full backward
compatibility for existing HTTP interfaces.
Key changes:
- Add WebSocket route registration: ws(), onWsConnect(), onWsDisconnect()
- Add client management: sendToWsClient(), broadcastWs(), disconnectWsClient()
- Rewrite Session class using beast::tcp_stream for unified HTTP/WS handling
- Support automatic ws:// and wss:// upgrades on HTTP/HTTPS ports
- Thread-safe message queue with asio::post for cross-thread sends
- Update Makefile to link OpenSSL and pthread for TLS/WebSocket support
- Add test_server.cpp demonstrating HTTP + WebSocket mixed usage
- Implement http.Hijacker on responseRecorder to enable WebSocket upgrade
(delegates Hijack() to underlying ResponseWriter, marks hijacked state)
- After WebSocket upgrade (101 Switching Protocols), skip HTTP response-phase
WAF checks since connection becomes a binary frame stream
- Log WebSocket upgrades at Info level for observability
- Fix P2-3: use r.Proto instead of hardcoded "HTTP/1.1" in ProcessResponseHeaders
for correct HTTP/2 protocol version reporting
- Add Write() method to responseRecorder to ensure WriteHeader is called
before first Write (complies with http.ResponseWriter contract)
- Add bufio/net imports required by Hijacker interface
- Update README with WebSocket support section and v1.2 changelog
- Update SECURITY_AUDIT to mark P2-3 and P2-4 as fixed
- Add dnfs.ini to confs/configs/ with server, database, ai_review, confs sections
- Refactor dnfs.py to use three-layer fail-safe config loading (defaults → confs → env vars)
- Unify timeouts for DB RPC and AI review calls via configuration
- Add _merge_ini() whitelist merging to prevent unexpected config injection
- Environment variables have highest priority for Docker networking (db/ai_review hostnames)
- Document config loading mechanism in both confs/README.md and dnfs/README.md
Follows the same pattern as static service integration; confs failure falls back
gracefully to built-in defaults + environment variables.
- P0: add 10MB hard request body size limit with io.LimitReader,
return 413 Request Entity Too Large when exceeded, prevents OOM
- P0: configure http.Server timeouts: ReadHeaderTimeout=10s (Slowloris),
ReadTimeout=15s, WriteTimeout=30s, IdleTimeout=120s, MaxHeaderBytes=1MB
- P1: properly handle body read errors, log and return on client disconnect
- P2: drain and close r.Body in defer to ensure TCP connection reuse
- Log errors from ReadRequestBodyFrom and ProcessRequestBody
- Inject security headers on 413 responses
- Add security audit report (docs/SECURITY_AUDIT_20260801.md)
- Update README with timeout/body-limit docs and changelog
- ReadFileTool: replace readlines() with on-demand range reading; add reverse block reading for large file tail; limit single line to 10KB; hard-cap max_lines to 500
- SubprocessCommandExecutor: replace capture_output with line-by-line Popen reading; enforce 1MB output limit with immediate process termination; merge stdout/stderr counting for both sync and streaming modes
- Raise TOOL_MAX_RESULT_LENGTH default from 10KB to 1MB as final safety net
- Add SECURITY_AUDIT.md documenting vulnerabilities and remediation roadmap
- Update README with memory safety features and v2.3 changelog
P1 fixes:
- P1-1 _handle_reload: empty string/0/false/falsy name no longer clears all
caches; only explicit null/omitted name triggers full cache clear; non-string
name raises ConfigError instead of being silently str()-converted
- P1-2 ConfigStore.get(): return dict(entry) shallow copy instead of direct
cache reference to prevent callers from mutating the internal cache
- P1-3 list(): resolve symlinks via os.path.realpath() and skip entries whose
real path escapes configs/ root; prevents leaking external file metadata
(size, mtime) through directory-traversing symlinks
- P1-4 _handle_get key: validate key is a string (isinstance check); reject
non-string/empty key with clear ConfigError instead of misleading str()
conversion leading to "key not found" errors
P2 fixes:
- P2-1 ConfigParser: use interpolation=None to disable %(key)s interpolation,
preventing InterpolationError on config values containing bare % characters
- P2-2 argparse: update description from "pymsg/TCP" to "pymsg RpcServer/TCP"
to match module docstring
Also update SECURITY_AUDIT_20260801.md to mark issues 9-14 as fixed and
refresh the security mechanisms summary table; all P0/P1 issues are now resolved.
- Split _resolve() into _validate_name() (pure string checks, lock-free
fast-fail) and _resolve_locked() (filesystem checks, must be called
under lock)
- Perform all filesystem operations (realpath → isfile → stat → open)
inside self._lock in get(), eliminating the window between path
validation and file open where a symlink swap could escape the
configs directory
- Apply same fix to reload(): validate name outside lock, resolve path
inside lock to prevent TOCTOU during targeted reload
- Hold lock during list() directory traversal so the `cached` flag
(which reads self._cache) is consistent with directory state; wrap
os.stat() in try/except to tolerate files disappearing mid-traversal
- Update docstring and README:
- Protocol description corrected to pymsg RpcServer/TCP
- Python client examples updated from legacy MsgClient to RpcClient
- Security section now documents TOCTOU mitigation and Slowloris
protection (added by earlier pymsg fix)
- Replace legacy MsgClient (low-level TCP frames) with RpcClient (high-level
RPC) to match dbv2's RpcServer interface; the old MsgClient-based version
was incompatible with the current RPC routing protocol
- Apply all hardening fixes from the service copies:
- Outer try/except in _flush_loop to prevent any exception from killing
the background thread
- _safe_json() helper for unserializable extra fields (fallback to str/repr)
- Timestamp generated at push() (enqueue time) instead of _send() time
- Event-based flush() synchronization instead of unreliable sleep(0.6)
- Queue maxsize=10000 with drop-oldest policy to prevent unbounded memory
- put_nowait with Full handling ensures push() never blocks business threads
- RpcClient construction wrapped in try/except
- Add explicit service_name parameter example for all 4 services in comments
- This is the template file referenced by db/README.md and copied by new
services; keeping it in sync prevents propagating old bugs
- Wrap _flush_loop main body in outer try/except to prevent any
unexpected exception (network errors, serialization errors, etc.)
from killing the background thread (which would cause unbounded
queue growth and eventual OOM)
- Wrap _send() call in inner try/except as final safety net
- Wrap RpcClient() construction in try/except (connection failures
during client creation were previously unhandled)
- Add _safe_json() helper: unserializable extra fields fall back to
repr()/str() instead of raising TypeError that could kill thread
- Generate timestamp at push() (enqueue time) instead of _send() time,
so timestamps reflect actual event time even when queue is backed up
- Replace unreliable time.sleep(0.6) in flush() with _flush_done Event
for proper synchronization between caller and background thread
- Add _QUEUE_MAXSIZE=10000 with drop-oldest policy to prevent unbounded
memory growth even in pathological cases
- Use put_nowait with Full handling to ensure push() never blocks
business threads
- Sync fix to all 4 copies: confs/static/smtp/dnfs
Prompt system bugs and improvements:
1. **SKILL_PREFIX bug fixed**: SKILL_PREFIX = "1_" only matched files starting
with "1_", causing 2_ through 10_ skill prompt files to be misclassified
as "other" files and sorted in wrong order (e.g., 10_task_handling.md
sorted before 2_memory_save.md due to lexicographic ordering). Replaced
with regex _SKILL_FILE_RE = re.compile(r'^(\d+)_') and _skill_sort_key()
that extracts the numeric prefix for proper numeric sorting.
2. **_init_system_prompts used unsorted list**: get_prompts_list() returned
prompts in os.listdir() lexicographic order, which put 100_STATUS.md
before 1_memory_search.md and 10_task_handling.md before 1_memory_search.md.
Now uses get_sorted_prompts_list() for correct ordering:
identity → skill prompts (1-10 by number) → soul → status.
3. **Verbose pre_prompt injected every turn (~30 lines)**: The pre_prompt
contained detailed workflow instructions (search_skills first, memory
search, ask_user, etc.) that duplicated what system skill prompts
already say, wasting ~300+ tokens per user message. Worse, the same
verbose prompt was injected on recursive think_and_respond calls after
tool execution, confusing the model with repeated "第一步搜索技能"
instructions when it was already mid-conversation with tool results.
- Reduced pre_prompt to 5 concise bullet points as a reminder
- Added is_continuation parameter: recursive calls after tool execution
skip pre_prompt entirely and just pass "请基于以上工具执行结果继续回答。"
- Removed post_prompt (save_memory/update_soul instructions already in
2_memory_save.md and 8_identity_management.md)
4. **Unreasonable large-file thresholds**: 3_file_operations.md default
said >50KB or >200 lines is "too large" — too conservative for code
files. Raised to 500KB/2000 lines; added tail=true and binary file
guidance.
5. **Overly rigid memory search instruction**: 1_memory_search.md said
"每次都要做！" (must do every time), causing unnecessary memory searches
for simple greetings/acknowledgments. Softened to "根据话题相关性判断",
with explicit examples of when to skip (greetings, confirmations).
New/changed methods:
- PromptManager._skill_sort_key(cls, filename) - extract numeric prefix
- PromptManager._prompt_sort_key(self, p) - tuple-based sort key
- PromptManager.get_sorted_prompts_list() - returns correctly ordered list
- PromptManager.get_skill_prompts() - uses regex instead of startswith("1_")
- DeepSeekChat.think_and_respond() - added is_continuation parameter
- Add MAX_CONFIG_SIZE constant (1 MiB) to limit single config file size
- Check file size via os.stat() before reading into memory
- Oversized files raise ConfigError with descriptive message instead of
being fully read (which could cause OOM with multi-GB files)
- Use single os.stat() call to get both mtime and size atomically
- Update README: document size limit in security mechanisms and notes
- Existing small configs (ini/json/yaml, typically <10KB) are unaffected
- Add configurable recv_timeout (default 10s) to MsgServer
- _handle_client now uses recv(s, timeout=...) instead of infinite blocking recv(s)
- Catch TimeoutError and close the slow client connection
- This prevents a single stalled connection from blocking the entire main select loop
- Update README with recv_timeout parameter and Slowloris mitigation docs
- Backward compatible: keyword-only arg with sensible default
- Fix all references from main.py to aibox.py (actual entry file)
- Update installation to use pip install -r requirements.txt
- Expand WebSocket message types table: add line/task_error/task_added/
notification/connection_ack/interrupt_ack, correct chunk/complete/error semantics
- Fix message flow example: tool calls are reported via 'line' messages,
not a non-existent 'tool_call' type
- Document _sanitize_tool_result() and error recovery (_rollback_messages)
in development guide
- Add FAQ entry for tool-call-induced API failures
- Add v2.1 changelog covering all robustness fixes (reasoning_content
filtering, tool_call id fallback, control character sanitization,
message rollback on API failure)
Root causes of API request failures after tool calls:
1. reasoning_content (an API response field) was echoed back in assistant
messages, which could cause API format errors
2. tool_call id could be empty string when model didn't return one in a
streaming chunk, producing invalid tool_calls that caused 400 errors
3. Tool execute() could return None or non-string types; direct string
operations (len(), + "\n", [:200]) raised TypeError and produced
non-string content in tool messages, breaking JSON serialization
4. Tool output (e.g. from run_command) could contain control characters
(NULL/BEL/BS) that corrupted JSON payloads
5. When API request failed (network/4xx/5xx/parse error), messages added
in that turn (user msg, assistant msg with tool_calls, tool responses)
remained in self.messages, permanently poisoning conversation history
so all subsequent requests would also fail
Fixes:
- Stop sending reasoning_content back to API in message construction;
normalize assistant message content: null when tool_calls present, ""
when neither content nor tool_calls exist, str otherwise
- Sanitize tool messages: ensure tool_call_id is non-empty (fallback id)
and content is always a string
- Generate fallback tool_call id when model omits it; handle streaming id
delivery where id arrives in a later chunk
- Add _sanitize_tool_result(): coerce None/non-string to str, strip
control characters (preserve \n \t \r) to prevent JSON/API corruption
- Wrap execute_tool call in try/except in _handle_tool_calls for double
safety; ensure history logging never crashes
- Truncate oversized results with an explicit note instead of replacing
with a generic error that lost all useful output
- Add _rollback_messages(to_length) and record msgs_before_turn at start
of think_and_respond; on any exception (RequestException, JSONDecodeError,
or unexpected Exception), rollback self.messages to the pre-turn state
so the conversation is not left in a broken state
- In _handle_tool_calls, wrap execute_tool in try/except and always
produce a safe string result
Root cause: aibox.py think_and_respond recursively calls itself after tool
execution, and each recursive call's complete callback carries only the
final-round full_response. The old finish_ai logic treated complete as a
full-text alignment signal — when full_text != self._ai_text (which always
happened after recursive tool calls because _ai_text accumulated all chunks
while complete only held the last round), it triggered remove+remount,
causing visible duplicate/flash of AI responses. A secondary issue was
dialogue_ended arriving before complete, which made a second finish_ai call
remount the message a second time.
Changes:
- Remove the full-text replacement branch in finish_ai; chunk streaming is
now the authoritative content source, complete is a finalization signal only
- Add _ai_finalized flag: start_ai resets it, finish_ai sets it; repeated
finish_ai calls (dialogue_ended + complete) return early to prevent remount
- append_ai drops late chunks after finalization to avoid dirty appends
- Empty pre-built widget without chunks still gets filled via w.update(full_text)
- Full lost-frame fallback (no widget + full_text) is preserved
- Update BotOps/README.md protocol section to reflect new complete semantics
qing