fix(libhttp): HttpClient 响应大小限制防 OOM

- request_stream_sync 改为真流式,按块回调不二次缓冲
- 新增 set_max_response_size/get_max_response_size(默认 10MB,对齐 WAF)
- write_callback 超限返回 0 中止传输并抛异常
- 更新审计文档 P1-2 状态、README v3.4.0、INDEX 移除不存在的文档
This commit is contained in:
qing committed 2026-08-04 15:12:06 +08:00
1 parent dc52ec13ad
commit 85e0d0740b
5 files changed
+64 -10

No files matched your search

+1 -3
View File
@@ -7,7 +7,6 @@
```
docs/
├── INDEX.md # 本文件(文档总索引)
├── SECURITY_AUDIT_20260801.md # 全局安全审计总览
├── security/ # 安全审计与加固相关
├── reviews/ # 代码审查日志
└── reports/ # 其他分析报告
@@ -29,7 +28,6 @@ docs/
| 文档 | 说明 |
|------|------|
| [SECURITY_AUDIT_20260801.md](security/SECURITY_AUDIT_20260801.md) | **全局总览** — 全模块安全审计汇总(问题等级定义、各模块问题汇总、修复跟踪) |
| [DEPLOY_SECURITY_AUDIT_20260801.md](security/DEPLOY_SECURITY_AUDIT_20260801.md) | 部署配置安全审计(docker-compose、systemd、防火墙、TLS 等) |
| [DEPS_SECURITY_AUDIT_20260801.md](security/DEPS_SECURITY_AUDIT_20260801.md) | 依赖安全审计(第三方库版本、已知漏洞、许可证) |
| [SECURITY_HARDENING_PLAN_20260803.md](security/SECURITY_HARDENING_PLAN_20260803.md) | 安全加固计划(内网通信安全加固开发计划) |
@@ -49,8 +47,8 @@ docs/
| smtp | [smtp_SECURITY_AUDIT_20260801.md](security/smtp_SECURITY_AUDIT_20260801.md) | Python 邮件服务(验证码发送/校验) |
| captcha | [captcha_SECURITY_AUDIT_20260801.md](security/captcha_SECURITY_AUDIT_20260801.md) | Python 验证码服务 |
| confs | [confs_SECURITY_AUDIT_20260801.md](security/confs_SECURITY_AUDIT_20260801.md) | Python 统一配置服务 |
| libhttp | [libhttp_SECURITY_AUDIT_20260803.md](security/libhttp_SECURITY_AUDIT_20260803.md) | C++ HTTP/WebSocket 库(请求/响应/WS/静态文件) |
| aibox | [aibox_SECURITY_AUDIT_20260801.md](security/aibox_SECURITY_AUDIT_20260801.md) | Python AI 智能体系统(WebSocket) |
| aibox (详版) | [aibox_SECURITY_AUDIT_full_20260801.md](security/aibox_SECURITY_AUDIT_full_20260801.md) | aibox 详细安全审计(含具体漏洞位置与修复建议) |
| BotOps | [BotOps_SECURITY_AUDIT_20260801.md](security/BotOps_SECURITY_AUDIT_20260801.md) | BotOps 运维控制面板 |
---
@@ -11,9 +11,9 @@
| 优先级 | 总数 | 已修复 | 部分修复 | 未修复 |
|--------|------|--------|----------|--------|
| P0 | 0 | 0 | 0 | 0 |
| P1 | 6 | 2 | 0 | 4 |
| P1 | 6 | 3 | 0 | 3 |
| P2 | 4 | 0 | 0 | 4 |
| **合计** | **10** | **4** | **0** | **6** |
| **合计** | **10** | **5** | **0** | **5** |
---
@@ -22,7 +22,7 @@
| # | 问题 | 状态 | 说明 |
|---|------|------|------|
| P1-1 | 请求体大小限制不可配置 + 超限无 413 | ✅ 已修复 | `Session::doRead`(`HttpServer_Request.cpp` L167-186)改用 `http::request_parser` + `body_limit()`(默认 10MB,与 WAF 对齐);`HttpServer` 新增 `setMaxBodySize()`/`getMaxBodySize()`(`HttpServer.hpp` L118-120)。超限时 `handleRead` 返回 **413 Payload Too Large** 并关闭连接(L256-265),而非静默断连 |
| P1-2 | HttpClient 响应无大小限制 | ❌ 未修复 | `HttpClient::write_callback`(`HttpClient.cpp` L329-334)把下游响应 `append` 进 `full_response` 无上限,超大响应可导致 OOM(`request_stream_sync` L63-81) |
| P1-2 | HttpClient 响应无大小限制 | ✅ 已修复 | `HttpClient.cpp` 新增 `WriteCtx`(流式回调 + 累计字节 + 上限),`write_callback` 超限返回 0 触发 `CURLE_WRITE_ERROR`,`request_stream_sync` 抛异常;`request_stream_sync` 改为**真流式**(callback 非空时每块即回调,不二次累积),`HttpClient` 新增 `set_max_response_size()`/`get_max_response_size()`(默认 10MB,对齐 WAF,0 表示不限制) |
| P1-3 | WebSocket 消息无大小限制 | ❌ 未修复 | `Session::onWsRead`(`HttpServer_WsSession.cpp` L88-111)把整条 WS 消息读入内存,依赖 Beast 默认 `read_message_max`(16MB)不可配置;超大消息可被恶意客户端推送导致 OOM |
| P1-4 | 服务器无读写超时(Slowloris) | ❌ 未修复 | 入站 `stream_`(`beast::tcp_stream`)**从未设置 `expires_after`**(仅代理后端设了超时)。攻击者建立连接后缓慢发送 HTTP 头,可无限占用连接与线程,Slowloris/连接耗尽(对应全局审计 P2-4) |
| P1-5 | 静态文件目录路径遍历 | ✅ 已修复 | 新增 `isSafeStaticPath()`(`HttpServer_Request.cpp` L139-156),静态文件服务前校验 target_path(拒绝 `..`、空字节、非 `/` 开头、URL 编码 `%2e`/`%2f`/`%5c`),逻辑与 gateway `is_safe_target()` 对齐(`HttpServer.cpp` L94) |
@@ -68,7 +68,7 @@
### 立即(24小时)
- [x] P1-1 请求体大小限制:`doRead` 改用 `http::request_parser` + `body_limit()`,`HttpServer` 增加 `setMaxBodySize()`(对齐 WAF 10MB),超限返回 413 而非断开 → ✅ 已实现(默认 10MB,`setMaxBodySize()` 可覆盖)
- [ ] P1-2 HttpClient 响应上限:`write_callback` 累计超过上限即失败,返回错误
- [x] P1-2 HttpClient 响应上限:`write_callback` 累计超过上限即失败,返回错误 → ✅ 已实现(`WriteCtx` 流式回调 + 上限,默认 10MB,`set_max_response_size()` 可覆盖)
- [ ] P1-4 服务器读超时:入站 `stream_` 设置 `expires_after`(首字节/头+体超时),防 Slowloris
### 短期(72小时)
+7
View File
@@ -16,6 +16,8 @@ C++20 network library for the UniServe microservice framework, providing unified
### HttpClient
- **HTTP Client**: Synchronous HTTP requests via libcurl (GET/POST/PUT/DELETE/PATCH/HEAD/OPTIONS)
- **Streaming Response**: `request_stream_sync` streams response body chunk-by-chunk to the callback (no full-body buffering)
- **Response Size Limit**: `set_max_response_size()` caps downstream response body (default 10MB, aligned with WAF; 0 = unlimited) to prevent OOM on oversized responses
- **WebSocket Client**: Synchronous WebSocket client via Boost.Beast
- **Dual Protocol**: Same client instance handles both HTTP and WebSocket
- **ws:// and wss://**: Supports both plain and TLS-encrypted WebSocket connections
@@ -221,6 +223,10 @@ void put_stream_sync(const std::string& url, const std::string& data, ResponseCa
void delete_stream_sync(const std::string& url, ResponseCallback cb);
void request_stream_sync(HttpMethod method, const std::string& url,
const std::string& data, ResponseCallback cb);
// Response size limit (防 OOM). Default 10MB (aligned with WAF); 0 = unlimited.
void set_max_response_size(size_t bytes);
size_t get_max_response_size() const;
```
**WebSocket Methods**
@@ -277,6 +283,7 @@ using WebSocketClientBase = HttpClient;
## Version History
- **v3.4.0**: HttpClient 响应大小限制(防 OOM)。`request_stream_sync` 改为真流式(按块回调,不二次缓冲);新增 `set_max_response_size()`/`get_max_response_size()`(默认 10MB,对齐 WAF,0 表示不限制),超限时 `write_callback` 返回 0 中止传输并抛异常。
- **v3.3.0**: Split HttpServer into 6 modular translation units by functional area (core/lifecycle, listener, routes, request dispatch, WS management, WS session/proxy) for dramatically faster incremental builds. Added internal detail header `inc/us/detail/HttpServerImpl.hpp` for shared definitions across TU boundaries.
- **v3.2.0**: Split into two shared libraries — `libus_http.so` (server only, no curl) and `libus_httpclient.so` (HttpClient, depends on curl). HttpClient implementation moved to `src/HttpClient.cpp`; `HttpClient.hpp` is declaration-only.
- **v3.1.0**: Added WebSocket reverse proxy (`ws_proxy()`) with bidirectional async bridging, ws:// and wss:// backend support, handshake-time authorization callback.
+7
View File
@@ -59,6 +59,12 @@ public:
void put_stream_sync(const std::string& url, const std::string& data, std::function<void(const std::string&)> callback);
void delete_stream_sync(const std::string& url, std::function<void(const std::string&)> callback);
// ===== 响应体大小限制(防 OOM)=====
// 设置下游响应体大小上限(字节)。0 表示不限制(默认 10MB,与 WAF 对齐)。
// 超限时 write_callback 返回 0 中止传输,request_stream_sync 抛异常。
void set_max_response_size(size_t bytes);
size_t get_max_response_size() const;
// ===== WebSocket Methods (Synchronous, Boost.Beast-based) =====
bool ws_connect(const std::string& url, int timeout_ms = 5000);
bool ws_connect(const std::string& host, const std::string& port,
@@ -77,6 +83,7 @@ private:
struct curl_slist* headers_ = nullptr;
std::map<std::string, std::string> response_headers_;
long response_code_ = 0;
size_t max_response_size_ = 10 * 1024 * 1024; // 默认 10MB,对齐 WAF;0 表示不限制
struct WsState {
asio::io_context ioc;
+45 -3
View File
@@ -4,6 +4,19 @@
namespace pmc::net {
namespace {
// write_callback 的用户态上下文:流式回调 + 响应大小上限
struct WriteCtx {
std::function<void(const std::string&)>* callback = nullptr; // 流式回调(非空时每块回调)
std::string* full_response = nullptr; // 退化累积(callback 为空时)
size_t received = 0; // 已接收字节数
size_t max = 0; // 响应上限(0 表示不限制)
bool overflow = false; // 是否超限
};
} // namespace
HttpClient::HttpClient() {
curl_ = curl_easy_init();
if (!curl_) {
@@ -21,6 +34,14 @@ HttpClient::~HttpClient() {
}
}
void HttpClient::set_max_response_size(size_t bytes) {
max_response_size_ = bytes;
}
size_t HttpClient::get_max_response_size() const {
return max_response_size_;
}
void HttpClient::set_header(const std::string& header) {
headers_ = curl_slist_append(headers_, header.c_str());
}
@@ -61,8 +82,12 @@ void HttpClient::request_stream_sync(
curl_easy_setopt(curl_, CURLOPT_HEADERDATA, this);
std::string full_response;
WriteCtx write_ctx;
write_ctx.callback = &callback;
write_ctx.full_response = &full_response;
write_ctx.max = max_response_size_;
curl_easy_setopt(curl_, CURLOPT_WRITEFUNCTION, write_callback);
curl_easy_setopt(curl_, CURLOPT_WRITEDATA, &full_response);
curl_easy_setopt(curl_, CURLOPT_WRITEDATA, &write_ctx);
curl_easy_setopt(curl_, CURLOPT_TIMEOUT, 30L);
curl_easy_setopt(curl_, CURLOPT_CONNECTTIMEOUT, 10L);
@@ -70,11 +95,16 @@ void HttpClient::request_stream_sync(
CURLcode res = curl_easy_perform(curl_);
if (res != CURLE_OK) {
if (res == CURLE_WRITE_ERROR && write_ctx.overflow) {
throw std::runtime_error("HTTP response exceeded max size ("
+ std::to_string(max_response_size_) + " bytes)");
}
throw std::runtime_error(std::string("CURL error: ") + curl_easy_strerror(res));
}
curl_easy_getinfo(curl_, CURLINFO_RESPONSE_CODE, &response_code_);
// 退化路径:调用方未传 callback 时,整块累积后一次性回调
if (!full_response.empty()) {
callback(full_response);
}
@@ -328,8 +358,20 @@ size_t HttpClient::header_callback(char* buffer, size_t size, size_t nitems, voi
size_t HttpClient::write_callback(char* ptr, size_t size, size_t nmemb, void* userdata) {
size_t total_size = size * nmemb;
std::string* response = static_cast<std::string*>(userdata);
response->append(ptr, total_size);
auto* ctx = static_cast<WriteCtx*>(userdata);
// 响应上限:超限返回 0,让 curl 以 CURLE_WRITE_ERROR 中止传输
if (ctx->max > 0 && ctx->received + total_size > ctx->max) {
ctx->overflow = true;
return 0;
}
ctx->received += total_size;
// 流式:callback 非空时每块即回调(不累积,避免二次内存)
if (ctx->callback && *ctx->callback) {
(*ctx->callback)(std::string(ptr, total_size));
} else if (ctx->full_response) {
// 退化:callback 为空时整块累积,供一次性回调
ctx->full_response->append(ptr, total_size);
}
return total_size;
}