feat(confs): integrate dnfs service with centralized config management
- Add dnfs.ini to confs/configs/ with server, database, ai_review, confs sections - Refactor dnfs.py to use three-layer fail-safe config loading (defaults → confs → env vars) - Unify timeouts for DB RPC and AI review calls via configuration - Add _merge_ini() whitelist merging to prevent unexpected config injection - Environment variables have highest priority for Docker networking (db/ai_review hostnames) - Document config loading mechanism in both confs/README.md and dnfs/README.md Follows the same pattern as static service integration; confs failure falls back gracefully to built-in defaults + environment variables.
This commit is contained in:
qing committed
1 parent
b4436b192a
commit
4e190953c0
4 files changed
+140
-31
No files matched your search
+9
-2
@@ -236,6 +236,7 @@ except RpcError as e:
|
||||
| 服务 | 配置文件 | 接入方式 |
|
||||
|------|---------|---------|
|
||||
| [static](../static/) | [static.ini](configs/static.ini) | 三层 fail-safe：内置默认值 → confs（优先）→ 本地 `config.ini` 回退 |
|
||||
| [dnfs](../dnfs/) | [dnfs.ini](configs/dnfs.ini) | 三层 fail-safe：内置默认值 → confs（优先）→ 环境变量覆盖 |
|
||||
|
||||
static 的接入实现见 [static/fsys.py](../static/fsys.py)：
|
||||
- `_fetch_from_confs()` 通过 `RpcClient` 调 `get` 路由取 `static.ini`，3 秒超时
|
||||
@@ -243,7 +244,12 @@ static 的接入实现见 [static/fsys.py](../static/fsys.py)：
|
||||
- `_merge_ini()` 仅覆盖已知的 section/key（白名单合并，不引入未预期字段）
|
||||
- 配置来源会记录到日志（`logs.info/warning`），便于排查实际用了哪一层
|
||||
|
||||
其他服务（gateway / waf / ai_review / smtp / dnfs / db）尚未接入，仍读各自本地配置。
|
||||
dnfs 的接入实现见 [dnfs/dnfs.py](../dnfs/dnfs.py)：
|
||||
- 三层加载顺序：内置默认值 → confs（dnfs.ini）→ 环境变量覆盖（优先级最高）
|
||||
- 支持的环境变量：`LISTEN_ADDR`/`LISTEN_PORT`/`DB_HOST`/`DB_PORT`/`REVIEW_URL`/`CONFS_HOST`/`CONFS_PORT`
|
||||
- Docker 环境通过环境变量传入服务名（db/ai_review），覆盖 confs 中的 127.0.0.1 默认值
|
||||
|
||||
其他服务（gateway / waf / ai_review / smtp / db）尚未接入，仍读各自本地配置。
|
||||
|
||||
## 项目结构
|
||||
|
||||
@@ -253,7 +259,8 @@ confs/
|
||||
├── log_client.py # 统一日志客户端 SDK（与其他服务一致的副本）
|
||||
├── configs/ # 配置文件存放目录（被分发对象）
|
||||
│ ├── example.ini # 示例配置
|
||||
│ └── static.ini # static 服务配置（已接入分发）
|
||||
│ ├── static.ini # static 服务配置（已接入分发）
|
||||
│ └── dnfs.ini # dnfs 服务配置（已接入分发）
|
||||
├── docs/
|
||||
│ └── SECURITY_AUDIT_20260801.md # 安全审计报告
|
||||
├── _set.sh # 初始化脚本（建 venv + 安装依赖）
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
[server]
|
||||
host = 127.0.0.1
|
||||
port = 8006
|
||||
|
||||
[database]
|
||||
host = 127.0.0.1
|
||||
port = 8005
|
||||
timeout = 10
|
||||
|
||||
[ai_review]
|
||||
url = http://127.0.0.1:8007/review
|
||||
timeout = 10
|
||||
|
||||
[confs]
|
||||
host = 127.0.0.1
|
||||
port = 8010
|
||||
+33
-7
@@ -196,15 +196,41 @@ curl -X POST http://localhost:8006/update_note \
|
||||
|
||||
## 配置
|
||||
|
||||
服务地址与上游依赖硬编码在 [dnfs.py](dnfs.py)：
|
||||
dnfs 已接入 confs 统一配置服务，采用**三层 fail-safe** 加载模式（与 static 服务一致）：
|
||||
|
||||
| 常量 | 值 | 说明 |
|
||||
|------|----|------|
|
||||
| `ADDR` / `PORT` | `127.0.0.1` / `8006` | 监听地址 |
|
||||
| `DB_SERVER_URL` | `http://127.0.0.1:8005/execute` | db 服务 |
|
||||
| `REVIEW_SERVER_URL` | `http://127.0.0.1:8007/review` | ai_review 服务 |
|
||||
**加载顺序**：内置默认值 → confs 统一配置服务（dnfs.ini，优先）→ 环境变量覆盖（最高优先级）
|
||||
|
||||
调用超时均为 10 秒。
|
||||
### 配置文件
|
||||
|
||||
集中托管在 [confs/configs/dnfs.ini](../confs/configs/dnfs.ini)，由 confs 服务分发：
|
||||
|
||||
| Section | Key | 默认值 | 说明 |
|
||||
|---------|-----|--------|------|
|
||||
| `[server]` | `host` | `127.0.0.1` | 监听地址 |
|
||||
| `[server]` | `port` | `8006` | 监听端口 |
|
||||
| `[database]` | `host` | `127.0.0.1` | db 服务地址 |
|
||||
| `[database]` | `port` | `8005` | db 服务端口 |
|
||||
| `[database]` | `timeout` | `10` | RPC 调用超时（秒） |
|
||||
| `[ai_review]` | `url` | `http://127.0.0.1:8007/review` | AI 审核接口地址 |
|
||||
| `[ai_review]` | `timeout` | `10` | AI 审核请求超时（秒） |
|
||||
| `[confs]` | `host` | `127.0.0.1` | confs 服务地址 |
|
||||
| `[confs]` | `port` | `8010` | confs 服务端口 |
|
||||
|
||||
### 环境变量（Docker 环境必需，优先级最高）
|
||||
|
||||
环境变量覆盖配置文件值，容器化部署时通过 docker-compose.yml 传入服务名：
|
||||
|
||||
| 变量 | 容器内值 | 说明 |
|
||||
|------|---------|------|
|
||||
| `LISTEN_ADDR` | `0.0.0.0` | 监听地址（容器内需绑定所有网卡） |
|
||||
| `LISTEN_PORT` | `8006` | 监听端口 |
|
||||
| `DB_HOST` | `db` | db 服务主机名（docker-compose 服务名） |
|
||||
| `DB_PORT` | `8005` | db 服务端口 |
|
||||
| `REVIEW_URL` | `http://ai_review:8007/review` | ai_review 服务地址 |
|
||||
| `CONFS_HOST` | - | 可选，confs 容器化后使用 |
|
||||
| `CONFS_PORT` | - | 可选，confs 容器化后使用 |
|
||||
|
||||
配置来源会记录到日志（`logs.info/warning`），便于排查实际用了哪一层。本地裸机运行时 confs 不可用会自动回退到内置默认值 + 环境变量。
|
||||
|
||||
## 前端
|
||||
|
||||
|
||||
+82
-22
@@ -1,9 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
'''
|
||||
|
||||
Dendrachy Network File-System (dnfs)
|
||||
@@ -21,8 +17,10 @@ from flask_cors import CORS
|
||||
#from flask_sqlalchemy import SQLAlchemy # 使用pymsg模块进行通信
|
||||
from waitress import serve
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import base64
|
||||
import configparser
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from typing import Dict, Any
|
||||
@@ -32,14 +30,75 @@ from log_client import logs
|
||||
# pymsg 导入
|
||||
# 我们有在依赖列表中加入pymsg
|
||||
# 这个可以省略也许
|
||||
import sys as _sys
|
||||
_sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
||||
from pymsg import RpcClient, RpcError # noqa: E402
|
||||
|
||||
# 从环境变量中获取参数
|
||||
# 随后我们专门做一个统一配置微服务
|
||||
_DBV2_ADDR = os.environ.get('DB_HOST', '127.0.0.1')
|
||||
_DBV2_PORT = int(os.environ.get('DB_PORT', '8005'))
|
||||
# ==================== 配置加载（三层 fail-safe） ====================
|
||||
# 加载顺序：内置默认值 → confs 统一配置服务（dnfs.ini） → 环境变量覆盖
|
||||
# 环境变量优先级最高（Docker 环境使用服务名 db/ai_review 而非 127.0.0.1）
|
||||
|
||||
_dnfs_config = {
|
||||
'server': {'host': '127.0.0.1', 'port': '8006'},
|
||||
'database': {'host': '127.0.0.1', 'port': '8005', 'timeout': '10'},
|
||||
'ai_review': {'url': 'http://127.0.0.1:8007/review', 'timeout': '10'},
|
||||
'confs': {'host': '127.0.0.1', 'port': '8010'},
|
||||
}
|
||||
|
||||
|
||||
def _merge_ini(text):
|
||||
"""把 ini 文本合并进 _dnfs_config（仅覆盖已知的 section/key，白名单合并）。"""
|
||||
ini = configparser.ConfigParser(interpolation=None)
|
||||
ini.read_string(text)
|
||||
for section in _dnfs_config:
|
||||
if ini.has_section(section):
|
||||
for key, value in ini[section].items():
|
||||
if key in _dnfs_config[section]:
|
||||
_dnfs_config[section][key] = value
|
||||
|
||||
|
||||
def _fetch_from_confs():
|
||||
"""从 confs 统一配置服务拉取 dnfs.ini 全文，失败返回 None（fail-safe）。"""
|
||||
try:
|
||||
host = os.environ.get('CONFS_HOST', _dnfs_config['confs']['host'])
|
||||
port = int(os.environ.get('CONFS_PORT', _dnfs_config['confs']['port']))
|
||||
data = RpcClient(host, port, 'dnfs', timeout=3).call(
|
||||
'get', {'name': 'dnfs.ini'})
|
||||
return data.get('content')
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
_confs_text = _fetch_from_confs()
|
||||
if _confs_text is not None:
|
||||
_merge_ini(_confs_text)
|
||||
logs.info("dnfs", "配置来源: confs 统一配置服务 (dnfs.ini)")
|
||||
else:
|
||||
logs.warning("dnfs", "confs 不可用，使用内置默认值 + 环境变量")
|
||||
|
||||
# 环境变量覆盖（优先级最高，Docker 环境需要）
|
||||
if os.environ.get('LISTEN_ADDR'):
|
||||
_dnfs_config['server']['host'] = os.environ['LISTEN_ADDR']
|
||||
if os.environ.get('LISTEN_PORT'):
|
||||
_dnfs_config['server']['port'] = os.environ['LISTEN_PORT']
|
||||
if os.environ.get('DB_HOST'):
|
||||
_dnfs_config['database']['host'] = os.environ['DB_HOST']
|
||||
if os.environ.get('DB_PORT'):
|
||||
_dnfs_config['database']['port'] = os.environ['DB_PORT']
|
||||
if os.environ.get('REVIEW_URL'):
|
||||
_dnfs_config['ai_review']['url'] = os.environ['REVIEW_URL']
|
||||
if os.environ.get('CONFS_HOST'):
|
||||
_dnfs_config['confs']['host'] = os.environ['CONFS_HOST']
|
||||
if os.environ.get('CONFS_PORT'):
|
||||
_dnfs_config['confs']['port'] = os.environ['CONFS_PORT']
|
||||
|
||||
# 提取配置常量
|
||||
_DBV2_ADDR = _dnfs_config['database']['host']
|
||||
_DBV2_PORT = int(_dnfs_config['database']['port'])
|
||||
_DB_TIMEOUT = int(_dnfs_config['database']['timeout'])
|
||||
_REVIEW_SERVER_URL = _dnfs_config['ai_review']['url']
|
||||
_REVIEW_TIMEOUT = int(_dnfs_config['ai_review']['timeout'])
|
||||
_ADDR = _dnfs_config['server']['host']
|
||||
_PORT = int(_dnfs_config['server']['port'])
|
||||
|
||||
# 管理员判定已上移至 gateway：登录时命中 [Admin] users 白名单即在 Token payload
|
||||
# 写入 is_admin=true，本服务通过 info.get('is_admin') 读取，不再本地维护名单。
|
||||
@@ -48,13 +107,15 @@ _DBV2_PORT = int(os.environ.get('DB_PORT', '8005'))
|
||||
_rpc = RpcClient(_DBV2_ADDR, _DBV2_PORT, "dnfs")
|
||||
|
||||
|
||||
def _call_dbv2(payload: dict, timeout: int = 10) -> dict:
|
||||
def _call_dbv2(payload: dict, timeout: int = None) -> dict:
|
||||
"""通过 pymsg RPC 调用 dbv2 数据库服务（自动注入 ``_route: "execute"``）。
|
||||
|
||||
返回完整响应 dict（``{"success": ..., "data"/"error": ...}``），
|
||||
与历史行为一致，调用方无需改动。服务端业务错误返回 success=False；
|
||||
网络层故障抛 ConnectionError。
|
||||
"""
|
||||
if timeout is None:
|
||||
timeout = _DB_TIMEOUT
|
||||
try:
|
||||
data = _rpc.call("execute", payload, timeout=timeout)
|
||||
return {"success": True, "data": data}
|
||||
@@ -65,15 +126,14 @@ def _call_dbv2(payload: dict, timeout: int = 10) -> dict:
|
||||
f"dbv2 调用失败 ({_DBV2_ADDR}:{_DBV2_PORT}): {e}"
|
||||
) from None
|
||||
|
||||
# AI审核服务地址，从环境变量获取
|
||||
REVIEW_SERVER_URL = os.environ.get('REVIEW_URL', 'http://127.0.0.1:8007/review')
|
||||
def review_content(content: str, timeout: int = 10) -> Dict[str, Any]:
|
||||
# AI审核服务地址
|
||||
def review_content(content: str, timeout: int = None) -> Dict[str, Any]:
|
||||
"""
|
||||
调用 AI 内容审核服务的 /review 接口，对给定文本进行审核。
|
||||
|
||||
Args:
|
||||
content: 待审核的文本内容，不可为空。
|
||||
timeout: 请求超时时间（秒），默认 10 秒。
|
||||
timeout: 请求超时时间（秒），默认使用配置值。
|
||||
|
||||
Returns:
|
||||
字典，包含接口返回的字段，例如：
|
||||
@@ -83,7 +143,9 @@ def review_content(content: str, timeout: int = 10) -> Dict[str, Any]:
|
||||
- 请求参数错误时：{"error": "content is required"} 等
|
||||
- 网络或异常时：{"error": "request_failed", "detail": "..."}
|
||||
"""
|
||||
url = REVIEW_SERVER_URL
|
||||
if timeout is None:
|
||||
timeout = _REVIEW_TIMEOUT
|
||||
url = _REVIEW_SERVER_URL
|
||||
payload = {"content": content}
|
||||
json_data = json.dumps(payload).encode("utf-8")
|
||||
|
||||
@@ -192,7 +254,7 @@ def add_note():
|
||||
|
||||
|
||||
# 内容审查
|
||||
review = review_content(title + '\n' + content, 10)
|
||||
review = review_content(title + '\n' + content)
|
||||
if True != review: raise DnfsExp('内容审核不通过，用户'+info.get('user')+'，从'+request.remote_addr, 400)
|
||||
|
||||
# 执行插入
|
||||
@@ -412,7 +474,7 @@ def update_note():
|
||||
return jsonify({"error": 'It\'s not your note'}), 403
|
||||
|
||||
# 内容审查
|
||||
review = review_content(title + '\n' + content, 10)
|
||||
review = review_content(title + '\n' + content)
|
||||
if True != review:
|
||||
#return jsonify({'error': 'review failed', 'data': review}), 400
|
||||
logs.warning('dnfs', f'update_note 内容审核未通过 id={id} title={title}', extra={'user': info['user']})
|
||||
@@ -502,9 +564,7 @@ def get_note():
|
||||
# """返回应用主页"""
|
||||
# return render_template('index.html')
|
||||
|
||||
ADDR = os.environ.get('LISTEN_ADDR', '0.0.0.0')
|
||||
PORT = int(os.environ.get('LISTEN_PORT', '8006'))
|
||||
if __name__ == "__main__":
|
||||
logs.info('dnfs', f'服务启动，监听 http://{ADDR}:{PORT}')
|
||||
print(f'Listen at http://{ADDR}:{PORT}')
|
||||
serve(app, host=ADDR, port=PORT)
|
||||
logs.info('dnfs', f'服务启动，监听 http://{_ADDR}:{_PORT}（db={_DBV2_ADDR}:{_DBV2_PORT}, ai_review={_REVIEW_SERVER_URL}）')
|
||||
print(f'Listen at http://{_ADDR}:{_PORT}')
|
||||
serve(app, host=_ADDR, port=_PORT)
|
||||
Reference in new issue
Block a user