Skip to content

feat: 启动任务接口添加帖子/视频数量与评论数量覆盖支持#900

Open
zanmeipaul wants to merge 3 commits into
NanmiCoder:mainfrom
zanmeipaul:main
Open

feat: 启动任务接口添加帖子/视频数量与评论数量覆盖支持#900
zanmeipaul wants to merge 3 commits into
NanmiCoder:mainfrom
zanmeipaul:main

Conversation

@zanmeipaul
Copy link
Copy Markdown

Background / 背景

目前在 WebUI 或通过 API 启动爬虫任务时,最大爬取帖子/视频数 (CRAWLER_MAX_NOTES_COUNT) 和每篇帖子最大评论数 (CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES) 只能使用静态配置文件中的默认值。若想在启动不同任务时采用不同的数量限制,必须修改本地配置文件,这对 API 集中化调度和多任务动态配置不够友好。

Changes / 变更内容

  1. API Schema 扩展 (api/schemas/crawler.py)
    • CrawlerStartRequest 结构体中添加了 max_notes_countmax_comments_count 两个可选整型字段。
  2. 命令行支持 (cmd_arg/arg.py)
    • 为命令行新增了 --crawler_max_notes_count 选项以支持覆盖 config.CRAWLER_MAX_NOTES_COUNT,原有 --max_comments_count_singlenotes 保持不变。
  3. 爬虫管理器透传 (api/services/crawler_manager.py)
    • 修改子进程命令构建逻辑。若接口中传入了限制数量,则动态拼装对应的命令行参数传给爬虫子进程;若未传,则继续沿用默认配置文件。
  4. 单元与集成测试 (tests/test_api_limits.py)
    • 编写了对应新特性的测试用例,覆盖了参数解析、命令行生成和 WebAPI 调用的测试,已验证全部通过。

Test / 验证情况

  • 运行测试套件:pytest tests/test_api_limits.pypytest tests/test_cmd_arg_tieba.py 均顺利通过。
  • 调用启动 API 验证:不携带参数时成功降级使用默认配置;携带参数时子进程参数拼接正确。

@zanmeipaul zanmeipaul requested a review from NanmiCoder as a code owner May 19, 2026 13:08
@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. enhancement New feature or request labels May 19, 2026
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant