feat(chunk upload): Support uploading by chunks on drivers/local#9472
Draft
thetapilla wants to merge 5 commits intoAlistGo:mainfrom
Draft
feat(chunk upload): Support uploading by chunks on drivers/local#9472thetapilla wants to merge 5 commits intoAlistGo:mainfrom
thetapilla wants to merge 5 commits intoAlistGo:mainfrom
Conversation
add upload_chunk_size_mb metadata for Local driver and validate 0-4096 in Local.Init Refs: N/A
add init append complete cancel flow with strict failure cleanup and stale reaper Refs: N/A
expose policy init chunk complete cancel handlers with permission checks and error mapping Refs: N/A
initialize reaper after task manager startup to clean stale sessions and part files Refs: N/A
Feature/chunk upload backend
Collaborator
|
感谢这个 PR,整体实现比较完整,尤其是上传会话生命周期(init/chunk/complete/cancel)、失败清理和后台回收这块考虑得比较周全 👍 不过目前仓库里已经有一个相关方向的实现:PR #9457,它是在 driver 层引入 chunker 能力,目标是把“分片能力”做成通用存储抽象。 对比下来,这个 PR(PR #9472)主要是在 上传协议 / API 层实现分块上传,并且当前逻辑是绑定 Local 存储,没有抽象到 driver capability。 从长期架构上看,我们更倾向于:
换句话说,这个 PR 中的上传链路(策略协商、会话管理、清理机制等)是有价值的,但更适合建立在 chunker driver 之上,而不是单独形成一套 Local-only 的实现。 因此这边的建议是:
如果你这边有意愿继续推进这一方向,也很欢迎一起把 driver 与上传链路这两层整合成一个更通用的方案。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Partial implementation to #5480
Summary
灵感来自cloudflare的上传文件大小限制导致报413。
本PR为仅适配本地存储的分块上传实现,补齐从用户到服务器端接收的上传链路,并兼容原上传协议:
upload_chunk_size_mb(范围 0-4096,0 表示关闭分块上传)。说明:本PR描述覆盖前后端整体方案;前端代码通过 companion PR 提交。
feat(chunk upload): Support uploading by chunks on drivers/local alist-web#295
Changes
后端(alist)
Local 存储配置项与参数校验
drivers/local/meta.go: 增加upload_chunk_size_mb配置元信息。drivers/local/driver.go: 在Init中增加 0-4096 范围校验。分块上传核心生命周期
internal/fs/chunk_upload.go:GetUploadPolicy(当前按 Local 存储配置返回分块策略)。InitChunkUpload、AppendChunk、CompleteChunkUpload、CancelChunkUpload。StartChunkUploadReaper定时清理过期会话与临时分块文件。API 与路由接线
server/handles/fsup_chunk.go: 新增 5 个 handler。server/router.go: 新增/api/fs/upload/policy|init|chunk|complete|cancel路由。cmd/server.go: 启动时挂接fs.StartChunkUploadReaper()。前端(alist-web,companion PR)
上传流程接入
src/pages/home/uploads/chunk.ts: 新增策略驱动分块上传器,失败时 best-effort cancel。src/pages/home/uploads/Upload.tsx: 接入uploadWithChunkPolicy(..., fallbackUploader)。Local 配置项展示与校验
src/pages/manage/storages/Item.tsx: 增加upload_chunk_size_mb字段输入约束与提示。src/pages/manage/storages/AddOrEdit.tsx: 保存前校验,拒绝非整数或超出 0-4096 的值。src/lang/en/drivers.json、src/lang/en/storages.json: 增补文案与错误提示。当前边界
Testing
自动化测试
已运行(开发过程中多次执行,均为 exit code 0):
联调与行为验证
/ping、/api/public/settings可访问)。upload_chunk_size_mb在保存时拒绝负数与大于 4096 的值。0与4096可保存。PR #9457 关联性
重合度评估
兼容性判断
Improvements
在想怎么能往边收块边喂给driver流这方面去靠,比如给driver一个PipeReader什么的,还在想。。
那时候说不定就能打通其他的driver了