Repository navigation
Conversation
TGFSDirectory.find_files/find_dirs scanned the full entry list on every name lookup. A depth-1 WebDAV PROPFIND resolves every child individually (Folder.member -> Resource -> Ops.stat_file -> DirectoryApi.get_fr -> find_files([name])), so listing a directory with N children cost O(N^2) entry visits and made wide directories unresponsive. Add a lazily-built name -> positions index per TGFSDirectory, validated by an O(1) freshness fingerprint (entry count + identity of the last entry) so direct list mutations (e.g. the GitHub metadata loader) cannot read a stale index. No public signature or serialization change. Measured: N=2000 children 1356.5ms -> 10.7ms (~127x); N=8000 children 23739.4ms -> 24.6ms (~965x); scaling changes from quadratic to linear.
|
Hello The owner of this project is inactive for the last months. we (the community) have moved on to a fork of this project, maintained by a community member. you can find it here: https://github.com/Xyvran/tgfs we are still looking for more contributors, so your help is very welcome there :) the fork also has a lot of more features and stuff, so chances are things you are trying to fix here are already present in the fork. Greetings |
|
https://github.com/VulcanoSoftware/dcfs i also forked xyvran's fork and tried to replace the telegram backend with a discord backend. not because i don't like the telegram backend. i actually like the telegram backend more than the discord backend. but just for redundancy. when my dcfs project is production ready, my goal is to merge it into xyvran's tgfs to add raid 1 like redundancy by combining telegram and discord. I am also looking for contributors :) |
Location, JSON envelope, freshness rule (SHA-pin vs branch/tag), write trigger, and GithubDirectory reconstruction requirement for the cache introduced in ccbcbf0/e364437/554afd4.
|
@Palgogo |
Summary
Fixes two production-scale TGFS failure modes that appear during large metadata recovery:
PROPFIND Depth: 1directory listings so a wide collection can start responding before all children are consumed.Safety / behaviour
Validation
TGFS_DATA_DIR=. TGFS_CONFIG_FILE=config-test.yaml .venv/bin/pytest tests/tgfs/core/repository/impl/metadata/test_github_repo.py::TestGitTreeLoader tests/test_asgidav/test_propfind_streaming.py -qgit diff --check upstream/master...HEADcleanScope
This PR intentionally does not change production deployment, Compose configuration, rclone configuration, or the separate local-first/SQLite metadata work.