feat: add PURGE directory mark expired #33

sendya · 2026-01-19T02:24:17Z

This pull request introduces several significant enhancements and refactorings to support directory-level cache push with mark-expired semantics, improve purge control, and add a path trie utility for efficient path matching. It also updates documentation to reflect new features and includes supporting code for marking cached directories as expired rather than immediately deleting them. The changes are grouped below by theme.

Directory Mark-Expired Push and Purge Enhancements

Introduces a new wrapper (storage/marked) around storage and buckets to support marking cached directories as expired (mark-expired) instead of immediate hard deletion. This is achieved by checking a path trie of purge tasks and updating object metadata to expire if matched. (storage/marked/wrap_storage.go, storage/marked/bucket.go, storage/marked/marked.go) [1] [2] [3]
Refactors the main application to wrap the storage layer with the new mark-expired logic and initializes the checker with the shared key-value store. (main.go) [1] [2] [3]
Updates the purge plugin to parse and handle new purge control header values, supporting both directory and file purges with mark-expired or hard delete modes. (plugin/purge/purge.go) [1] [2] [3] [4]

PathTrie Utility for Path Matching

Adds a generic PathTrie implementation for efficient path and prefix matching, supporting wildcards and regex, along with an example test. This is used by the mark-expired checker to track directory-level purge tasks. (pkg/pathtire/pathtire.go, pkg/pathtire/pathtire_example_test.go) [1] [2]

Documentation Updates

Updates both English and Chinese README.md files to show that directory mark-expired push and fuzzy refresh are now implemented. [1] [2]

Other Improvements and Refactoring

Refines the PurgeControl struct with a String() method for better logging and debugging, and improves import organization. (api/defined/v1/storage/storage.go) [1] [2]
Adjusts the native storage purge logic to properly handle mark-expired directory purges by skipping shared key-value index hits and falling back to a full scan. (storage/storage.go) [1] [2]
Minor logging and import cleanups. (storage/bucket/disk/disk.go)

References:
Directory mark-expired push: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]
PathTrie utility: [1] [2]
Documentation: [1] [2]
Other improvements:

…ctory purge enhancements

…nd update related logic

Copilot

Pull request overview

This pull request implements directory-level cache purge with mark-expired semantics, adding the ability to mark cached directories as expired rather than immediately deleting them. The implementation introduces a new PathTrie data structure for efficient path matching and wraps the storage layer to intercept lookup operations and mark matching objects as expired.

Changes:

Introduces a PathTrie utility for efficient path prefix matching with wildcard and regex support
Adds a storage wrapper layer that intercepts lookups and marks objects as expired based on purge tasks stored in the trie
Updates the purge plugin to parse and handle mark-expired vs hard-delete modes for both files and directories
Updates documentation to reflect that directory mark-expired push and fuzzy refresh are now implemented

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
pkg/pathtire/pathtire.go	New PathTrie implementation for efficient path/prefix matching
pkg/pathtire/pathtire_example_test.go	Example test demonstrating PathTrie usage
storage/marked/wrap_storage.go	Storage wrapper that integrates mark-expired logic
storage/marked/marked.go	Checker implementation using PathTrie and SharedKV
storage/marked/bucket.go	Bucket wrapper that marks objects as expired during lookup
main.go	Integration of marked storage wrapper into application initialization
storage/storage.go	Skip SharedKV index for mark-expired directory purges
plugin/purge/purge.go	Parse purge control headers and default to mark-expired mode
api/defined/v1/storage/storage.go	Add String() method to PurgeControl for logging
storage/bucket/disk/disk.go	Minor formatting cleanup
README.md	Update feature checklist to show DIR mark-expired as implemented
README.zh-CN.md	Update Chinese feature checklist to show DIR mark-expired as implemented

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-19T02:29:06Z

storage/marked/marked.go

+func NewSharedKVChecker(kv storagev1.SharedKV, opts ...SharedKVOption) *checker {
+	c := &checker{
+		KV:        kv,
+		pathtire:  pathtire.NewPathTrie[string, int64](),


Function call has a spelling error. Should be "pathtrie.NewPathTrie" (not "pathtire.NewPathTrie").

Copilot · 2026-01-19T02:29:07Z

storage/marked/bucket.go

+	// Mark expired by setting ExpiresAt into the past and store back.
+	md.ExpiresAt = time.Now().Add(-1 * time.Second).Unix()
+	// _ = b.base.Store(ctx, md)


Commented-out code should either be removed or implemented. The Store call at line 40 is commented out, which means metadata changes are not being persisted back to storage. This could be intentional for performance reasons, but needs clarification or TODO comment explaining why.

Suggested change

// Mark expired by setting ExpiresAt into the past and store back.

md.ExpiresAt = time.Now().Add(-1 * time.Second).Unix()

// _ = b.base.Store(ctx, md)

// Mark as expired for this lookup by setting ExpiresAt into the past.

// NOTE: We intentionally do not persist this change via Store here to avoid

// additional writes during Lookup; callers should treat the returned metadata

// as expired.

md.ExpiresAt = time.Now().Add(-1 * time.Second).Unix()

Copilot · 2026-01-19T02:29:07Z

plugin/purge/purge.go

+	ctrl := storagev1.PurgeControl{
+		MarkExpired: true, // 默认用过期标记
+	}
+
+	if len(param) >= 1 {
+		ctrl.Dir = param[0] == "dir"
+	}
+
+	// 配置推送模式 hard / mark_expired
+	if len(param) == 2 {
+		hasHard := param[1] == "hard"
+
+		ctrl.Hard = hasHard
+		ctrl.MarkExpired = !hasHard
+	}
+
+	// 没有配置推送选项


The default behavior sets MarkExpired to true even when no header is provided. This changes the default purge behavior from hard delete to mark-expired, which could be a breaking change for existing deployments. Consider whether this default is intentional and if it should be documented or made configurable.

Suggested change

ctrl := storagev1.PurgeControl{

MarkExpired: true, // 默认用过期标记

}

if len(param) >= 1 {

ctrl.Dir = param[0] == "dir"

}

// 配置推送模式 hard / mark_expired

if len(param) == 2 {

hasHard := param[1] == "hard"

ctrl.Hard = hasHard

ctrl.MarkExpired = !hasHard

}

// 没有配置推送选项

// Do not change default purge behavior when no header is provided.

// Only set Hard / MarkExpired when explicitly configured via the header.

ctrl := storagev1.PurgeControl{}

if len(param) >= 1 {

ctrl.Dir = param[0] == "dir"

}

// 配置推送模式 hard / mark_expired

if len(param) >= 2 {

switch param[1] {

case "hard":

ctrl.Hard = true

ctrl.MarkExpired = false

case "mark_expired":

ctrl.MarkExpired = true

ctrl.Hard = false

}

}

// 没有配置推送选项时，保持 storagev1.PurgeControl 的零值，让底层保持原有默认行为

Copilot · 2026-01-19T02:29:07Z

pkg/pathtire/pathtire_example_test.go

+
+func ExampleNewPathTrie() {
+
+	trie := pathtire.NewPathTrie[string, int64]()


Variable name has a spelling error. Should be "pathtrie.NewPathTrie" (not "pathtire.NewPathTrie").

Copilot · 2026-01-19T02:29:08Z

storage/marked/marked.go

+func (c *checker) TireAdd(ctx context.Context, storePath string) {
+	unix := time.Now().Unix()
+	c.pathtire.Insert(storePath, unix)
+	log.Infof("purge add pathtire %s, drop-time %d", storePath, unix)
+}


Method naming issue. "TireAdd" should be renamed to "TrieAdd" or a more descriptive name like "AddPath" to match Go naming conventions and correct spelling.

Suggested change

func (c *checker) TireAdd(ctx context.Context, storePath string) {

unix := time.Now().Unix()

c.pathtire.Insert(storePath, unix)

log.Infof("purge add pathtire %s, drop-time %d", storePath, unix)

}

// TrieAdd adds the given storePath to the path trie with the current Unix timestamp.

func (c *checker) TrieAdd(ctx context.Context, storePath string) {

unix := time.Now().Unix()

c.pathtire.Insert(storePath, unix)

log.Infof("purge add pathtire %s, drop-time %d", storePath, unix)

}

// TireAdd is kept for backward compatibility; use TrieAdd instead.

func (c *checker) TireAdd(ctx context.Context, storePath string) {

c.TrieAdd(ctx, storePath)

}

Copilot · 2026-01-19T02:29:10Z

pkg/pathtire/pathtire.go

@@ -0,0 +1,214 @@
+package pathtire


Package name has a spelling error. The correct spelling should be "pathtrie" (not "pathtire"). A trie is a tree-like data structure, whereas "tire" refers to a wheel component. This typo affects the package name, which should be corrected throughout the codebase.

Suggested change

package pathtire

package pathtrie

Copilot · 2026-01-19T02:29:11Z

pkg/pathtire/pathtire_example_test.go

@@ -0,0 +1,25 @@
+package pathtire_test


Package name has a spelling error. The correct spelling should be "pathtrie_test" (not "pathtire_test"). This should match the corrected package name.

Suggested change

package pathtire_test

package pathtrie_test

Copilot · 2026-01-19T02:29:11Z

pkg/pathtire/pathtire_example_test.go

+	"github.com/omalloc/tavern/pkg/pathtire"
+)
+
+func ExampleNewPathTrie() {
+
+	trie := pathtire.NewPathTrie[string, int64]()


Import path has a spelling error. Should be "github.com/omalloc/tavern/pkg/pathtrie" (not "pathtire").

Suggested change

"github.com/omalloc/tavern/pkg/pathtire"

)

func ExampleNewPathTrie() {

trie := pathtire.NewPathTrie[string, int64]()

"github.com/omalloc/tavern/pkg/pathtrie"

)

func ExampleNewPathTrie() {

trie := pathtrie.NewPathTrie[string, int64]()

Copilot · 2026-01-19T02:29:11Z

storage/marked/marked.go

+	storagev1 "github.com/omalloc/tavern/api/defined/v1/storage"
+	"github.com/omalloc/tavern/api/defined/v1/storage/object"
+	"github.com/omalloc/tavern/contrib/log"
+	"github.com/omalloc/tavern/pkg/pathtire"


Import path has a spelling error. Should be "github.com/omalloc/tavern/pkg/pathtrie" (not "pathtire").

Copilot · 2026-01-19T02:29:11Z

storage/marked/marked.go

+// If AutoClear is true, the key is deleted after a hit.
+type checker struct {
+	KV        storagev1.SharedKV
+	pathtire  *pathtire.PathTrie[string, int64]


Variable type has a spelling error. Should be "pathtrie.PathTrie" (not "pathtire.PathTrie").

…nd update related tests

sendya added 2 commits January 15, 2026 20:20

feat(storage): implement marked storage with push-mark logic and dire…

281df2b

…ctory purge enhancements

feat(pathtire): enhance PathTrie to support generic key-value types a…

c0f2fd0

…nd update related logic

Copilot AI review requested due to automatic review settings January 19, 2026 02:24

Copilot started reviewing on behalf of sendya January 19, 2026 02:24 View session

Copilot AI reviewed Jan 19, 2026

View reviewed changes

sendya added 3 commits January 19, 2026 10:49

fix(pathtrie): pathtrie typo

cbd2dd9

feat(purge): save purge dir task to sharedkv and add design doc

fda26c6

feat(e2e): enhance DiscardBody function to support read speed limit a…

c8113da

…nd update related tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add PURGE directory mark expired #33

feat: add PURGE directory mark expired #33

Uh oh!

sendya commented Jan 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Copilot AI Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		func ExampleNewPathTrie() {

		trie := pathtire.NewPathTrie[string, int64]()

feat: add PURGE directory mark expired #33

Are you sure you want to change the base?

feat: add PURGE directory mark expired #33

Uh oh!

Conversation

sendya commented Jan 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants