Fix scheduling startup backlog catch-up

[sfmskywalker]

Summary

• Rebuild local scheduling triggers/bookmarks in configured pages instead of loading the entire startup backlog at once.
• Add a singleton past-due schedule staggerer so orphaned Delay/Timer/StartAt bookmark catch-up is distributed over a bounded window instead of all firing after 1ms.
• Add efficient EF count paths for paged trigger/bookmark reads and fix paged trigger cache keys.

Closes #7735.

Validation

• dotnet test test/unit/Elsa.Scheduling.UnitTests/Elsa.Scheduling.UnitTests.csproj --no-restore --framework net10.0
• dotnet test test/unit/Elsa.Http.UnitTests/Elsa.Http.UnitTests.csproj --no-restore --framework net10.0 --filter FullyQualifiedName~HttpWorkflowsMiddlewareTests
• dotnet build src/modules/Elsa.Persistence.EFCore/Elsa.Persistence.EFCore.csproj --no-restore --framework net10.0
• dotnet build src/modules/Elsa.Workflows.Runtime/Elsa.Workflows.Runtime.csproj --no-restore --framework net10.0
• git --no-pager diff --check

[greptile-apps]

Greptile Summary

This PR fixes startup scheduling backlog catch-up by paging through stored triggers and bookmarks (avoiding an unbounded in-memory load) and adding a singleton PastDueScheduleStaggerer that distributes already-due schedules evenly over a bounded window instead of firing everything with a 1 ms delay.

• Adds a new FindManyAsync(BookmarkFilter, PageArgs, CancellationToken) overload to IBookmarkStore and both EF Core store implementations, and pages both triggers and bookmarks in CreateSchedulesStartupTask.
• Introduces PastDueScheduleStaggerer and SchedulingOptions to configure the stagger window, interval, and minimum delay; the staggerer is registered as a singleton in both SchedulingFeature variants.
• Removes the early continue that was dropping past-due StartAt triggers in DefaultTriggerScheduler, letting them reach ScheduleAtAsync for catch-up; and fixes CachingTriggerStore to include pageArgs and order in its cache key.

Confidence Score: 5/5

Safe to merge; the core paging loop terminates correctly in all tested scenarios, the stagger math stays within the configured window, and the DefaultTriggerScheduler catch-up fix is validated by a new unit test.

All three changed execution paths (paged startup load, past-due stagger delay, StartAt catch-up scheduling) are covered by new unit tests that pass. The only observable risk is a stale TotalCount when rows are inserted between the COUNT and SELECT queries — a window that is essentially unreachable during startup; even if triggered, it would cause at most one extra unscheduled page that recovers on the next restart.

EFCoreBookmarkStore and EFCoreTriggerStore both use non-atomic COUNT + SELECT in the new paged overloads; worth revisiting if the startup task ever needs to run concurrently with active workflow writers.

Important Files Changed

Filename	Overview
src/modules/Elsa.Scheduling/Services/PastDueScheduleStaggerer.cs	New singleton staggerer; correctly subtracts minimumDelay from availableWindow so the maximum returned delay never exceeds PastDueScheduleStaggerWindow; uses Interlocked for thread-safe sequence numbering.
src/modules/Elsa.Scheduling/StartupTasks/CreateSchedulesStartupTask.cs	Switched from a single unbounded load to a paged loop; dual-break on empty page and nextOffset >= TotalCount covers both the common case and concurrent-deletion edge cases correctly.
src/modules/Elsa.Persistence.EFCore/Modules/Runtime/BookmarkStore.cs	New paged FindManyAsync issues a COUNT then a separate SELECT; the two queries are not wrapped in a snapshot transaction, so TotalCount can be stale under concurrent writers.
src/modules/Elsa.Persistence.EFCore/Modules/Runtime/TriggerStore.cs	Same non-atomic COUNT+SELECT pattern as BookmarkStore; also adds filter.TenantAgnostic to the paged overload, which was the missing fix for multi-tenant correctness.
src/modules/Elsa.Workflows.Runtime/Stores/CachingTriggerStore.cs	Cache keys for both paged FindManyAsync overloads now include pageArgs (and order), fixing the key collision bug that caused all pages to return the same cached first page.
src/modules/Elsa.Scheduling/ScheduledTasks/ScheduledSpecificInstantTask.cs	Wires PastDueScheduleStaggerer into the task; uses [ActivatorUtilitiesConstructor] on the full constructor so DI picks the right overload; the static default staggerer for the legacy 5-parameter constructor is intentional backward compat.
src/modules/Elsa.Scheduling/Services/DefaultTriggerScheduler.cs	Removes the continue that was silently dropping past-due StartAt triggers; they now flow through to ScheduleAtAsync for catch-up scheduling.
src/modules/Elsa.Workflows.Runtime/Contracts/IBookmarkStore.cs	Adds FindManyAsync(BookmarkFilter, PageArgs, CancellationToken) as a new abstract interface member; both in-box implementations are updated, but any third-party IBookmarkStore will fail to compile without the new method.
src/modules/Elsa.Workflows.Runtime/Stores/MemoryBookmarkStore.cs	Implements the new paged overload with in-memory double-scan (count then filter+page); acceptable for the in-memory/test store.
src/modules/Elsa.Scheduling/Options/SchedulingOptions.cs	New options class with sensible defaults; no-op Configure call in both SchedulingFeature variants ensures IOptions resolves even when callers don't call Configure.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant ST as CreateSchedulesStartupTask
    participant TS as ITriggerStore
    participant BS as IBookmarkStore
    participant TrSched as ITriggerScheduler
    participant SSIT as ScheduledSpecificInstantTask
    participant PDS as PastDueScheduleStaggerer

    ST->>TS: FindManyAsync(filter, page[0..N], ct)
    TS-->>ST: "Page<StoredTrigger>(items, totalCount)"
    ST->>TrSched: ScheduleAsync(page.Items)
    TrSched->>SSIT: new(task, startAt, ...)
    SSIT->>PDS: GetDelay(startAt - now)
    alt startAt in future
        PDS-->>SSIT: original positive delay
    else startAt in past (catch-up)
        PDS-->>SSIT: minimumDelay + staggerInterval x slot
    end
    SSIT-->>SSIT: arm timer with adjusted delay
    ST->>TS: FindManyAsync(filter, page[N..2N], ct)
    Note over ST,TS: loop until nextOffset >= totalCount or empty page
    ST->>BS: FindManyAsync(filter, page[0..N], ct)
    BS-->>ST: "Page<StoredBookmark>(items, totalCount)"
    Note over ST,BS: same paged loop for bookmarks

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant ST as CreateSchedulesStartupTask
    participant TS as ITriggerStore
    participant BS as IBookmarkStore
    participant TrSched as ITriggerScheduler
    participant SSIT as ScheduledSpecificInstantTask
    participant PDS as PastDueScheduleStaggerer

    ST->>TS: FindManyAsync(filter, page[0..N], ct)
    TS-->>ST: "Page<StoredTrigger>(items, totalCount)"
    ST->>TrSched: ScheduleAsync(page.Items)
    TrSched->>SSIT: new(task, startAt, ...)
    SSIT->>PDS: GetDelay(startAt - now)
    alt startAt in future
        PDS-->>SSIT: original positive delay
    else startAt in past (catch-up)
        PDS-->>SSIT: minimumDelay + staggerInterval x slot
    end
    SSIT-->>SSIT: arm timer with adjusted delay
    ST->>TS: FindManyAsync(filter, page[N..2N], ct)
    Note over ST,TS: loop until nextOffset >= totalCount or empty page
    ST->>BS: FindManyAsync(filter, page[0..N], ct)
    BS-->>ST: "Page<StoredBookmark>(items, totalCount)"
    Note over ST,BS: same paged loop for bookmarks

Loading

_{Reviews (3): Last reviewed commit: "address greptile review feedback (greplo..." | Re-trigger Greptile}

[sfmskywalker]

Retargeted this PR to release/3.8.0 by cherry-picking the scheduling startup backlog fix onto the release branch. The main-targeted copy is now PR #7747: #7747