Skip to content

Add bounded TaskQueue admission, cancellation and shutdown for 2.x - #23

Draft
huntharo wants to merge 3 commits into
mainfrom
feat/2x-event-task-queue
Draft

huntharo wants to merge 3 commits into
mainfrom
feat/2x-event-task-queue

Conversation

@huntharo

@huntharo huntharo commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Thanks for maintaining p-map-iterable. I use it in PwrGit and PwrSnap to bound background I/O, and in PwrAgnt to stream concurrent peer results. Reviewing those applications showed repeated completion/cancellation ticket code around the legacy queue and an admission gap for non-awaiting event callbacks.

Merge prerequisites: maintenance PR #19 and the release workflow/tooling changes have landed; the updated 1.1.x release / releases/1.1 handoff remains a prerequisite before any 2.x merge. This draft targets main for design review and must not be merged before that handoff. The branch includes current main at de8a2b3, preserving aliases from #16, the pinned pnpm 12.7.0 toolchain and shared CI setup from #26, and the validation/security changes from #28/#27. The pnpm migration conflicts are resolved: package-lock.json stays removed, development/example commands use pnpm, and the Mermaid dev dependencies are added to the single-document v9 pnpm lock without changing main's existing locked versions or install policy. Its diff now contains only 10 intended API/docs files; main's generated-docs exclusion is retained. TaskQueue's implementation and the Awaited<T> result correction are preserved. Package/module integration is handled by #20. Proposed iterator/cancellation corrections in #22 remain separate and are not merged into this branch; the parent will validate the combined stack locally.

BlockingQueue.enqueue() inserts its item before awaiting a reader. Three unawaited submissions with maxUnread: 1 therefore retain three items. Concurrency remains controlled, but a burst of UI/IPC events can create an unbounded input backlog. IterableQueueMapperSimple.onIdle() also permanently calls done(), and task failures are collected in an unbounded array rather than delivered to the submitting handler.

This adds TaskQueue with lazy operation admission, finite independent limits (concurrency: 4, maxPending: 8 by default), per-task outcomes and explicit overload. Fulfilled outcomes use Awaited<T>, and add<T> returns Promise<Awaited<T>>, matching native promise assimilation even for explicitly promise-valued and nested promise-like task types. Existing iterable exports remain available, and the existing Simple options type is exported from the package root.

Contract Resulting behavior
Admission Synchronous reservation; at most concurrency + maxPending unfinished tasks. Full/closed admission has an explicit error. User code starts in a microtask.
Event completion submit() returns a cancellable handle with an always-resolving fulfilled/rejected outcome; ignored failed submissions cannot create unhandled task rejections.
Request completion add() returns the value or rejects with the original reason, including admission errors. Its promise must be observed.
Backpressure onCapacity() waits without retaining task payloads; concurrent producers retry if a contender takes the capacity.
Cancellation Waiting tasks are removed immediately. Running abort signals the operation and retains its slot until actual settlement.
Ordering Surviving tasks start FIFO; results settle in completion order. Concurrency 1 serializes operations.
Lifecycle onIdle() remains reusable, drain() snapshots accepted unfinished work, and close() immediately ends admission before draining. Repeated close calls can escalate cancellation.

The scheduler is separate from the legacy iterable runner because removable pending tasks, reusable idle and per-task settlement cannot safely be layered over its current input/result buffering and terminal accounting. add() is implemented on submit(); there is one new scheduler. Queue-owned task count is bounded, while caller-owned payloads/results, capacity waiter count and separate queues remain application resource policy. Running operations that ignore abort can succeed or delay shutdown indefinitely. Drain/close confirm settlement rather than aggregate success.

DESIGN-2.md records API choices, migration, cancellation/shutdown semantics and the assessment of #16/#17. TaskQueue implements #17's intended independent serial backlog; forwarding queue depth to maxUnread does not establish that input bound. Main now includes #16's ConcurrentMapper, MappingQueue, WorkerQueue and option aliases, which remain exported and documented alongside TaskQueue. The earlier experiment assessment records the originally inspected heads; this workstream did not close or merge #15/#16/#17.

REVIEW-2.md prioritizes reproduced legacy terminal hangs, undefined value/error loss, missing source-generator cleanup and operational limits. #22 proposes those mapper corrections and external cancellation at c25aaef, based on the same 0784c65 maintenance ancestor. Mapper failure releases blocked/future producers without a result consumer and preserves exact reasons, including undefined; the parent reports 94 tests and 25 lifecycle regressions passing for that separate PR. Its guaranteed third mapper signal remains assignable to existing two-argument callbacks, but directly invoking a Mapper-typed function requires the third argument when that change lands. Iterator return releases consumers and signals callbacks without awaiting uncooperative work; TaskQueue close waits for actual settlement. These companion semantics are documented as proposed, not implemented by this PR's legacy source.

The README includes three Mermaid diagrams, runnable examples, explicit lifecycle/error guidance and four verified public application source links pinned to commits. No application code was edited. The authorized docs follow-up excludes generated docs from TypeScript inputs and adds dev-only Mermaid rendering to build:docs. The scoped plugin's official repository documents support for current TypeDoc; its peers accept TypeDoc 0.28.20 and Mermaid 11.17.2. The original unscoped 1.12.0 plugin targets older TypeDoc APIs. Mermaid assets are copied locally so diagrams do not depend on a CDN. Packaging #20 must retain empty runtime and peer dependency sets when combining these dev-only additions.

Validation Observed result
pnpm 12.7.0 frozen install on Node 24.21.0 Passed with main's seven-day release-age policy; Mermaid dev dependencies retained in the v9 lock, existing main dependency versions unchanged
Final maintenance versions ESLint 10.11.0, globals 17.12.0, @types/node 26.6.3 and typescript-eslint 8.70.1 retained
Node 24.21.0, TS 6.0.3 / ESLint 10 / Jest 30 Build and repository lint passed; 9 suites / 141 tests passed
TaskQueue regression suite 35 tests; 100% statements/lines/functions and 97.87% branches
Promise-valued result type regressions Two cases cover Promise and nested PromiseLike task types; the original declarations failed four compile-time number/result assignments, corrected declarations and runtime results pass
Strict Node subprocess Ignored synchronous/asynchronous task failures and cancellation drained without unhandled rejection
Resource-limit stress 1,000 simultaneous admission attempts respect the exact bound; 1,000 queued cancellations preserve capacity without retained queue tombstones
Node 22.22.1 compatibility smoke All 35 queue tests, strict subprocess, both examples, docs generation and subsequent build passed
Runnable examples Event burst: 4 accepted, 2 dropped, 1 cancelled; producer: 12 admitted, 11 ordered writes, 1 handled failure
Current-main merge resolution Alias tests and semantic-aliases example pass; repeated docs generation/build pass with main's exclusion; TaskQueue implementation, queue tests and root exports are byte-identical to the prior API head 1a1160f
TypeDoc rendering in an actual browser Three diagrams in both themes = six SVGs, unique IDs, complete viewboxes, visible dimensions, zero browser exceptions; external network requests blocked
Repeat docs generation followed by build Passed; zero generated-doc files among the compiler's 405 inputs
Documentation diagnostics Zero errors and one existing unexported legacy Errors-type warning
Full dependency audit Zero vulnerabilities

The remaining TypeDoc warning concerns Simple's existing error-array type. Native AggregateError, the final Node/ESM/CommonJS support policy and runtime/peer dependency removal are owned by the package/release workstream. TaskQueue introduces no runtime dependency or aggregate-error use.

Rendered TypeDoc admission/concurrency diagram

This screenshot is from generated TypeDoc HTML after browser-side SVG rendering, with external requests blocked. All three diagrams were verified in light and dark themes.

Rendered TypeDoc admission diagram

Reproducible validation commands
pnpm install --frozen-lockfile
pnpm run lint
pnpm run build
pnpm run test --runInBand
pnpm run build:docs
pnpm run build:docs
pnpm run build
pnpm exec tsc --listFilesOnly
pnpm run example:semantic-aliases
pnpm exec ts-node -r tsconfig-paths/register examples/task-queue.ts
pnpm exec ts-node -r tsconfig-paths/register examples/task-queue-producer.ts
pnpm exec jest src/task-queue.test.ts --runInBand --coverage=false
pnpm audit

Full tests ran on Node 24.21.0 with the pinned pnpm 12.7.0 toolchain. All 35 TaskQueue tests also passed on Node 22.22.1 after the pnpm migration; the earlier Node 22 smoke included examples and docs/build. The strict subprocess is part of the queue suite. A headless Chrome check waited for all six complete SVGs, switched themes and measured visible diagrams while blocking nonlocal network requests; parsing alone was not accepted as rendering evidence. Public URLs were checked through GitHub's contents API and public pages. Legacy reproductions and proposed #22 fixes are recorded in REVIEW-2.md.

Parent combined validation integrates packaging b176585, API/docs 77cec70, iterator fixes 023f0dc, and maintenance 0784c65/a116155 locally. All 130 tests pass on Node 24; all 58 TaskQueue/lifecycle tests pass on Node 22. The real-tarball consumer matrix passes 72 configurations: 18 each with TypeScript 6 on Node 22/24/26, plus 18 with TypeScript 5.9 on Node 24. Each run installs both CJS and ESM apps from the same 21-file archive, verifies matching lockfile integrity, and checks all ten shared runtime constructors and bidirectional class types, including Node-only typings without DOM. Build, lint, examples, docs, 13 release-metadata checks, and npm audit pass; runtime and peer dependencies remain empty. Docs retain one existing warning for the legacy Simple Errors type.

The maintenance PRs' final checks pass. Publication of 1.1.3 and the subsequent releases/1.1 branch cut remain pending the required approving review or explicit authorization for admin merge. The 2.x PRs remain draft.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

📊 Node.js Code Coverage Report

File Lines Lines Hit / Found Uncovered Lines Branches
src/blocking-queue.ts ✅ 100.0% 227 / 227 0 100.0%
src/index.ts ✅ 100.0% 43 / 43 0 100.0%
src/iterable-mapper.ts ✅ 100.0% 564 / 564 0 100.0%
src/iterable-queue-mapper-simple.ts ✅ 100.0% 162 / 162 0 100.0%
src/iterable-queue-mapper.ts ✅ 100.0% 110 / 110 0 100.0%
src/iterable-queue.ts ✅ 100.0% 43 / 43 0 100.0%
src/queue.ts ✅ 100.0% 35 / 35 0 100.0%
src/task-queue.ts ✅ 100.0% 271 / 271 0 97.9%
Overall ✅ 100.0% 1455 / 1455 0 99.2%

@huntharo
huntharo force-pushed the feat/2x-event-task-queue branch 2 times, most recently from 50400f6 to 77cec70 Compare October 4, 2026 17:37
@huntharo
huntharo force-pushed the feat/2x-event-task-queue branch 4 times, most recently from 168ef53 to 923fc23 Compare October 5, 2026 19:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant