feat: add maxParallel option to limit workspace build concurrency - #940
feat: add maxParallel option to limit workspace build concurrency#940jpzwarte wants to merge 5 commits into
maxParallel option to limit workspace build concurrency#940Conversation
Fixes rolldown#929 by introducing a `maxParallel` config option that limits how many workspace package builds can run concurrently. Without a limit, `buildWithConfigs` uses `Promise.all()` which starts all workspace builds simultaneously. When combined with `dts.tsgo: true`, each package spawns a `tsgo` subprocess, which can result in dozens of concurrent native processes and exhaust system resources. When `maxParallel` is set to a positive integer, at most that many builds run at the same time via a simple worker-pool implementation. When omitted, existing unbounded concurrent behavior is preserved. Changes: - Add `maxParallel?: number` to `UserConfig` and keep it optional in `ResolvedConfig` (added to `MarkPartial` keys) - Add `--max-parallel <number>` CLI flag; parse it as a number in the action handler so the string from argv is correctly coerced - Add `normalizeMaxParallel`, `resolveMaxParallel`, and `mapWithConcurrency` helpers in `src/build.ts` - Update `buildWithConfigs` to use the concurrency-limited pool when `maxParallel` is specified Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The per-option validation (positive integer check) belongs in options.ts alongside all other single-config validation, not in build.ts. build.ts still owns the cross-config resolveMaxParallel and the mapWithConcurrency utility since those operate on the already-resolved config array. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
maxParallel is a root-level option; workspace merging ensures all resolved configs have the same value. Reading configs[0].maxParallel is sufficient and simpler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ Deploy Preview for tsdown-main ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Hi! I'm I would like to apply some automated changes to this pull request, but it looks like I don't have the necessary permissions to do so. To get this pull request into a mergeable state, please do one of the following two things:
|
There was a problem hiding this comment.
Pull request overview
Adds a maxParallel configuration/CLI option to cap concurrent builds when running in workspace mode, preventing excessive parallel tsgo subprocesses that can exhaust system resources (Fixes #929).
Changes:
- Extend config schema with
maxParallel?: numberand keep it optional inResolvedConfig. - Add
--max-parallel <number>CLI flag. - Introduce a simple worker-pool (
mapWithConcurrency) and use it inbuildWithConfigswhenmaxParallelis set.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/config/types.ts | Adds maxParallel to UserConfig docs/types and keeps it optional in ResolvedConfig. |
| src/config/options.ts | Parses/validates maxParallel from user config during resolution. |
| src/cli.ts | Adds --max-parallel CLI option. |
| src/build.ts | Adds concurrency-limited mapping and uses it to limit config builds. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| const maxParallel = | ||
| maxParallelRaw == null ? undefined : Number(maxParallelRaw) | ||
|
|
| globalLogger.info('Build start') | ||
| const bundles = await Promise.all( | ||
| configs.map((options) => { | ||
| const isDualFormat = options.pkg | ||
| ? configChunksByPkg[options.pkg.packageJsonPath].formats.size > 1 | ||
| : true | ||
| return buildSingle( | ||
| options, | ||
| configDeps, | ||
| isDualFormat, | ||
| clean, | ||
| restart, | ||
| done, | ||
| ) | ||
| }), | ||
| ) | ||
| const { maxParallel } = configs[0] ?? {} | ||
| const buildConfig = (options: ResolvedConfig) => { | ||
| const isDualFormat = options.pkg | ||
| ? configChunksByPkg[options.pkg.packageJsonPath].formats.size > 1 | ||
| : true | ||
| return buildSingle(options, configDeps, isDualFormat, clean, restart, done) | ||
| } | ||
| const bundles = maxParallel | ||
| ? await mapWithConcurrency(configs, maxParallel, buildConfig) |
| const bundles = maxParallel | ||
| ? await mapWithConcurrency(configs, maxParallel, buildConfig) | ||
| : await Promise.all(configs.map(buildConfig)) |
|
any updates on this... |
|
@sxzz I haven't done anything further on this PR because nobody responded on Discord. Now that i see that you pushed main to it, do you want to review it? Should i continue work on this? |
|
Thanks for following up. I haven’t reviewed this PR because it is not currently in a reviewable state:
Please don’t continue polishing the current implementation as-is. If you would like to keep working on this, please start over with a simpler design and place Please also restore and complete the PR template, rebase the branch, and get CI green before requesting another review. Once those are addressed, feel free to ping me again. P.S. I’m not opposed to the use of AI. My point is that the PR author—not the AI—remains fully responsible for the contribution. This comment was also drafted with AI assistance, but I take full responsibility for it. |
cccf4ff to
e0266c1
Compare
|
Closing this PR in favor of #1030 |
Fixes #929.
Problem
When using
workspacemode withdts.tsgo: true,buildWithConfigsstarts all package builds concurrently viaPromise.all(). Each package spawns atsgosubprocess, so with 64 workspace packages you can end up with 64 concurrent native processes — exhausting CPU/memory and crashing macOS.Solution
Add a
maxParalleloption that limits how many workspace builds run at the same time. When set, a simple worker-pool (mapWithConcurrency) is used instead ofPromise.all(). When omitted, existing unbounded concurrent behavior is preserved.Changes
src/config/types.ts: AddmaxParallel?: numbertoUserConfig; add it to theMarkPartialkeys so it stays optional inResolvedConfigsrc/cli.ts: Add--max-parallel <number>CLI flag with numeric coercionsrc/build.ts: AddnormalizeMaxParallel,resolveMaxParallel, andmapWithConcurrencyhelpers; updatebuildWithConfigsto use the pool whenmaxParallelis setUsage
Or via CLI:
tsdown --workspace packages/* --max-parallel 4Verification
pnpm typecheck✅pnpm test run✅ (446 passed, 4 expected fail, 4 skipped)pnpm lint✅