Skip to content

Bump default tokenBatchSize from 1 to 4#25

Merged
corasan merged 1 commit into
mainfrom
t3code/71d91bb5
Apr 23, 2026
Merged

Bump default tokenBatchSize from 1 to 4#25
corasan merged 1 commit into
mainfrom
t3code/71d91bb5

Conversation

@corasan
Copy link
Copy Markdown
Owner

@corasan corasan commented Apr 23, 2026

Summary

  • Raise the default tokenBatchSize from 1 to 4 in HybridLLM so streaming emits ~4× fewer TokenEvents, JSON encodes, and JS bridge crossings out of the box, with negligible chunking at typical on-device generation speeds.
  • Update all three default sites (property initializer, load() reset, unload() reset) plus the fallback when options.tokenBatchSize is omitted.

Test plan

  • Stream a long response in the example app and confirm typewriter UX still feels smooth
  • Verify explicit tokenBatchSize: 1 still produces per-token events
  • Verify explicit tokenBatchSize: 8 still produces 8-token batches

@corasan corasan changed the title {"title":"Parallel model downloads, CI, and higher default token batch","body":"## Summary\n- Stream byte-level download progress using bounded parallel fetches (#23), with a ProgressAggregator actor and KVO on countOfBytesReceived.\n- Add GitHub Actions CI running Biome lint, TypeScript typecheck, and Bun tests on push/PR (#24).\n- Update Biome to 2.4.12, exclude generated/native folders, and reformat touched TS files.\n- Bump default tokenBatchSize from 1 to 4 in HybridLLM for faster generation out of the box.\n- Throttle download-modal re-renders to visible progress changes (>=0.5%) to avoid UI thrash.\n\n## Testing\n- bun typescript — Not run\n- bun test — Not run\n- bunx biome ci . — Not run\n- iOS model download end-to-end with progress callback — Not run"} Bump default tokenBatchSize from 1 to 4 Apr 23, 2026
@corasan corasan merged commit ab69079 into main Apr 23, 2026
3 checks passed
@corasan corasan deleted the t3code/71d91bb5 branch April 23, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant