feat(helpers): streaming bulk-ingest with bounded memory

[margaretjgu]

Replaces the buffered readFileSync + splitIntoBatches approach with line-by-line streaming bounded to flush_bytes * concurrency bytes, so ingesting large files no longer requires loading the full dataset into memory. Concurrency is controlled with a semaphore that provides backpressure. NDJSON, JSON arrays, and CSV are all supported.

Closes #443

[github-actions]

❌MegaLinter analysis: Error

Descriptor	Linter	Files	Fixed	Errors	Warnings	Elapsed time
✅ COPYPASTE	jscpd	yes		no	no	0.62s
✅ REPOSITORY	gitleaks	yes		no	no	35.79s
✅ REPOSITORY	git_diff	yes		no	no	0.29s
✅ REPOSITORY	secretlint	yes		no	no	17.48s
✅ REPOSITORY	trivy	yes		no	no	10.75s
❌ TYPESCRIPT	eslint	2		1	0	2.36s

Detailed Issues

❌ TYPESCRIPT / eslint - 1 error

test/es/helpers/bulk-ingest.test.ts
  92:42  error  Empty block statement  no-empty

✖ 1 problem (1 error, 0 warnings)

Notices

📣 MegaLinter 9.5.0 is out! Discover the new features and security recommendations in the release announcement. (Skip this info by defining SECURITY_SUGGESTIONS: false)

See detailed reports in MegaLinter artifacts
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security
Show us your support by starring ⭐ the repository