Skip to content

[codex] Truncate inference rank logs with tee#2817

Draft
samsja wants to merge 1 commit into
mainfrom
codex/truncate-inference-rank-logs
Draft

[codex] Truncate inference rank logs with tee#2817
samsja wants to merge 1 commit into
mainfrom
codex/truncate-inference-rank-logs

Conversation

@samsja

@samsja samsja commented Jun 14, 2026

Copy link
Copy Markdown
Member

Summary

Replaces the startup rm -f logs/inference/*.log cleanup with writer-side truncation for rank-specific inference logs.

Why

The node-level inference log already matches trainer/orchestrator behavior: it gets a startup line through truncating tee, then appends server output during the same run. The auxiliary node_*_rank*.log files did not have that initial truncating write and only used tee -a, so they could retain content when reusing an output directory.

Impact

Validation

  • git diff --check -- src/prime_rl/templates/_launch_rank.sh.j2 src/prime_rl/templates/inference.sbatch.j2 src/prime_rl/templates/multi_node_rl.sbatch.j2
  • Loaded the edited Jinja templates with jinja2.Environment(...).get_template(...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant