Skip to content

fix(BA-3522): Prevent race condition in kernel metrics cleanup #8250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HyeockJinKim merged 4 commits into from
Jan 28, 2026
Merged

fix(BA-3522): Prevent race condition in kernel metrics cleanup #8250

HyeockJinKim merged 4 commits into from
Jan 28, 2026

Conversation

@fregataa
Copy link
Member

@fregataa fregataa commented Jan 23, 2026 •
edited
Loading

resolves #7536 (BA-3522)

  • Remove automatic cleanup of unused kernel metrics from collect_container_stat() to prevent race conditions where remove_kernel_metric() could fail when kernel info was already removed.
  • Add warning log when attempting to remove metrics for an unknown kernel.

Checklist: (if applicable)

  • Milestone metadata specifying the target backport version
  • Mention to the original issue
  • Installer updates including:
    • Fixtures for db schema changes
    • New mandatory config options
  • Update of end-to-end CLI integration tests in ai.backend.test
  • API server-client counterparts (e.g., manager API -> client SDK)
  • Test case(s) to:
    • Demonstrate the difference of before/after
    • Demonstrate the flow of abstract/conceptual models with a concrete implementation
  • Documentation
    • Contents in the docs directory
    • docstrings in public interfaces and type annotations

@fregataa fregataa added this to the 25.15 milestone Jan 23, 2026
@fregataa fregataa requested a review from HyeockJinKim January 23, 2026 09:42
@fregataa fregataa self-assigned this Jan 23, 2026
Copilot AI review requested due to automatic review settings January 23, 2026 09:42
@github-actions github-actions bot added size:XS ~10 LoC comp:agent Related to Agent component labels Jan 23, 2026
@fregataa fregataa requested a review from jopemachine January 23, 2026 09:43
Copilot AI reviewed Jan 23, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@fregataa fregataa marked this pull request as draft January 23, 2026 10:10
@github-actions github-actions bot added size:S 10~30 LoC and removed size:XS ~10 LoC labels Jan 23, 2026
@fregataa fregataa marked this pull request as ready for review January 23, 2026 10:19
HyeockJinKim
HyeockJinKim reviewed Jan 26, 2026
View reviewed changes
Copy link
Collaborator

@HyeockJinKim HyeockJinKim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand—if this task isn't included, how does it break the behavior?

@fregataa
fix(BA-3522): Prevent race condition in kernel metrics cleanup
4ad5500 
Remove automatic cleanup of unused kernel metrics from
collect_container_stat() to prevent race conditions where
remove_kernel_metric() could fail when kernel info was already
removed. Add warning log when attempting to remove metrics for
an unknown kernel.
@fregataa
add news fragment
a5fcf4f
@fregataa
cleanup kernel_metrics
b36a338
@fregataa
fix comment
673b73a
@HyeockJinKim HyeockJinKim added this pull request to the merge queue Jan 28, 2026
Merged via the queue into main with commit 3a53f92 Jan 28, 2026
30 checks passed
@HyeockJinKim HyeockJinKim deleted the fix/BA-3522 branch January 28, 2026 05:38
lablup-octodog pushed a commit that referenced this pull request Jan 28, 2026
@fregataa
fix(BA-3522): Prevent race condition in kernel metrics cleanup (#8250)
614f1ed 
Backported-from: main (26.1)
Backported-to: 26.1
Backport-of: 8250
@lablup-octodog lablup-octodog mentioned this pull request Jan 28, 2026
Merged
lablup-octodog pushed a commit that referenced this pull request Jan 28, 2026
@fregataa
fix(BA-3522): Prevent race condition in kernel metrics cleanup (#8250)
0f439f2 
Backported-from: main (26.1)
Backported-to: 25.15
Backport-of: 8250
@lablup-octodog lablup-octodog mentioned this pull request Jan 28, 2026
Merged
github-merge-queue bot pushed a commit that referenced this pull request Jan 28, 2026
@lablup-octodog @fregataa
fix(BA-3522): Prevent race condition in kernel metrics cleanup (#8250) (
8fa6197 
#8398)

Co-authored-by: Sanghun Lee <[email protected]>
fregataa added a commit that referenced this pull request Jan 28, 2026
@lablup-octodog @fregataa
fix(BA-3522): Prevent race condition in kernel metrics cleanup (#8250) (
1976411 
#8397)

Co-authored-by: Sanghun Lee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@HyeockJinKim HyeockJinKim HyeockJinKim approved these changes

@jopemachine jopemachine Awaiting requested review from jopemachine

Assignees

@fregataa fregataa

Labels

comp:agent Related to Agent component size:S 10~30 LoC

Projects

None yet

Milestone

25.15

Development

Successfully merging this pull request may close these issues.

Fix immediate timing issue in kernel metrics cleanup

3 participants

@fregataa @HyeockJinKim