Skip to content

Make sure we clear the logger observer before starting a new session#1387

Open
ryanzhang22 wants to merge 2 commits into
pytorch:mainfrom
ryanzhang22:export-D103278391
Open

Make sure we clear the logger observer before starting a new session#1387
ryanzhang22 wants to merge 2 commits into
pytorch:mainfrom
ryanzhang22:export-D103278391

Conversation

@ryanzhang22
Copy link
Copy Markdown
Contributor

Summary:
Currently, we don't reset the UST logger between sync (auto-trace)/async (on-demand) runs, so the data from one run can pollute the other. In particular, if we have an async then sync run, the sync run's UST logs will have artifacts from the async run.

Before sync/async's configure method, we add a resetLoggerObservers call. We add it here because configure is when we know the request was accepted.

Differential Revision: D103278391

…rch#1383)

Summary:

Currently, we mark all areas where we want to indicate stage completion with `UST_LOGGER_MARK_COMPLETED`. This macro routes us to the UST logger `write` method which keep a running list of all LOG(ERROR) messages. If no ERROR messages have been logged, we'll mark the stage as a success. Otherwise, we will include the error message in the Scuba write.

With this setup it's pretty easy for us to miss an exit path, especially in the post-processing path where we have a bunch of early exit branches. This change introduces a `USTLoggerStageGuard` to help automatically manage stage exit without needing to explicitly remember to mark `UST_LOGGER_MARK_COMPLETED`.

Note that this doesn't help us capture cases where something failed catastrophically and the process immediately exits. Many of these callsites were already pretty well instrumented, this just makes stuff a little cleaner.

Differential Revision: D102829439
Summary:
Currently, we don't reset the UST logger between sync (auto-trace)/async (on-demand) runs, so the data from one run can pollute the other. In particular, if we have an async then sync run, the sync run's UST logs will have artifacts from the async run.

Before sync/async's `configure` method, we add a `resetLoggerObservers` call. We add it here because `configure` is when we know the request was accepted.

Differential Revision: D103278391
@meta-cla meta-cla Bot added the cla signed label May 1, 2026
@meta-codesync
Copy link
Copy Markdown

meta-codesync Bot commented May 1, 2026

@ryanzhang22 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D103278391.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants