Add support for oneCCL and Synchronization events. by tsocha · Pull Request #1335 · pytorch/kineto

tsocha · 2026-03-30T11:57:22Z

Add communication and synchronization profiling support — new activity handlers for collective communication events and device synchronization events
new XPU_SYNC ActivityType
Add integration tests — test suite for the new communication and synchronization handlers with proper MSVC test discovery via gtest_add_tests
Harden error handling — log warnings instead of throwing when PTI_VIEW_COMMUNICATION is unsupported
Fix Windows/MSVC build issues — fix XpuptiProfilerTestLib linking, ExternalProject Debug/Release mismatch, general MSVC compilation fixes
correct XPUPTI_BUILD_FLAG logging in CMake

Commits marked with 🤖 are made with usage of gen AI.

ptiViewGetApiIdName requires valid _api_group and _api_id values. Use PTI_API_GROUP_LEVELZERO (1) with zeEventHostSynchronize_id (84) in test records to avoid PTI_ERROR_BAD_ARGUMENT.

gtest_discover_tests runs the test executable at build time to enumerate tests. With the MSVC generator, the oneAPI DLLs are not in PATH, causing STATUS_DLL_NOT_FOUND (0xc0000135). gtest_add_tests parses the source at configure time instead, avoiding the runtime dependency.

…pported PTI does not support communication events on Windows. Replace XPUPTI_CALL with LOG(WARNING) for PTI_VIEW_COMMUNICATION enable and disable to allow the session to be constructed gracefully.

Separate PTI_VIEW_DEVICE_SYNCHRONIZATION from COLLECTIVE_COMM into its own XPU_SYNC activity type, as device synchronization is unrelated to collective communication.

Add FORCE:UNRESOLVED and WINDOWS_EXPORT_ALL_SYMBOLS for Windows DLL builds, and use platform-aware import library path instead of hardcoded .so extension. With multi-config generators (Visual Studio), CMAKE_BUILD_TYPE is empty at configure time. The inner ExternalProject (Ninja, single-config) was receiving an empty build type and defaulting to Release, causing _ITERATOR_DEBUG_LEVEL and RuntimeLibrary linker mismatches when building Debug. - Use custom CONFIGURE_COMMAND with $<CONFIG> generator expression to forward the active configuration to the inner Ninja build - Simplify find_library PATH_SUFFIXES in compute/CMakeLists.txt to prioritize CMAKE_BUILD_TYPE-matching libraries

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

- Fix invalid JSON in synchronization metadata: use addMetadataQuoted for string (Type) and pointer (handle) fields - Increment gpuOpCount for GPU barrier synchronization types - Guard communication tests with PTI_VERSION_AT_LEAST(0, 17) - Fix ExternalProject_Add for single/multi-config generators on Linux/Windows

Co-authored-by: Slawomir Siwek <slawomir.siwek@intel.com>

tsocha · 2026-03-30T11:58:35Z

@EikanWang, @gujinghui please review.

gujinghui · 2026-04-09T01:20:41Z

-  PRIVATEUSE1_RUNTIME = 24, // host side privateUse1 runtime events
-  PRIVATEUSE1_DRIVER = 25, // host side privateUse1 driver events
+  PRIVATEUSE1_RUNTIME = 25, // host side privateUse1 runtime events
+  PRIVATEUSE1_DRIVER = 26, // host side privateUse1 driver events


Do not change the existing values in the list. Let's add the new item before the ENUM_COUNT only.

I see, I will fix it.

gujinghui · 2026-04-09T01:27:54Z

        XPUPTI_CALL(ptiViewEnable(PTI_VIEW_COLLECTION_OVERHEAD));
        break;

+#if PTI_VERSION_AT_LEAST(0, 17)


Can we follow the similar design in #1174 , instead of introducing many #if here? @moksiuc is able to give more details.

Could you be more precise here?
This change is consistent with this function: example in line 233.

This PR have 7 hits with #if PTI_VERSION_AT_LEAST(0, 17):

2x ptiViewEnable and ptiViewDisable. It's consistent within the function handling PTI versions.

3x in event handlers .cpp and .h for XpuptiActivityProfilerSession::handleCommunicationActivity.
I could move .cpp part to another file but it's a single function.

2x in tests.

I think that I can't use @moksiuc design here.
Most of these #ifs are here because it's the new feature in PTI.

@moksiuc could you confirm that?

At first glance I don't see how new code could be extracted to separate files or classes except separate Handlers file.

The --export-dynamic linker flag is Linux-specific and causes build failures on Windows. Guard it with a platform check to only apply on non-Windows systems. On Windows, symbol exporting is handled differently via DLL exports and doesn't require this linker flag.

…m-oneccl

gujinghui · 2026-05-06T02:33:49Z

    set(XPUPTI_INCLUDE_DIR ${XPUPTI_INCLUDE_DIR} PARENT_SCOPE)

-    set(XPUPTI_BUILD_FLAG "-DHAS_XPUPTI" PARENT_SCOPE)
+    set(XPUPTI_BUILD_FLAG "-DHAS_XPUPTI")


@tsocha with this PR #1337 , do we still need to add the flag here?

#1337 Remove only part of HAS_XPUPTI usage.
My change fix the logging of this variable.
Currently it always produce empty string in the logs.

gujinghui · 2026-05-06T02:43:13Z

@tsocha @aostrowski-hbn
This PR is too large and not easy to review. Let's split the PR into 3 separate ones,

Code cleaning-up in libkineto/test/xpupti/CMakeLists.txt and libkineto/src/plugin/xpupti/XpuptiActivityHandlers.cpp
Enable the ActivityType::XPU_SYNC
Enable the ActivityType::COLLECTIVE_COMM

tsocha and others added 15 commits March 30, 2026 13:40

Fix logging of XPUPTI_BUILD_FLAG

33f1850

Add support for collective comunication events.

8fd8eff

Add support for synchronization events

f4b4c93

🤖 Add integration tests for communication and synchronization handlers

4b7d3de

ptiViewGetApiIdName requires valid _api_group and _api_id values. Use PTI_API_GROUP_LEVELZERO (1) with zeEventHostSynchronize_id (84) in test records to avoid PTI_ERROR_BAD_ARGUMENT.

🤖 Log warning instead of throwing when PTI_VIEW_COMMUNICATION is unsu…

6871698

…pported PTI does not support communication events on Windows. Replace XPUPTI_CALL with LOG(WARNING) for PTI_VIEW_COMMUNICATION enable and disable to allow the session to be constructed gracefully.

🤖 Add XPU_SYNC ActivityType for device synchronization events

4e12909

Separate PTI_VIEW_DEVICE_SYNCHRONIZATION from COLLECTIVE_COMM into its own XPU_SYNC activity type, as device synchronization is unrelated to collective communication.

Fix compilation on MSVC

4cde5a1

Apply suggestions from 🤖 code review

38efe31

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Self-review I

ff15033

AI Review II

7d36e18

remove else as clang-tidy suggests.

52d795a

Co-authored-by: Slawomir Siwek <slawomir.siwek@intel.com>

Add information about env flag for the user

057ef30

meta-cla Bot added the cla signed label Mar 30, 2026

Merge branch 'pytorch:main' into dev/tsocha/oneccl

8d173de

gujinghui reviewed Apr 9, 2026

View reviewed changes

tsocha added 4 commits April 9, 2026 11:22

Merge remote-tracking branch 'origin/main' into upstream-oneccl

7ce741e

Don't change order of Activity types

274a559

Merge remote-tracking branch 'staging/dev/tsocha/oneccl' into upstrea…

7aee399

…m-oneccl

tsocha requested a review from gujinghui April 9, 2026 09:44

tsocha added 4 commits April 9, 2026 11:55

Remove duplicated list

66ea51c

Merge branch 'pytorch:main' into dev/tsocha/oneccl

1037218

Fix order of activities

e9c88e4

Merge branch 'pytorch:main' into dev/tsocha/oneccl

3bbe1ac

gujinghui reviewed May 6, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into upstream-oneccl

014433b

This was referenced May 11, 2026

Fix Windows build support for xpupti tests and #1394

Open

Add XPU synchronization events to XPUPTI profiler #1395

Open

Add XCCL collective communication activity tracing to XPU plugin #1396

Open

tsocha requested a review from gujinghui May 11, 2026 13:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for oneCCL and Synchronization events.#1335

Add support for oneCCL and Synchronization events.#1335
tsocha wants to merge 25 commits into
pytorch:mainfrom
intel-staging:dev/tsocha/oneccl

tsocha commented Mar 30, 2026

Uh oh!

tsocha commented Mar 30, 2026

Uh oh!

gujinghui Apr 9, 2026

Uh oh!

tsocha Apr 9, 2026

Uh oh!

gujinghui Apr 9, 2026 •

edited

Loading

Uh oh!

tsocha Apr 9, 2026

Uh oh!

moksiuc Apr 9, 2026

Uh oh!

gujinghui May 6, 2026 •

edited

Loading

Uh oh!

tsocha May 7, 2026

Uh oh!

gujinghui commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tsocha commented Mar 30, 2026

Uh oh!

tsocha commented Mar 30, 2026

Uh oh!

gujinghui Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

tsocha Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

gujinghui Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsocha Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

moksiuc Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

gujinghui May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsocha May 7, 2026

Choose a reason for hiding this comment

Uh oh!

gujinghui commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gujinghui Apr 9, 2026 •

edited

Loading

gujinghui May 6, 2026 •

edited

Loading