Skip to content

[SYCL][UR][OpenCL] Fix profiling tag ordering on OpenCL GPU#22641

Open
crystarm wants to merge 1 commit into
intel:syclfrom
crystarm:profiling-tag-opencl-timestamp-recording
Open

[SYCL][UR][OpenCL] Fix profiling tag ordering on OpenCL GPU#22641
crystarm wants to merge 1 commit into
intel:syclfrom
crystarm:profiling-tag-opencl-timestamp-recording

Conversation

@crystarm

Copy link
Copy Markdown
Contributor

Related to #22229.

submit_profiling_tag returned timestamps in the wrong order on opencl:gpu: the end tag's command_end could be earlier than the completion of work submitted before it, so E2End <= EndTagEnd failed and the e2e tests were masked with UNSUPPORTED: opencl && gpu.

Why

OpenCL devices don't advertise ext_oneapi_queue_profiling_tag (the adapter hard-codes UR_DEVICE_INFO_TIMESTAMP_RECORDING_SUPPORT_EXP to false), so the tag fell back to ext_oneapi_submit_barrier() and read its profiling info straight off the barrier event.

The NEO driver timestamps clEnqueueBarrierWithWaitList / clEnqueueMarkerWithWaitList when the command enters the pipeline, not when the preceding work completes. So the barrier's CL_PROFILING_COMMAND_END can be earlier than the end of a kernel enqueued before it.

I confirmed this with a small pure-OpenCL reproducer on Intel Iris Xe (NEO 26.05.037020): any synchronization-only tag (barrier/marker, empty or explicit wait list) fails 100% of the time, while any real device command (kernel or buffer fill) gives correct timestamps - including a fill into a
dedicated buffer, which is what this PR does.

Changes

  • unified-runtime/.../opencl/event.cpp: implement urEnqueueTimestampRecordingExpusingclEnqueueFillBufferon a small internal buffer, whose timestamps do reflect completion of prior work. It needs command profiling, so if the queue isn't profiling-enabled we returnUR_RESULT_ERROR_UNSUPPORTED_FEATURE`.

  • unified-runtime/.../opencl/context.hpp: lazily-created, thread-safe 4-byte buffer per context, released with the context. It's never read - only the fill command's timestamps are used.

  • sycl/.../scheduler/commands.cpp (CGType::ProfilingTag): issue the recording with call_nocheck and fall back to a barrier on UR_RESULT_ERROR_UNSUPPORTED_FEATURE. Also addresses the existing TODO here.

  • sycl/.../experimental/profiling_tag.hpp: when the device lacks the aspect but the queue has profiling enabled, submit a profiling-tag command group (reusing the scheduler's in-order/out-of-order marker+barrier handling) instead of a bare barrier.

The device aspect stays false for OpenCL: the emulation needs queue profiling, but the aspect promises the tag works without it. Enabling the aspect (e.g. via an internal profiling queue) is a possible follow-up.

Level Zero / CUDA / HIP are unaffected - they advertise the aspect and already implement the recording, so they never take the fallback branch. The scheduler change only adds the fallback branch and turns a throwing call into a checked one; the success path is unchanged.

Testing

  • Unit tests (sycl/unittests/Extensions/ProfilingTag.cpp): split the old fallback test into two - native recording used on a profiling queue without the aspect, and the barrier fallback when the backend reports the recording unsupported.

  • E2E: dropped UNSUPPORTED: opencl && gpu from ProfilingTag/in_order_profiling_queue.cpp and ProfilingTag profiling_queue.cpp.

  • The reproducer's dedicated-buffer fill (matching this implementation) passes 200/200 iterations on Intel Iris Xe, vs. the old barrier fallback failing 200/200.

The NEO timestamping behavior is arguably a driver bug worth reporting separately; this is a runtime workaround that doesn't touch the public API or other backends.

P.S. I may have gone a little overboard with the code comments. If anyone else feels the same way, I will clean them up.

@crystarm
crystarm requested review from a team as code owners July 15, 2026 11:33
@crystarm
crystarm requested a review from againull July 15, 2026 11:33
@bratpiorka
bratpiorka requested a review from Copilot July 16, 2026 09:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes incorrect profiling-tag timestamp ordering on OpenCL GPUs by changing the Unified Runtime (UR) OpenCL adapter to record timestamps using a lightweight real device command (buffer fill) instead of synchronization-only commands whose profiling timestamps can be misordered on some drivers. This helps ensure profiling data ordering invariants hold and un-masks previously disabled SYCL ProfilingTag E2E coverage for opencl:gpu.

Changes:

  • Implement urEnqueueTimestampRecordingExp for OpenCL using clEnqueueFillBuffer on a lazily created per-context internal buffer (requires a profiling-enabled queue; otherwise reports unsupported).
  • Update SYCL scheduler ProfilingTag enqueue to attempt native timestamp recording via call_nocheck and fall back to a barrier on UR_RESULT_ERROR_UNSUPPORTED_FEATURE; update submit_profiling_tag() to route non-aspect + profiling-enabled queues through the internal profiling-tag CG path.
  • Split/adjust unit tests for both “timestamp recording works” and “timestamp recording unsupported → barrier fallback”, and re-enable OpenCL GPU E2E tests by removing the UNSUPPORTED lines.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
unified-runtime/source/adapters/opencl/event.cpp Implements OpenCL timestamp recording via a small buffer fill and requires queue profiling.
unified-runtime/source/adapters/opencl/context.hpp Adds lazy, thread-safe internal buffer storage to support the OpenCL timestamp emulation.
sycl/source/detail/scheduler/commands.cpp Uses non-throwing UR calls for timestamp recording with an explicit unsupported-feature fallback to a barrier.
sycl/include/sycl/ext/oneapi/experimental/profiling_tag.hpp Submits an internal profiling-tag CG when queue profiling is enabled even without the device aspect.
sycl/unittests/Extensions/ProfilingTag.cpp Expands unit tests to cover native timestamp path vs unsupported-feature barrier fallback.
sycl/test-e2e/ProfilingTag/profiling_queue.cpp Re-enables OpenCL GPU E2E coverage by removing the UNSUPPORTED directive.
sycl/test-e2e/ProfilingTag/in_order_profiling_queue.cpp Re-enables OpenCL GPU E2E coverage by removing the UNSUPPORTED directive.

Comment on lines +340 to +349
std::vector<cl_event> CLWaitEvents(numEventsInWaitList);
for (uint32_t I = 0; I < numEventsInWaitList; I++)
CLWaitEvents[I] = cast(phEventWaitList[I])->CLEvent;

const cl_uint Pattern = 0;
cl_event Event = nullptr;
CL_RETURN_ON_FAILURE(clEnqueueFillBuffer(
Queue->CLQueue, Buffer, &Pattern, sizeof(Pattern), /*offset=*/0,
/*size=*/sizeof(Pattern), numEventsInWaitList, CLWaitEvents.data(),
ifUrEvent(phEvent, Event)));
Comment on lines 75 to 83
for (uint32_t i = 0; i < DeviceCount; i++) {
ur::opencl::urDeviceRelease(cast(Devices[i]));
}
if (TimestampRecordingBuffer) {
clReleaseMemObject(TimestampRecordingBuffer);
}
if (IsNativeHandleOwned) {
clReleaseContext(CLContext);
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants