[jantonguirao]

Category:

New feature (non-breaking change which adds functionality)

Description:

Problem. FramesDecoderGpu kept exactly one picture in flight per decoder. The parser was created with ulMaxDisplayDelay = 0 and no display callback. Every decode callback submitted the picture with cuvidDecodePicture and then, inline, mapped it (cuvidMapVideoFrame blocks until that picture is decoded), converted it to RGB, synchronized the stream and unmapped it. Only then did the next packet get read, filtered and parsed. NVDEC was idle for the whole map wait, conversion, sync and host-side packet handling. That cost is per picture, so it limits every single-stream read.

Mechanism. Pictures are now mapped in display order by the parser's display callback (pfnDisplayPicture), with a display delay of 1 (kDisplayDelay):

• The parser keeps decoded pictures ahead of the one being displayed, so NVDEC keeps decoding the next pictures while the host waits for, converts and syncs the current one.
• The parser and the decoder use min_num_decode_surfaces + kDisplayDelay + 2 surfaces. The display delay keeps that many more pictures, and the picture being mapped and converted must not be decoded over. The NVDEC cache key already includes the surface count.
• The decode callback only submits the picture and records its pts with its surface (CurrPicIdx). The display callback looks the pts up by picture_index and does the existing map, convert, sync and unmap.
• flush_ turns the display callback into a no-op, so pictures that the parser releases while flushing on a seek or reset are dropped, as before.
• Nothing changes in the reorder buffer, the conversion, the host sync or the reader operators.

Pitfall 1: open-GOP HEVC timestamps. The obvious implementation takes the pts from CUVIDPARSERDISPINFO::timestamp. That is wrong after a seek into an HEVC open GOP:

• The parser does not output the leading (RASL) pictures of the CRA.
• The timestamps it then passes to the display callback are shifted: the CRA was displayed with the pts of a later packet.
• The requested frame is never found and the reader runs to EOF.

That variant passed every H.264 check and gave wrong frames on shuffled HEVC reads. The pts is therefore taken, as before, from the decode-order queue of sent packets at decode time, and kept per surface.

New gtest FramesDecoderGpuTest.HevcSeekToOpenGopKeyframes reads the HEVC Sintel trailer sequentially and with shuffled seeks to its keyframes, and requires exact equality. It passes with this change and fails on a build that uses the display timestamp.

Pitfall 2: the last picture of a VP8/VP9 stream. VP8/VP9 pictures are decoded while their own packet is parsed. H.264/HEVC instead have a one-packet lag. So at EOF the queue of sent packets can be empty while the last picture still waits in the parser for its display callback (the display delay). The EOF check DALI_ENFORCE(piped_pts_.size() >= 1) then failed whenever a sample read the last frame of a VP8/VP9 file. The decoder now tracks pictures that were decoded but not displayed yet (surface_to_display_, cleared on flush), and the EOF check accepts those.

New gtest FramesDecoderGpuTest.ReadsTheLastFrameOfCodecsWithoutDecodeLag covers VP8 and VP9: it reads sequentially to the end and seeks near the end. It fails with the original assertion on the build without this fix and passes with it.

Measured gains (an H100, fn.experimental.readers.video on the GPU, 720p Sintel trailer):

Scenario	Change vs main (4 rounds)	A/A (same build twice)
Single stream, H.264, sequential	+9.5% to +9.7%	within 0.2%
Single stream, HEVC, sequential	+21.7% to +22.0%	within 0.2%
Shuffled (random-access) reads, HEVC	+14.0% to +14.4%	within 0.3%

Protocol:

• 4 interleaved rounds, with the arm order randomized per round.
• Each value is the median of 3 fresh processes.
• CPU threads pinned; the GPU was not used by anything else during a run.
• An A/A arm (the main build measured as a second arm) bounds the noise at 0.3% or less.
• Output hashes were identical to main on all benchmark groups (sequential, random, multi-stream, reader depth, CPU) at 1, 2, 4 and 8 threads.

The timings were taken on an earlier revision of this change. What was added since does not change the work per picture of a progressive stream: the end-of-stream check and the clearing of the pending flags, which run once per stream end, the skipped pts pop for the second field of a field pair, and the tests.

Mechanism, from one Nsight Systems capture per arm (H.264, 4 threads):

• Pictures outstanding on NVDEC when a map starts: 1 on main, 7 (p10-p90 range 4-10) with this change.
• Mean interval between consecutive cuvidDecodePicture calls: 788 us on main, 699 us with this change.
• Larger display delays (2, 4) remove the map wait almost completely but give no further throughput: the session is then bound by the decode engine. That is why the delay stays at 1, with the fewest extra surfaces.

Measured on a second GPU (an RTX 5080, fn.experimental.readers.video on the GPU, 720p Sintel trailer clips; builds of this change and of its base commit that differ only in libdali_operators.so):

Scenario	fps, base -> this change	Change (95% interval)
H.264, sequential, sequence length 8	4249 -> 5587	+31.5% (29 to 34)
HEVC, sequential, sequence length 8	3577 -> 4560	+27.5% (26 to 31)
15 H.264 clips, batch 8, sequential	2757 -> 3496	+26.8% (22 to 31)
H.264, shuffled reads, batch 1	411 -> 472	+14.7% (11 to 16)
HEVC, shuffled reads, batch 1	2299 -> 2433	+5.8% (4.8 to 9.5)
HEVC, shuffled reads, batch 8	2348 -> 2418	+3.0% (2.7 to 3.3)
Short looping clips (a restart every few samples), H.264 / HEVC	4585 -> 5640 / 3872 -> 4488	+23.0% / +15.9%

• 13 workloads (sequential, shuffled, looping short clips; sequence lengths 8 and 32; batch 1 and 8), 10 interleaved rounds each, with the order of the two builds alternated and the order of the workloads shuffled; a fresh process per run, the first epoch discarded, at least 20 batches and 6 s timed.
• The change is the median of the per-pair differences, with a bootstrap 95% interval. This change was faster in all 130 pairs.
• There was no A/A arm on this machine. The spread between runs was 0.3% to 3.3% for the base and below 0.5% with this change. Nothing else used the GPU.
• GPU memory grows by 6 MiB per process (the 3 extra surfaces at 720p), independent of the batch size. The mean NVDEC utilization goes from 35-41% to 44-47%.

Correctness checks (an H100; every comparison was made against main built from the same base on the same machine):

• C++: *FramesDecoder*, *FrameIndexCache*, *VideoReader*, *VideoDecoder*, *VideoInput*, one process per test: 101 run.
  • The only failures are VideoReaderTest_{readers__Video,experimental__readers__Video}.MultipleVideoResolution, which fail on main on the same machine too.
• Python, identical results on main and on this branch:
  • decoder/test_video.py: 148 OK, 10 skipped.
  • input/test_video.py: 42 OK.
  • test_video_reader.py: 60 OK.
  • test_video_pipeline.py: 65 OK.
  • test_video_reader_resize.py: 14 OK.
  • Checkpointing test_video_reader, test_video_reader_resize_reader and test_experimental_video_reader: 124 OK.
• readers.video GPU, shuffled sequences, checked against PyAV per frame:
  • Byte-identical to main on every H.264, HEVC, VP8, VP9 and MPEG-4 clip of the test data, including the AVI/MKV containers.
  • The packed B-frame AVI gives the same frames as main on every frame. main is not deterministic run to run on that clip either.
• fn.decoders.video and fn.experimental.inputs.video on the GPU, over 8 clips (H.264, HEVC, VP8, VP9, MPEG-4, VFR) plus start_frame / sequence_length reads of the long H.264, HEVC and HEVC 10-bit clips: all 22 outputs byte-identical to main.

Limitations:

• AV1 is not tested in CI. The GPUs used for the timings and tests above report no NVDEC support for AV1, and the Python AV1 cases are skipped unconditionally. The AV1 test clips contain hidden frames and show_existing_frame temporal units, which have no decode callback. The pts recorded per surface from the decode-order packet queue assumes one decoded picture per packet, so timestamps of AV1 streams with such frames may be off. (On main, those frames were never displayed and hidden frames were.) This was checked on a Blackwell GPU with AV1 decode: the reader already returns wrongly indexed AV1 frames on main, because a temporal unit does not map to one decoded picture (it can hold hidden frames, or only a show_existing_frame), and this change neither fixes nor worsens that. Tracked in DALI-5008.
• The interaction with other in-flight decoder changes (e.g. multiple decode lanes per reader) was not measured.
• Random-access reads of H.264 were not timed on the H100. On the RTX 5080 they improve too (+14% to +15% for shuffled 720p reads), so the extra decode-ahead did not cost on short seeks there. The shuffled HEVC case improves on both GPUs.
• Timings come from one H100 and one RTX 5080 configuration, with 720p clips. The gains differ between the two GPUs, and they will differ on other GPUs and on multi-stream workloads that already keep NVDEC busy. Several concurrent decoders (several readers, or several GPUs) were not timed.

Additional information:

Affected modules and functionalities:

• dali/operators/video/frames_decoder_gpu.{h,cc}:
  • Display delay and display callback.
  • Surface count for the parser and the decoder.
  • Pts per surface.
  • EOF check for decoded pictures not yet displayed.
• dali/operators/video/frames_decoder_test.cc: two new GPU tests, and the slack of FramesDecoderGpuTest.DecodeWork: the decoder is now ahead of the AddDecodeWork estimate by the reordering depth of the stream plus kDisplayDelay, plus a packet of lag, so the test bounds that lead (as a number of frames of lookahead) instead of a single picture. The CPU decoder keeps the exact check.
• Affects every GPU user of FramesDecoderGpu: the GPU video readers, fn.decoders.video and fn.experimental.inputs.video.

Key points relevant for the review:

• Where the pts comes from: per surface at decode time, not CUVIDPARSERDISPINFO::timestamp. See pitfall 1.
• The surface count (NumDecodeSurfaces), returned from the sequence callback and used for the cached decoder.
• The EOF condition and its reset on flush (pitfall 2).
• Codecs where a packet produces zero or several decoded pictures (AV1, see Limitations).
• The lead model in DecodeWork: the reordering depth is chosen by codec (2 for H.264 and HEVC, 0 otherwise), and the constants were measured, not derived from the parser; several clips meet the bound exactly. AddDecodeWork still estimates the pictures up to the last frame read, so for the GPU decoder it is a lower bound. Only tests call it and DecodedPictures() in this tree.

Tests:

• Existing tests apply
  • frames_decoder_test.cc: FramesDecoderGpuTest.*
  • video reader / decoder / input gtests: VideoReader*, VideoReaderDecoder*, VideoInput*
  • decoder/test_video.py, input/test_video.py, test_video_reader.py, test_video_pipeline.py, test_video_reader_resize.py
  • checkpointing/test_dali_checkpointing.py: test_video_reader*, test_experimental_video_reader
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4946

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

🟢 Approval recommended

No active review issues are identified for the GPU decoder changes.

No open issues 🎉

What changed in this PR

This PR enables multiple pictures to remain in flight in FramesDecoderGpu, reducing NVDEC idle time for GPU video readers, decoders, and inputs.

Changes:

• Moves frame mapping and conversion to the parser’s display callback, with a display delay of one and additional decode surfaces.
• Tracks timestamps and pending display state per surface to preserve seek behavior and handle delayed frames at end of stream.
• Adds GPU regression tests for HEVC open-GOP seeks and VP8/VP9 last-frame reads.

File	Description
dali/​operators/​video/​frames_decoder_test.cc	Adds HEVC seek and VP8/VP9 end-of-stream regression tests.
dali/​operators/​video/​frames_decoder_gpu.h	Declares display delay, surface-count helper, and per-surface tracking state.
dali/​operators/​video/​frames_decoder_gpu.cc	Implements delayed display, expanded surface allocation, timestamp tracking, and EOF/flush handling.

_{Review effort: Balanced}

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Keeps several pictures decoding on NVDEC while the GPU converts an earlier picture.

• GPU video decoding overlaps later pictures with frame display.
• GPU video reads can return the last VP8 and VP9 frame.

Both earlier, unnumbered findings are addressed: the VP8/VP9 test now compares pixels against the CPU decoder, and the HEVC test uses make_string(...). No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Read packet] --> B[Submit picture to NVDEC]
  B --> C[Save timestamp per surface]
  C --> D[Parser delays display]
  D --> E[Display callback]
  E --> F[Map and convert picture]
  F --> G[Wait and unmap]

Loading

_{Reviews (5) · Last reviewed commit: "Fix the picture slack of the decode work..." · Reviewed by Greptile}

[MykolaSemylitko]

please use const var instead of magic number

[jantonguirao]

Done: the 2 is now kSpareDecodeSurfaces, declared next to kDisplayDelay in the header with the reason for it.

[jantonguirao]

!build

[dali-automaton]

CI MESSAGE: [79539833]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [79539833]: BUILD FAILED

[jantonguirao]

!build

[dali-automaton]

CI MESSAGE: [79818830]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [79818830]: BUILD FAILED

[dali-automaton]

CI MESSAGE: [79818830]: BUILD PASSED