VideoTexture error on WebGPU break pipeline.

[whatisor]

Description

If Video or Video Decoder return 1 or few bad frames, pipeline cannot restore when getting good frame.

Issue may be from browser but I wonder if we can catch or precheck all those to avoid pipeline broken.

Reproduction steps

1. Only in Chrome MacOS. Use 1 video with some bad frames, maybe from other reason, not video itself.
2. Wait for onloadedData and check frame width, height
3. Render video frame using VideoTexture.

Code

Just any VideoTexture for WebGPU

Live example

https://github.com/mrdoob/three.js/blob/master/examples/webgpu_materials_video.html
I cannot share video because of privacy.

Screenshots

No response

Version

0.181.2

Device

Desktop

Browser

Chrome

OS

MacOS

[whatisor]

My workaround for this is listening 'error' event of windows and re-create renderer and texture manually.

[Mugen87]

So this error can happen at every moment, meaning when during the entire playback, right?

@greggman I have seen BabylonJS encountered the same error. Would you recommend a similar fix meaning guarding the copyExternalImageToTexture() call with a try/catch and ignore the error?

https://github.com/BabylonJS/Babylon.js/pull/15177/files

[whatisor]

So this error can happen at every moment, meaning when during the entire playback, right?

@greggman I have seen BabylonJS encountered the same error. Would you recommend a similar fix meaning guarding the copyExternalImageToTexture() call with a try/catch and ignore the error?

https://github.com/BabylonJS/Babylon.js/pull/15177/files

It is not easy to reproduce locally. It occurs every time for my 1 specific video at loading time in production deployment and cannot reproduce in local environment. And it is fine for WebGL renderer.
Using loadeddata event of video works fine for me for all other videos in all other environments but not this new video.

In any case of root cause, if we can catch those error, it makes sense to avoid pipeline broken.

[whatisor]

Confirmed simple catching works:
whatisor@d0a9098

[Mugen87]

Thanks for testing the fix! If @greggman approves, feel free to file a PR with your change.

[greggman]

To be honest I don't know. A repo would help track this down but I get that repos for video are hard. Clearly you should not have to catch an error to use video. But, it's a pragmatic fix.

Ideally someone who knows this better? @beaufortfrancois? @kainino0x? Could officially document what the correct steps are to never trigger this error. If you do manage to get a repo, filing a bug with a minimal example, ideally one that doesn't use three.js as that's not minimal, would help track down the issue quickly. Also, making a non three.js repo will help validate it's not a usage issue.

[Mugen87]

Thanks for the feeback!

Also, making a non three.js repo will help validate it's not a usage issue.

Since BabylonJS has the same fix in place, it seems to be not three.js specific. That said, I'll try to keep my eyes open for this runtime error. Unfortunately, the OP can't share the problematic video for privacy reasons. But if I encounter the issue with another video, I'll setup a fiddle.

@whatisor If you find some time, please open a PR with your changes from whatisor@d0a9098 👍 .

[greggman]

Since BabylonJS has the same fix in place, it seems to be not three.js specific.

It could be both BabylonJS and three.js are doing it wrong. Or it could be a browser bug.

[whatisor]

Since BabylonJS has the same fix in place, it seems to be not three.js specific.

It could be both BabylonJS and three.js are doing it wrong. Or it could be a browser bug.

We can say it is browser bug or limitation, we just try to protect our pipeline.

[Mugen87]

@greggman is right. We must have a reproducible test case first before we can state where the issue is actually located.

[whatisor]

If you want unit test for that API call( not specific this case), you just need to call rendering before video frame ready which should fire same exception

[greggman]

If you want unit test for that API call( not specific this case), you just need to call rendering before video frame ready which should fire same exception

I'm not sure what this means. If you're not waiting for the video to be ready then that's a bug in your code not waiting for it to be ready.

My understanding is, the correct, modern way to know its ready is to use the play promise

await someVideoElement.play();
// it's ready

or as a callback

someVideoElement.play().then(() => {
 // it's ready
});

Further, it's best to use requestVideoFrameCallback if it exists so you only copy when the frames changes (eg 30fps) and not every frame rendered (60~120fps)

Here's an example

Maybe you can plug your video in there and see if the issue repos