Skip to content

fix(core): use raw data-start for media elements in preview visibility loop#1078

Closed
JamesXiaoFF wants to merge 2 commits into
heygen-com:mainfrom
JamesXiaoFF:fix/preview-pip-video-double-offset
Closed

fix(core): use raw data-start for media elements in preview visibility loop#1078
JamesXiaoFF wants to merge 2 commits into
heygen-com:mainfrom
JamesXiaoFF:fix/preview-pip-video-double-offset

Conversation

@JamesXiaoFF
Copy link
Copy Markdown
Contributor

Problem

When a pip-wired <video> or <audio> element has a data-start value in global (composition-root) time and lives inside a sub-composition that also starts at that same global time, the preview visibility loop was keeping it permanently hidden.

Root cause: The visibility loop called resolveStartForElement, which adds the nearest ancestor composition's global start on top of the element's own data-start. For a pip video with data-start="45.40" inside a host at data-start="45.40", the resolved start became 45.40 + 45.40 = 90.80 — well past the video's actual window.

Why media elements are different: The render pipeline's discoverMediaFromBrowser reads data-start on <video> and <audio> as a raw global timestamp (not composition-relative). The preview visibility loop must use the same semantics to stay consistent with render output.

Fix

In the visibility loop inside init.ts, media elements (video, audio) now read data-start directly from the DOM attribute instead of going through resolveStartForElement. Non-media elements (divs, sections, etc.) continue to use the accumulating resolver because their data-start values are local to their composition timeline.

Test

Added a regression test in init.test.ts that reproduces the exact double-offset scenario: a pip video at global data-start="45.40" inside a host composition starting at 45.40 must be visible at t=46 and hidden at t=53.

…y loop

For video and audio elements, data-start is authored in global (composition-root)
time — the same contract used by the render pipeline's discoverMediaFromBrowser,
which reads the raw attribute directly. Previously, the visibility loop called
resolveStartForElement which adds the nearest ancestor composition's global start
on top, causing a double-offset that kept pip-wired media permanently hidden when
the host composition did not start at t=0.

Example: a pip video with data-start="45.40" inside a host composition that also
starts at data-start="45.40" resolved to 90.80, so the video was always hidden
during its actual [45.40, 52.46] window.

Non-media elements (divs, sections, etc.) continue to use the accumulating
resolver because their data-start values are local to their composition.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@JamesXiaoFF JamesXiaoFF force-pushed the fix/preview-pip-video-double-offset branch from 0211b02 to 53e3175 Compare May 26, 2026 08:05
miguel-heygen

This comment was marked as duplicate.

miguel-heygen

This comment was marked as duplicate.

@miguel-heygen miguel-heygen dismissed their stale review May 26, 2026 15:33

Superseded — resubmitting after reproducing the bug.

miguel-heygen

This comment was marked as duplicate.

@miguel-heygen miguel-heygen dismissed their stale review May 26, 2026 15:35

Cleaning up — resubmitting as a single clean review.

miguel-heygen

This comment was marked as duplicate.

@miguel-heygen miguel-heygen dismissed their stale review May 26, 2026 15:43

Rewriting

Copy link
Copy Markdown
Collaborator

@miguel-heygen miguel-heygen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed the bug — I reproduced the double-offset on main. The pip video gets resolved to 90.80 instead of 45.40 and stays permanently hidden. Good catch.

Two things I'd want before merging:

The fix only covers the visibility loop, but refreshRuntimeMediaCache (init.ts:1284) and resolveMediaWindowEndSeconds (timeline.ts:223) resolve media start times the same way. Without patching those too, the video shows up at the right moment but its playback timing is still off. Fixing this in resolveHostOffsetForElement instead would cover all three call sites in one shot.

Also worth adding a test for the auto-injected data-start="0" case — a video with no explicit timing inside a host at data-start="10" should still appear at t=10, not t=0. If that already works, a quick test proving it would round this out nicely.

Add also a regression test in the producer tests to avoid regressions.

Narrow the raw data-start read to media elements without
data-hf-auto-start (explicitly authored global coordinates). Elements
with auto-injected data-start="0" remain composition-local via the
resolver. Apply consistently across all three consumers:
- visibility loop (init.ts)
- refreshRuntimeMediaCache start/duration (init.ts)
- resolveMediaWindowEndSeconds (timeline.ts)

Add regression test for auto-injected data-start="0" inside a
late-starting host to prove it doesn't regress.
@miguel-heygen miguel-heygen dismissed their stale review May 26, 2026 16:53

Addressed in new commit

Copy link
Copy Markdown
Collaborator

@miguel-heygen miguel-heygen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a follow-up commit addressing the review feedback:

  • Narrowed the raw read to media elements without data-hf-auto-start so auto-injected timing stays composition-local
  • Extended the fix to refreshRuntimeMediaCache and resolveMediaWindowEndSeconds in timeline.ts
  • Added regression test for data-start="0" inside a late-starting host

All 998 core tests pass.

@miguel-heygen
Copy link
Copy Markdown
Collaborator

I had to do a new PR including a regression test with LFS! Thanks for contributing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants