Position on emerging standard: video.requestAnimationFrame()

# Thomas Guilbert (16 days ago)

Hello,

I'm reaching out to see if webkit would like to weigh in on the following proposal:discourse.wicg.io/t/proposal-video-requestanimationframe/3691

The HTMLVideoElement.requestAnimationFrame() API allows web developers to be notified when a video frame has been presented for composition, and provides metadata for that frame.

If you want to try it out, a prototype is available in Chromium Dev, behind the enable-experimental-web-platform-features flag. Thank you! Thomas

Contact us to advertise here
# Simon Fraser (16 days ago)

On Jan 21, 2020, at 12:37 PM, Thomas Guilbert <tguilbert at google.com> wrote:

Hello, I'm reaching out to see if webkit would like to weigh in on the following proposal: discourse.wicg.io/t/proposal-video-requestanimationframe/3691, discourse.wicg.io/t/proposal-video-requestanimationframe/3691 The HTMLVideoElement.requestAnimationFrame() API allows web developers to be notified when a video frame has been presented for composition, and provides metadata for that frame. If you want to try it out, a prototype is available in Chromium Dev, behind the enable-experimental-web-platform-features flag.

This is not official feedback, but I have some issues with the proposal.

First, the name is confusing. It sounds like you're requesting a frame from the video, but it's really a "frame available" callback. Why not call it onFrameAvailable()?

Second, its interaction with normal requestAnimationFrame() and the HTML event loop needs to be better defined. Where in in the html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model, html.spec.whatwg.org/multipage/webappapis.html#event-loop-processing-model do these callbacks fire?

Simon

# Thomas Guilbert (16 days ago)

The idea was to reuse an API name that developers are already familiar with, in a similar context. The name is also being used in XRSession ( developer.mozilla.org/en-US/docs/Web/API/XRSession/requestAnimationFrame), and in OffscreenCanvas (or technically DedicatedWorkerGlobalScope). The AnimationFrameProvider mixin html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider

could also be updated so HTMLVideoElement can extend it.

Yes, this isn't formally spec'ed out, but it will be. For now, they are added to the task queue and run like any other task. So, going off the spec you linked, I think this would be "5) Perform oldestTask's step" and not "10) Rendering: [...] 11. Foreach document run animation frame callbacks for that Document".

# Simon Fraser (15 days ago)

On Jan 21, 2020, at 5:27 PM, Thomas Guilbert <tguilbert at google.com> wrote:

The idea was to reuse an API name that developers are already familiar with, in a similar context. The name is also being used in XRSession (developer.mozilla.org/en-US/docs/Web/API/XRSession/requestAnimationFrame, developer.mozilla.org/en-US/docs/Web/API/XRSession/requestAnimationFrame), and in OffscreenCanvas (or technically DedicatedWorkerGlobalScope). The AnimationFrameProvider mixin html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animationframeprovider could also be updated so HTMLVideoElement can extend it.

Yes, this isn't formally spec'ed out, but it will be. For now, they are added to the task queue and run like any other task. So, going off the spec you linked, I think this would be "5) Perform oldestTask's step" and not "10) Rendering: [...] 11. Foreach document run animation frame callbacks for that Document".

I would expect something that's called "requestAnimationFrame" to only fire in the "update the rendering" steps; requestAnimationFrame is a "before rendering" callback. So firing a callback with the same name at other times seems like it will lead to author confusion.

The author's expectation should be that any content/style changes they make inside a requestAnimationFrame callback will appear on-screen in the same frame as other changes in the same event loop cycle, and that requestAnimationFrame won't be called more often than is necessary to update the screen at the appropriate frame rate.

Simon

# Thomas Guilbert (14 days ago)

Yes, I agree with your comments, and I have some clarifications (but I know there are still some unanswered questions). Some of this might be implementation details of the prototype rather than abstract spec... I made a mistake, the callbacks are not actually queued in the event loop task queue. They are fired on the main thread ASAP after frame+info is sent from the compositor thread. The compositor thread submits that info only as part of the rendering phase, so the callbacks are actually tied to the "update the rendering" phase (but the precise timing is undefined at the moment). This means that the callbacks should be fired no more than is necessary (e.g. a 60fps video in a 30fps will receive video-rAF callbacks at 30fps). Due to the thread jump, guaranteeing that content/style changes in the video-raf callback will appear in the same event loop-cycle is tricky (and definitively not guaranteed in the current state of the Chromium prototype). However, the video-raf callback guarantees that there is enough time before the next window.rAF to do something useful with the video frame. The following is currently always true:

video.requestAnimationFrame((x, metadata) => { window.requestAnimationFrame((now) => { console.log("Useful?: " + (now <= metadata.expectedPresentationTime)); }); });

This means that a video.rAF callback can always be used to update a webgl texture, and paint it during the window.rAF, in time for its expected frame.

# Elliott Sprehn (10 days ago)

Would a method that returns a Promise be more in line with what this is doing?

# Philip Jägenstedt (4 days ago)

Naming is hard as usual and was discussed on w3ctag/design-reviews#429, where you've already commented.

Is there a pair of names that you think would work better here? onFrameAvailable() would IMHO be quite unidiomatic, the web platform doesn't have any other onFoo() methods, and what would the "cancel" variant be called?

Can you file an issue in WICG/video-raf/issues if you see a good alternative?

Also curious if +Eric Carlson <eric.carlson at apple.com> has any feedback on

this?

On Wed, Jan 22, 2020 at 10:49 PM Simon Fraser <simon.fraser at apple.com>

wrote:

# Ken Russell (3 days ago)

The name "requestAnimationFrame" was chosen mainly to achieve parity with the AnimationFrameProvider mixin, which now provides the same animation facility to the main thread and dedicated workers: html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animation-frames, discourse.wicg.io/t/proposal-video-requestanimationframe/3691

It offers a nice symmetry with other JavaScript-driven animations.

-Ken

On Mon, Feb 3, 2020 at 6:58 AM Philip Jägenstedt <foolip at chromium.org>

wrote:

# Simon Fraser (3 days ago)

On Feb 3, 2020, at 11:25 AM, Ken Russell <kbr at google.com> wrote:

The name "requestAnimationFrame" was chosen mainly to achieve parity with the AnimationFrameProvider mixin, which now provides the same animation facility to the main thread and dedicated workers: html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animation-frames, html.spec.whatwg.org/multipage/imagebitmap-and-animations.html#animation-frames, discourse.wicg.io/t/proposal-video-requestanimationframe/3691, discourse.wicg.io/t/proposal-video-requestanimationframe/3691

It offers a nice symmetry with other JavaScript-driven animations.

But the video.requestAnimationFrame behavior seems fundamentally different to window.requestAnimationFrame. It feels like it's conflating two different things.

Simon

# Ken Russell (3 days ago)

It seems to have similar behavior to me: on a window or worker, the requestAnimationFrame callback needs to re-schedule itself before exiting, in order to be called again in the future. The proposed HTMLVideoElement.requestAnimationFrame works similarly. A single onFrameAvailable callback defined on the HTMLVideoElement would behave fairly differently. Domenic Denicola's feedback on using requestAnimationFrame vs. a promise w3ctag/design-reviews#429

resonates with me.

# Thomas Guilbert (a day ago)

I am moving the execution of the video.rAF callbacks to be part of the rendering steps. This should guarantee that "changes made inside the callback appear at the same time as other changes in the same event loop cycle". I don't know if there is a place within the rendering steps where it makes most sense for them to run. I'm planning on having them run before the window.rAF callbacks, but I don't know after which of the other steps for now.

That being said, if you paint a 24fps <video> into a <canvas> on every

video.rAF call, the canvas might still be 1/60th of a second behind the video. This happens when we miss the rendering steps after jumping back on the main thread, and have to wait for the next rendering steps (waiting doesn't really change anything though, if the callbacks were fired earlier, their changes would still appear later).

Simon, does having video.rAF callbacks run as part of the rendering steps address some of your concerns?

# Simon Fraser (a day ago)

On Feb 5, 2020, at 4:52 PM, Thomas Guilbert <tguilbert at google.com> wrote:

I am moving the execution of the video.rAF callbacks to be part of the rendering steps. This should guarantee that "changes made inside the callback appear at the same time as other changes in the same event loop cycle". I don't know if there is a place within the rendering steps where it makes most sense for them to run. I'm planning on having them run before the window.rAF callbacks, but I don't know after which of the other steps for now.

That being said, if you paint a 24fps <video> into a <canvas> on every video.rAF call, the canvas might still be 1/60th of a second behind the video. This happens when we miss the rendering steps after jumping back on the main thread, and have to wait for the next rendering steps (waiting doesn't really change anything though, if the callbacks were fired earlier, their changes would still appear later).

Simon, does having video.rAF callbacks run as part of the rendering steps address some of your concerns?

It does, although I still don't like the name.

The spec also needs somehow trigger the "update the rendering" steps if a new video frame is available and there is a pending video.rAF (video playback alone does not trigger the rendering steps, at least in WebKit, for performance reasons).

Simon

# Thomas Guilbert (a day ago)

You're right, it doesn't in Chromium either (and I ran into that issue testing it :)). I will be updating the spec over the next few days.

Want more features?

Request early access to our private beta of readable email premium.