1.7.2.9
Prysm
Video support

Prysm can play video/audio through the standard <video> element like so:

<video width="320" height="240" src="my_movie.webm">
</video>

The video player will:

  • play opaque videos
  • play transparent videos
  • send audio data to the engine for playing (it won't play it by itself)
  • currently work only with videos
    • in the WebM format
    • with 1 video track encoded with VP8 or VP9
    • with 1 audio track encoded in Vorbis

The supported standard attributes are "autoplay", "loop" and "muted". The supported events are "durationchange", "ended", "loadstart", "seeking", "seeked", "volumechange".

Distribution

The video player is distributed in a separate dynamic library to reduce binary size for users who don't need it. The separate library is called MediaDecoders.[PlatformName].[PlatformExtension] (e.g. MediaDecoders.WindowsDesktop.dll on Windows). You need to place that extra binary next to core cohtml library in your distribution package and it will be automatically loaded.

Distribution on iOS

iOS doesn't support dynamic libraries so it's an exception to the rule above. Instead of placing a dynamic library, on iOS you'll need to statically link to either libMediaDecoders.iOS.a or libMediaDecodersEmpty.iOS.a depending on whether you want to use the video feature or not.

How to encode videos

Prysm doesn't have any special requirements for the video files as long as the format and the codecs used are supported. However video quality and playback performance depend highly on how the video was encoded. Using the latest versions of VPX encoders is highly recommended. Performance may suffer if old encoders are used or certain encoder features are disabled.

We recommend using the latest release version of ffmpeg for encoding your videos even if your preferred video editing software has an option to export in WebM. We have found ffmpeg to provide the best video quality and resulting decoding performance with its default VPX encoder settings, which makes ffmpeg easy to use, without the need to pass additional encoder options.

The basic usage of ffmpeg for transcoding is the following:

  • specify the input file with "-i"
  • choose a video codec with "-c:v" (either "libvpx-vp9" or "libvpx" for VP8)
  • specify a target bitrate for the video stream with "-b:v" (use "k" suffix for kilobits e.g. 512k or capital "M" suffix for megabits e.g. 3M)
  • choose an audio coded with "-c:a" (e.g. "libvorbis")
  • specify a target bitrate for the audio stream with "-b:a"
  • finish the command with a filename which extension is "webm", "ffmpeg" will automatically set the format accordingly

Example command for transcoding:

ffmpeg -i VideoIn.mp4 -c:v libvpx-vp9 -b:v 1M -c:a libvorbis -b:a 128k VideoOut.webm

For a better video quality results we recommend two-pass encoding, in which you will need to run ffmpeg twice, with almost the same settings, except for:

  • in pass 1 and 2, use the "-pass 1" and "-pass 2" options, respectively.
  • in pass 1, output to a null file descriptor, not an actual file. (This will generate a data file that ffmpeg needs for the second pass.)
  • in pass 1, you need to specify an output format (with "-f") that matches the output format you will use in pass 2.
  • in pass 1, you can leave audio out by specifying "-an".

Example commands for two-pass encoding:

ffmpeg -i VideoIn.mp4 -c:v libvpx-vp9 -b:v 1M -an -pass 1 -f webm NUL
ffmpeg -i VideoIn.mp4 -c:v libvpx-vp9 -b:v 1M -c:a libvorbis -b:a 128k -pass 2 VideoOut.webm
Note
Unix users should use "/dev/null" instead of "NUL"
When encoding VP9 videos, you can include the "-row-mt 1" switch to activate the multithreaded vpx encoder for faster video encoding.

You can refer to ffmpeg documentation on how to use it also to adjust resolution, framerate, audio channels and other media properties.

Transparent video support

The basic authoring process of transparent videos goes as follows:

  • Export a video with an alpha channels if such format is available in your video editing tool (e.g Quicktime PNG with RGBA) OR export a sequence of transparent PNGs
  • Feed the transparent video OR the sequence of PNGs (e.g -i sequence-%05d.png) as input to ffmpeg. The ffmpeg commands are the same with the addition of the "-pix_fmt yuva420p" switch which enables transparency for VPX.

Video playback performance

The video playback performance depends on the amount of data that needs to be processed and the amount of video data that needs to be displayed. The size the processing data is determined by the bitrate and the size of the display data is determined by the resolution. Using transparency in videos adds additional data for processing and additional channel to display.

  • Bitrate - lower bitrate yields better performance
  • Resolution - lower resolution yields better performance
  • Transparency - adds up to 60% more data for processing, depending on how complex is the alpha masking and 25% more display data

Most video compression algorithms are reusing the frame data from the previous frame and are only processing changes as this proves to be a very efficient compression technique for the most produced videos. This makes the size of the processing data to depend heavily on the amount of motion and scene switches present in the video. Due to the dynamic of nature of most videos, the amount of data to process varies from frame to frame.

The bitrate is the number of bits that are processed in a unit of time. Video data rates are given in bits per second. There are two methods of compression:

  • Using constant bitrate (CBR), the raw data will be compressed as much as needed to meet the target bitrate value for the frame. This means that the video quality will vary depending on the dynamics of the scene. This method of compression will ensure stable performance for the entire duration of the video, but it is considered wasteful and better to be avoided.
  • Using variable bitrate (VBR), the raw data will be compressed with a fixed amount which is calculated to meet the target bitrate as an average for the entire duration of the video. This means that the video quality will be constant during the entire video, but it may introduce performance issues in certain parts of the video where the bitrate is very high. We recommend you to use the option for setting the upper limit of the bitrate when using VBR ("-maxrate" in ffmpeg).

The resolution isn't usually changed during playback, so it can be said that it has a fixed performance cost. Higher resolution or frame rate means more data to process, hence it will require higher bitrate to preserve the video quality.

VP8 vs VP9 - We have found that using the exact same properties for encoding, VP8 gives better performance, but produces worse video quality. VP9 performs slower, but will produce better video quality. If we are about to compensate for the quality difference by adjusting the bitrate, either higher for VP8 or lower for VP9, then the performance difference will be evened out. Additionally, for the same perceivable quality, using VP9 will result in smaller file sizes due to the lower bitrate. We recommend using VP9 when possible, VP8 is generally used for compatibility reasons.

Multi-threaded decoding

Video decoding always happens through Cohtml's task system by posting Resources tasks to be executed on worker threads. You can read more about the task system in the Technical Overview section. Each frame's decoding job will be split on multiple tasks for more efficient decoding, depending on how many worker threads are hinted to the SDK. This is done by specifying a value to cohtml::LibraryParams::ResourceThreadsCountHint. The video player will schedule up to 4 tasks for opaque videos and up to 8 for transparent ones.

Warning
If you set cohtml::LibraryParams::ResourceThreadsCountHint to a number higher than the actual number of threads serving the task queue, Cohtml is very likely to enter a deadlock. Always set cohtml::LibraryParams::ResourceThreadsCountHint to a number less than or equal to the actual thread count.

Audio support

Prysm does not play audio by itself. All audio data is decoded, converted to Pulse-code modulation (PCM) and passed to the engine for further processing. The PCM data is passed through several callbacks on the cohtml::IViewListener interface (look for the OnAudio* methods)

You can use your engine's audio system to enqueue the PCM data in the sound buffers and get it playing. There are two reference implementation available

  • one based on Windows' XAudio2 and one on OpenAL. Both can be found under Modules/AudioSystem/. The Audio system module provides an abstraction over both implementations and can also be used directly in the engine by including the source file and linking to the corresponding third party dependencies.

Take a look at the Sample_VideoPlayer sample in the distribution package for more info.

How to play video with controls

In order to use controls, you need to include an additional JavaScript library which you can find under Samples/uiresources/VideoPlayback. After you include the library, use the custom HTML element:<video-with-controls>.

<video-with-controls id="myVideo" src="my_movie.webm" width="320px" height="240px">
</video-with-controls>

To use the library you must:

  • copy the files from Samples/uiresources/VideoPlayback to your UI directory
    • video_controls_images folder
    • video_controls.js
    • video_controls.css
  • include video_controls.js in the beginning of your HTML file
  • include video_controls.css which contains the styles for the controls

JavaScript API is the same as the one for HTMLMediaElement:

  • Supported attributes:
    • src
    • width
    • height
    • autoplay
    • loop
    • muted
  • Supported properties:
    • paused
    • ended
    • loop
    • autoplay
    • currentTime
    • duration
    • volume
    • muted
    • src
  • Supported methods:
    • play()
    • pause()
  • Events are not supported

If the API exposed by our video container element is not sufficient for your needs, you can get the video element itself:

let video = document.getElementById("myVideo").querySelector("video");

Media playback events

The media element supports the following standard events:

  • durationchange: The metadata has loaded or changed, indicating a change in duration of the media. This is sent, for example, when the media has loaded enough that the duration is known.
  • emptied: The media has become empty; for example, this event is sent if the media has already been loaded (or partially loaded), and the load() method is called to reload it.
  • ended: Sent when playback completes.
  • pause: Sent when the playback state is changed to paused (paused property is true).
  • play: Fired after the play() method has returned, or when the autoplay attribute has caused playback state to change. Note that this does not necessarily mean there's actual playback, since the network request can be delayed. See the played event for more information.
  • playing: Sent when the media has enough data to start playing, after the play event.
  • seeked: Sent when a seek operation completes.
  • seeking: Sent when a seek operation begins.
  • timeupdate: The time indicated by the element's currentTime attribute has changed. Note that this event is not fired during normal playback.
  • volumechange: Sent when the audio volume changes (both when the volume is set and when the muted attribute is changed).

There are a few custom events not present in the standard:

  • cohplaybackstalled: Sent when a decoder is unable to keep up with the playback rate and cannot provide new frames quickly enough. This event is useful when trying to synchronize the video playback with other animations, since the frame decoding is asynchronous and not tied in any way to the View timer. Usually, pausing any animations that need to be in sync with the video on the cohplaybackstalled event and resuming them on the cohplaybackresumed event is enough.
  • cohplaybackresumed: Sent when the video/audio decoder that previously stalled the playback is now caught up and provided new frames.

Customizing controls

If you want to customize the way your video player looks - copy your custom images in the uiresources/VideoPlayback/video_controls_images folder and keep the names the same.

If you want to customize the font size or another inherited, then add CSS style to our video container element.

Showing video preview

Currently Prysm does not render video frames unless the video is playing or seeked. This means that there will be no image of the video shown initially. In order to show a preview, either play the video, or simply seek to the desired time in seconds, e.g. videoElement.currentTime = 0;.

Resource handler

When playing a video, Prysm will send range requests to the client's implementation of cohtml::IAsyncResourceHandler::OnResourceRequest to avoid loading the entire video file in memory and potentially run out of memory on platforms with limited hardware. You check the reference implementation on how to handle range requests in the class resource::ResourceHandler, that is used all across the samples.