Media Source API

Draft: This page is not complete.

The Media Source API, formally known as Media Source Extensions (MSE), provides functionality enabling plugin-free web-based streaming media. Using MSE, media streams can be created via JavaScript, and played using <audio> and <video> elements.

Media Source Extensions concepts and usage

Playing video and audio has been available in web applications without plugins for a few years now, but the basic features offered have really only been useful for playing single whole tracks. We can't, for example, combine/split arraybuffers. Streaming media has up until recently been the domain of Flash, with technologies like Flash Media Server serving video streams using the RTMP protocol.

The MSE standard

With Media Source Extensions (MSE), this is changing. MSE allows us to replace the usual single track src value fed to media elements with a reference to a MediaSource object, which is a container for information like the ready state of the media for being played, and references to multiple SourceBuffer objects that represent the different chunks of media that make up the entire stream. MSE gives us finer grained control over how much and how often content is fetched, and some control over memory usage details, such as when buffers are evicted. It lays the groundwork for adaptive bitrate streaming clients (such as those using DASH or HLS) to be built on its extensible API.

Creating assets that work with MSE in modern browsers is a laborious process, taking significant time, computing power, and energy. The usage of external utilities to massage the content into a suitable format is required. While browser support for the various media containers with MSE is spotty, usage of the H.264 video codec, AAC audio codec, and MP4 container format is a common baseline. MSE also provides an API for runtime detection of container and codec support.

If you do not require explicit control of video quality over time, the rate at which content is fetched, or the rate at which memory is evicted, then the <video> and <source> tags may well be a simple and adequate solution.

DASH

Dynamic Adaptive Streaming over HTTP (DASH) is a protocol for specifying how adaptive content should be fetched. It is effectively a layer built on top of MSE for building adaptive bitrate streaming clients. While there are other protocols available (such as HTTP Live Streaming (HLS)), DASH has the most platform support.

DASH moves lots of logic out of the network protocol and into the client side application logic, using the simpler HTTP protocol to fetch files. Indeed, one can support DASH with a simple static file server, which is also great for CDNs. This is in direct contrast with previous streaming solutions that required expensive licenses for proprietary non-standard client/server protocol implementations.

The two most common use cases for DASH involve watching content “on demand” or “live.” On demand allows a developer to take their time transcoding the assets into multiple resolutions of various quality.

Live profile content can introduce latency due to its transcoding and broadcasting, so DASH is not suitable for real time communication like WebRTC is. It can however support significantly more client connections than WebRTC.

There are numerous available free and open source tools for transcoding content and preparing it for use with DASH, DASH file servers, and DASH client libraries written in JavaScript.

Interfaces

MediaSource

Represents a media source to be played via an HTMLMediaElement object.

SourceBuffer

Represents a chunk of media to be passed into an HTMLMediaElement via a MediaSource object.

SourceBufferList

A simple container list for multiple SourceBuffer objects.

VideoPlaybackQuality

Contains information about the quality of video being played by a <video> element, such as number of dropped or corrupted frames. Returned by the HTMLVideoElement.getVideoPlaybackQuality() method.

Extensions to other interfaces

URL.createObjectURL()

Creates an object URL pointing to a MediaSource object that can then be specified as the src value of an HTML media element to play a media stream.

HTMLMediaElement.seekable

When a MediaSource object is played by an HTML media element, this property will return a TimeRanges object that contains the time ranges that the user is able to seek to.

HTMLVideoElement.getVideoPlaybackQuality()

Returns a VideoPlaybackQuality object for the currently played video.

AudioTrack.sourceBuffer, VideoTrack.sourceBuffer, TextTrack.sourceBuffer

Returns the SourceBuffer that created the track in question.

Specifications

Specification Status Comment
Media Source Extensions Recommendation Initial definition.

Browser compatibility

Desktop Mobile
Chrome Edge Firefox Internet Explorer Opera Safari WebView Android Chrome Android Firefox for Android Opera Android Safari on IOS Samsung Internet
Media_Source_Extensions_API
31
23-31
12
42
11
18
15-18
8
4.4.3
31
25-31
41
18
14-18
13
Exposed in Mobile Safari on iPad but not on iPhone.
2.0
1.5-2.0
MediaSource
31
23-31
12
42
11
Only works on Windows 8+.
15
8
4.4.3
33
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
2.0
activeSourceBuffers
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5
addSourceBuffer
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5
clearLiveSeekableRange
62
17
No
No
49
10
62
62
?
46
13
Exposed in Mobile Safari on iPad but not on iPhone.
8.0
duration
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5
endOfStream
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5
isTypeSupported
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5
onsourceclose
53
17
No
This event handler attribute is not supported; however, the event itself is supported since Firefox 42. The event can be listened to via mediaSource.addEventListener('sourceclose', function() {});. See bug 1689222.
11
Only works on Windows 8+.
15
10.1
4.4.3
33
No
This event handler attribute is not supported; however, the event itself is supported since Firefox 42. The event can be listened to via mediaSource.addEventListener('sourceclose', function() {});. See bug 1689222.
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
2.0
onsourceended
53
17
42
11
Only works on Windows 8+.
15
10.1
4.4.3
33
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
2.0
onsourceopen
53
17
42
11
Only works on Windows 8+.
15
10.1
4.4.3
33
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
2.0
readyState
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
33
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
2.0
removeSourceBuffer
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5
setLiveSeekableRange
62
17
No
No
49
10.1
62
62
?
46
13
Exposed in Mobile Safari on iPad but not on iPhone.
8.0
sourceBuffers
23
12
42
11
Only works on Windows 8+.
15
8
4.4.3
25
41
14
13
Exposed in Mobile Safari on iPad but not on iPhone.
1.5

BCD tables only load in the browser

See also

© 2005–2021 MDN contributors.
Licensed under the Creative Commons Attribution-ShareAlike License v2.5 or later.
https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API