We have written a fair amount about MV-HEVC over the years: how to encode it with FFmpeg, how to encode it in HDR, and how to make HDR10 MV-HEVC play over HLS.
All of those are about files. This one is about live MV-HEVC, where the encode has to finish before the next frame arrives.
For the product side of this, see our live MV-HEVC page.
A quick refresher
MV-HEVC stands for Multiview High Efficiency Video Coding. For stereo video it stores:
- a base layer, which is one eye encoded as ordinary HEVC, and
- a second layer, which is the other eye encoded mostly as differences from the first.
Because the two eyes see nearly the same scene, the second layer is much cheaper than a full second video. The base layer also plays as a flat picture on a decoder that only understands HEVC.
On Apple devices, MV-HEVC is how stereo video is delivered. Apple’s HLS authoring specification requires it for stereo content.
What a file encode gets to do
When we encode a file, the encoder has luxuries:
- It can take ten seconds on a hard frame and nobody notices.
- It can look ahead to see what is coming and spend bits accordingly.
- It can make two passes.
- If something goes wrong, we run it again.
What a live encode has to do
Live takes all four away.
A fixed deadline. At 60 fps there are about 16.7 milliseconds per frame. At the 90 fps that Apple Immersive Video runs at, there are about 11. Both eyes have to be done in that window, every time.
No independent halves. The obvious way to go faster is to split the work. But the second view predicts from the first, so the two layers cannot be handed to two encoders that know nothing about each other. Parallelism has to be designed around that dependency.
Several encodes at once. Adaptive streaming needs a ladder. A live system is producing every rung simultaneously, and the rungs have to stay aligned so a player can switch between them mid-stream.
Rate control without a crystal ball. A live encoder cannot see the fast break or the confetti cannon coming. It has to hold quality through hard content using only what it has already seen.
It cannot stop. A three-hour event is 972,000 frame pairs at 90 fps. The system needs to handle the last one as well as the first.
Resolution makes all of it worse
Spatial video from a phone is small. Immersive video is not. The feed we ingest for Spectrum Front Row is 16,320 x 7,200 at 90 fps, which is about 10.5 billion pixels per second.
That is why live MV-HEVC at immersive resolutions is a hardware problem as much as a software one. The input alone is ProRes over SMPTE ST 2110 at a rate that needs a 100 Gb/s class network path.
Getting it to the viewer
Once encoded, live MV-HEVC is packaged as HLS like any other live stream: short segments, a playlist per rung, and a multivariant playlist on top. The stereo-specific part is the signaling. The playlist tells the player the stream is stereo with REQ-VIDEO-LAYOUT, and for Apple Immersive Video it adds the AIV projection and a reference to the venue description file.
HDR has its own traps. The ones we documented for HDR10 MV-HEVC over HLS apply to live streams just as much as to files.
Where we landed
We built Zeus to do this in real time. It ingests ProRes 2110 at up to 16K and 90 FPS, encodes MV-HEVC live, and can produce HEVC, H.264, and AV1 from the same feed for flat destinations. It runs on site, next to the cameras.
If your goal is specifically Apple Immersive Video, the next read is live Apple Immersive Video encoding, explained.
Questions about a live stereo workflow? Reach out. We like talking about this stuff.
SpatialGen
SpatialGen
contact@spatialgen.com