We get asked some version of this every week: “We already stream 4K live. How different can live Apple Immersive Video encoding be?”
Very. This post walks through why, and what a live Apple Immersive Video (AIV) encoder actually has to do.
If you want the short product version, it lives on our live Apple Immersive Video encoding page.
Start with the pixel math
Encoders are sized by how many pixels they have to get through every second. Here is how live AIV compares to formats most broadcast teams already know:
| Format | Resolution | Frame rate | Pixels per second |
|---|---|---|---|
| 1080p60 | 1920 x 1080 | 60 | ~124 million |
| 4K60 | 3840 x 2160 | 60 | ~498 million |
| 8K60 | 7680 x 4320 | 60 | ~2 billion |
| 16K immersive ingest | 16,320 x 7,200 | 90 | ~10.5 billion |
That last row is the feed we ingest for Spectrum Front Row. It is roughly twenty times a 4K60 broadcast, and it does not pause.
On demand, a big frame just means a longer encode. Live, every frame has a deadline. At 90 frames per second the encoder has about 11 milliseconds before the next frame pair shows up.
The codec: MV-HEVC
Apple Immersive Video is stereoscopic, and Apple requires stereo video over HLS to be MV-HEVC. One eye is encoded as a regular HEVC layer and the other is encoded as a second layer that predicts from the first.
That saves bitrate, but it ties the two eyes together. You cannot split the job across two unrelated encoders and glue the results back together. We cover that side of the problem in Live MV-HEVC.
The metadata that has to survive
A live AIV stream is more than video. Three things need to reach the headset intact:
- The venue description. This is the AIME file. It holds the lens calibration for the camera that shot the footage, and Apple Vision Pro uses it to project the image. Wrong or missing calibration means a picture that looks subtly bent, which in a headset is not subtle at all.
- Projection and layout signaling. The playlist has to say the stream is stereo and uses the AIV projection.
- Color and audio. AIV is HDR, in P3-D65 with the PQ transfer function, with Apple Spatial Audio alongside it.
If you have read our guide on how to stream Apple Immersive Video, this will look familiar. It is the same set of requirements as on demand. The difference is that nobody gets to open the Apple Immersive Video Utility and fix things by hand before the stream goes out.
What the stream looks like
The output is HLS. A variant line from an AIV multivariant playlist looks like this:
#EXT-X-SESSION-DATA:DATA-ID="com.apple.hls.venue-description",URI="primary.aime"
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=99766343,BANDWIDTH=110318142,AUDIO="audio",VIDEO-RANGE=PQ,CODECS="hvc1.2.20000000.H153.B0,apac.31.00",RESOLUTION=4320x4320,FRAME-RATE=90,REQ-VIDEO-LAYOUT="CH-STEREO/PACK-NONE/PROJ-AIV"A few things worth noticing:
REQ-VIDEO-LAYOUTcarriesCH-STEREOandPROJ-AIV. That is the signal that this is Apple Immersive Video and not a generic stereo stream.- The venue description is referenced as session data.
- The delivery resolution is 4320 x 4320 per eye at 90 fps. Apple recommends a ladder with average bitrates from about 25 to 150 Mbps.
So the encoder is not only compressing a huge frame. It is producing several of these variants at once, in step with each other, so the player can switch between them.
What to ask of a live AIV encoder
If you are evaluating how to do this, these are the questions that matter:
- Can it ingest the camera directly? For the Blackmagic URSA Cine Immersive that means ProRes over SMPTE ST 2110, at a throughput most encoders were never built for.
- Does it hold 90 fps without dropping frames? A dropped frame on a TV is a glitch. In a headset it is discomfort.
- Does it produce a full ladder live? One bitrate is a demo. A ladder is a product.
- Does the metadata come through? Venue description, projection, HDR, audio.
- What happens when it breaks at 6pm on game day? Ask who picks up the phone.
How we do it
This is what we built Zeus for. It is a 3U system that ingests ProRes 2110 at up to 16K and 90 FPS with 100 Gb/s of sustained throughput, encodes MV-HEVC in real time, and delivers to SpatialGen or the cloud and CDN of your choice.
It is the system behind Spectrum Front Row, the first Apple Immersive NBA experience: five cameras, 180 Mbps output, and under 20 seconds glass-to-glass. The Spectrum case study has the full story.
If you are planning a live immersive production, get in touch. We are always happy to talk through a workflow.
SpatialGen
SpatialGen
contact@spatialgen.com