The architecture everybody reaches for first.
The brief sounds simple. A viewer sits in a virtual space with several screens around them, each showing a different live camera from the same event — a broadcast cut, an in-car view, a pit-lane camera, a trackside angle. They glance between them the way you would in a real control room.
The obvious implementation is one video player per screen. Six screens, six streams, six players. It works immediately in the editor on a desktop machine, which is exactly why it is a trap.
The naive architecture
Stream 1 --> Decoder 1 --> Texture 1 --> Screen 1
Stream 2 --> Decoder 2 --> Texture 2 --> Screen 2
Stream 3 --> Decoder 3 --> Texture 3 --> Screen 3
Stream 4 --> Decoder 4 --> Texture 4 --> Screen 4
Stream 5 --> Decoder 5 --> Texture 5 --> Screen 5
Stream 6 --> Decoder 6 --> Texture 6 --> Screen 6
Works on a desktop. Falls over on a headset.It falls over because each of those rows is multiplied by six on a device with a fraction of the resources you tested against.
Hardware decoders are the hard limit, not bandwidth.
Most people reason about this in terms of network bandwidth, and bandwidth is genuinely a factor. It is not usually the one that stops you.
A standalone headset has a small, fixed number of hardware video decoders. They are a physical resource, not a software abstraction that scales with effort. Exceed them and one of three things happens: the platform silently falls back to software decoding and your frame rate collapses, additional players fail to start at all, or playback becomes intermittent in ways that are maddening to reproduce because they depend on what else was running when the session began.
Underneath that sit two more costs that scale with stream count rather than pixel count:
- Memory. Each independent player holds its own decode buffers and its own output textures. Six players is six sets, and headset memory budgets are unforgiving.
- Upload bandwidth. Every decoded frame has to reach the GPU. Six separate texture uploads per frame is six times the traffic of one, and memory bandwidth on mobile-class silicon is one of the first things you run out of.
The realisation that changes the design is that all six views were coming from the same event, cut at the same moment, and destined for the same frame. There was never a good reason for them to travel separately.
The mosaic: one texture, several UV rectangles.
Instead of six streams we received a single high-resolution matrix feed — every camera view composited into one grid by the broadcast side, before it ever reached the application.
That feed is decoded exactly once. Each virtual screen is then a quad with a material that samples a different rectangle of the shared texture.
The mosaic architecture
+---------------------------+
| Feed 1 | Feed 2 |
|-------------+-------------| one matrix feed
| Feed 3 | Feed 4 |
+---------------------------+
|
v
ONE hardware decoder
|
v
ONE shared texture
|
+---------+---------+---------+
| | | |
UV rect UV rect UV rect UV rect
| | | |
Screen 1 Screen 2 Screen 3 Screen 4Everything that previously scaled with the number of screens now costs the same as one stream. One decoder occupied instead of six. One set of decode buffers in memory. One texture upload per frame. Adding a seventh screen showing an existing feed costs a quad and a material — effectively nothing.
It also fixes a problem nobody had listed as a requirement: the views are now perfectly synchronised with each other by construction. Six independent players drift, and there is no clean way to pull them back together. Six regions of the same frame cannot drift, because they are the same frame.
The problem you have just created.
This is the part that gets left out of the architecture diagram, and it is where the real engineering time goes.
By moving from six streams to one, layout stopped being a property of the video pipeline and became data the application has to interpret correctly. The decoder no longer tells you which pixels are which camera. A set of UV coordinates does — and UV coordinates are quietly wrong in ways that never throw an exception.
| What goes wrong | How it presents |
|---|---|
| Rectangle offset by a small amount | A thin band of the neighbouring feed along one edge, easy to miss on a curved screen at distance |
| Width or height wrong | Correct footage at the wrong aspect ratio — cars subtly too tall, and nobody can say why it looks off |
| Two rectangles transposed | Swapped cameras. Everything looks perfect and the labels are lying |
| Rectangle extends past the region | Overlap, showing part of an adjacent feed inside the frame |
| Layout changed upstream | Every screen silently wrong at once, from a change nobody in the application team made |
None of these fails loudly. All of them look, at a glance in the editor, like video playing. That is why the work that mattered was not the shader — it was the tooling around it: a way to declare each broadcast layout as data, validate the rectangles against the source dimensions, and inspect every configuration visually before it went near a headset.
The shader took an afternoon. Being certain the rectangles were right took considerably longer, and was worth more.
If you build this, build the layout validator in the same week. The alternative is discovering a transposed camera pair during a live broadcast, which is a category of problem that no amount of engineering skill fixes at the time.
When the mosaic is the wrong answer.
It is not a universal architecture. It is the right answer under a specific set of conditions, and it is worth being able to recognise when they do not hold.
- The feeds must be genuinely simultaneous. The whole benefit comes from them sharing a frame. If viewers select one camera at a time and the others are never on screen, a single switchable stream is simpler and gives that one view the full resolution.
- You need control of the upstream composite. Somebody has to produce the matrix feed. If you cannot influence the broadcast side, you cannot use this at all.
- Per-view resolution is divided, not multiplied. A 4K matrix carrying four views gives each view roughly 1080p. If one view needs to be the hero at full resolution, it probably wants its own stream alongside a mosaic of the rest.
- The layout has to be stable enough to validate. If the grid changes shape frequently and unpredictably, the validation burden can outweigh the saving.
Where those conditions do hold — and for multi-camera live event coverage they usually do — the mosaic is not a clever optimisation. It is the difference between an application that runs on the target hardware and one that does not.
What to settle before you commit.
Four questions, answered early, decide whether this architecture will hold:
- How many hardware decoders does the target device actually have, and how many is the rest of the application already using? Not the number you assume. The number you measure on device with everything else running.
- Who owns the matrix layout, and how will they tell you when it changes? This is a process question, not a technical one, and it is the one most likely to break you in production.
- What resolution does each view need to be readable at the distance and screen size you are placing it? Work backwards from that to the matrix resolution, not forwards from whatever the source happens to be.
- How will a wrong layout be caught before a live audience sees it? If the honest answer is that someone will notice, you do not have an answer yet.
Get those four right and the implementation is comfortable. Get the fourth one wrong and the architecture is still correct, but the delivery is not.
What this means for a buyer.
Start with the business decision, audience, and evidence the project must produce. Simam Digital can turn that into a focused discovery, prototype, MVP, or production roadmap across AI applications, SaaS platforms, digital twins, real-time 3D, XR, and interactive systems.
