Short answer: Pick media time as the authority and derive everything else from it. In an application combining video, telemetry, a digital twin, UI and replay controls, every subsystem arrives at a different rate and none of them agrees with the render loop. If each runs on its own clock the result is not slightly out of step, it is incoherent: numbers change before the footage they describe. Treating video playback position as the single source of truth — and having telemetry, twin state and overlays query it rather than run alongside it — makes scrubbing, pausing and replay work correctly for free, because they are all just the same clock moving differently.

Five subsystems, five update rates.

Live event applications are usually described as video players with extra features. They are not. They are several independent real-time systems that happen to be presented in the same field of view, and the interesting engineering is entirely in the joins.

On the A2RL VR application the pieces were broadcast video, live race telemetry, a 3D digital twin of the circuit with car positions on it, UI overlays showing standings and timing, and replay controls over all of it. Every one of those updated at a different rate.

SubsystemUpdate rateNature
Broadcast video30fpsFixed, externally determined, occasionally stalls
Race telemetry6HzFixed, network-delivered, arrives in bursts
UI and overlaysPer rendered frameVariable, whatever the headset is achieving
Digital twinPer rendered frame, fed by telemetryInterpolated between sparse updates
NetworkingVariableLatency and jitter you do not control

Five rates, none of which divides evenly into any other. The naive approach lets each run freely and updates its own presentation. That produces an application where nothing is technically broken and the whole thing feels wrong.

Why frame time is the wrong clock.

The default clock in a game engine is frame time, and it is the natural thing to reach for. It is also the wrong authority here, for a reason that becomes obvious once stated: frame time is a property of your renderer, and none of these data sources knows or cares about your renderer.

Drive telemetry off frame time and it advances at whatever rate the headset happens to be achieving. Drive video off its own player and it advances at whatever rate the decoder achieves. The two are related only by coincidence, and coincidences drift.

The failure is not subtle when it happens in front of an audience. A speed readout drops before the braking it describes appears on screen. The car on the twin is through a corner the broadcast has not reached. Someone pauses, and the video stops while the telemetry keeps counting. Each of those individually reads as a bug in the number, which is why teams often spend a long time debugging the wrong subsystem.

Viewers do not perceive latency. They perceive disagreement.

A quarter of a second of uniform delay across everything is invisible. A quarter of a second between the number and the picture is glaring, because human perception is extremely good at catching two things that should have happened together and did not.

Media time as the single authority.

The design decision that resolves it: video playback position is the clock, and every other subsystem asks the video where it is rather than tracking time independently.

Free-running (what not to do) Video player --> screens Telemetry feed --> readouts four clocks, Twin update --> circuit model no agreement UI tick --> overlays Media-time authority media time (t) | +---------+-----+-----+---------+ | | | | screens telemetry twin overlays at t at t at t One clock. Everything else is a function of it.

Concretely, telemetry samples are buffered against their own timestamps rather than consumed on arrival. When a frame renders, the application asks the player for its current media time, then asks the telemetry buffer for the state at that time. The twin does the same. The overlays do the same.

Note that this makes the ordering explicit: telemetry is looked up, not applied. That single change is what turns a set of loosely coupled systems into one coherent one, and it is a smaller code change than it sounds — it is mostly the discipline of never letting a subsystem write its own state from a callback.

What the single clock buys you.

The synchronisation is the point, but three other things fall out of the design without extra work, and they are the reason this is worth doing properly rather than approximately.

Replay and scrubbing become trivial. If every subsystem is a function of media time, dragging a scrub bar is just changing the value of that function. The twin, the numbers and the overlays follow because they were never doing anything else. Build this any other way and scrubbing means writing bespoke seek logic for every subsystem, which is where most of the bugs in this class of application live.

Pause is correct by default. Pause stops the clock. Nothing else needs to be told.

Recorded and live sessions share one code path. A live event and a stored replay differ only in how media time advances. There is no second mode to build, and no second mode to test.

That last one has real delivery value. On a fixed-date project, the ability to test the entire application against recorded footage — repeatedly, deterministically, without a live feed — is worth more than most features.

Where interpolation is right, and where it is a lie.

Telemetry at 6Hz against a display running many times faster leaves gaps, and the gaps have to be filled with something. What you fill them with is a design decision with an honesty dimension, not just a smoothness one.

Interpolate continuous physical quantities. A car’s position between two samples was somewhere on the path between them; showing it moving smoothly along that path is a better representation of reality than teleporting it six times a second. The same holds for speed, and for anything else that physically cannot jump.

Do not interpolate discrete state. Gear number, lap count, position in the standings, flag status, pit status. These change instantaneously in the real world, and a value between two of them does not exist. Showing gear 3.4 is not a smoother reading, it is a wrong one.

And do not extrapolate past the last sample when a feed stalls. It is tempting, because it keeps the display moving, but it means the application is inventing data and presenting it with the same confidence as measured data. On a live broadcast that is a genuine credibility problem. Hold the last known value, mark it stale if the gap grows, and let the interface be honest about the fact that it does not currently know.

The rule, stated plainly.

Any application combining a media stream with live data should decide, before writing the first subsystem, which clock is the authority — and should then be strict about it, because the discipline is easy to erode one convenient callback at a time.

When there is video on screen, media time is almost always the right choice, for the simple reason that video is the subsystem a viewer is least able to be flexible about. They will accept data that lags. They will not accept a picture that stutters, and they will instantly catch a number that contradicts what they are watching.

Everything else in the application can be a function of that clock. The moment something starts keeping its own time instead, you have two clocks, and two clocks always eventually disagree in front of an audience.

What this means for a buyer.

Start with the business decision, audience, and evidence the project must produce. Simam Digital can turn that into a focused discovery, prototype, MVP, or production roadmap across AI applications, SaaS platforms, digital twins, real-time 3D, XR, and interactive systems.

Sources and further reading