It's not just that, it's the whole system.

The human brain has a perception rate of motion up to 300 frames per second and visual data up to 1000 frames per second. Now it's non-sensical to think of the human eye and brain like a CMOS image sensor and CPU because they simply do not run on the same principles but for the sake of argument we'll take the upper limit of 1000 frame per second because the brain relies on an intuitive sense of hand-eye position that needs to be processed unconsciously for it to react fast enough to environmental stimulation.

So that's 1000 times a second minimum (1kHz) for the whole VR system to measure the change in relative position of the controller to the headset in 3 spacial dimensions, 5 degrees of freedom, the acceleration of that change, packet up these numbers, transmit them via a radio link to the headset, unpack that data in multiple processing threads all waiting for their sprint in a CPU core, get repackaged for other threads a dizzying amount of times, be used in calculations for the game's physics engine, which then produces graphics data shoved through a HUGE buffer to the GPU before that sends the appropriate electrical signals to 100,000 of nanoscale LED lights to shine into your retinas.

So yeah, it might suck a bit of power and it's a fucking miracle that GNERATIONS of engineers around the world have contributed incalculable hours of their lives to ensure that your beat saber or goon session last just slightly longer than your stamina!