Coveted rendering addresses a growing challenge for VR headsets, providing crisp details for your eye's visual sweet spot – fovea – and a simpler, sharper version for your peripheral vision. Now engineers at Facebook Reality Labs have come up with DeepFovea, an AI assistant alternative that creates "plausible peripheral video" rather than actually making accurate peripheral images. The new process is called "coveted reconstruction" and Facebook says it achieves more than 14 times compression on RGB video without any significant degradation of user perceived quality.
When taking a video stream, DeepFovea only tests 10% of the pixels in each video frame, largely but not exclusively focused on the area where the user's eye is focused, represented by the lizard head above. By comparison, the peripheral area is sampled only with scattered dots that become less dense further from the eye's focus area. The system then uses trained generative opposing nerve networks to reconstruct each frame from the small samples, while relying on the temporal and spatial content of the stream to fill in details in a stable rather than jittery way.
As the images above show, the heavy but not fully sampled lizard head is essentially indistinguishable from frame to frame, while the adjacent tree bark in the "reconstructed" image is not as sharp or detailed as the "reference" pixels. But that's not the point. A traditional coveted rendering system would image these pixels as flat-shaded, low-resolution blocks, while DeepFovea preserves ̵
The main reason why DeepFovea is important is that it offers a superior combination of effect efficiency and image quality compared to standard coveted rendering. Facebook's claim is that the 14x reduction in rendering means that it will be able to deliver real-time, low-latency video to screens due to glare detection – a necessary step to build lightweight VR and AR headsets that display high-resolution graphics originally rendered in the cloud. Wall-mounted Oculus AR headsets throughout the day are said to be impractical until the energy consumption of the mobile chip drops as dramatically for real-time 3D mapping as we see for streaming video.
Facebook's Michael Abrash first hinted at the concepts behind DeepFovea last year at Oculus Connect 5, suggesting that in the future – at some point in the next five years – deep learning-based foveation and good eye tracking would be gathered to enable VR headsets with higher resolution as its prototype Half Dome. At Oculus Connect 6 this year, Abrash said the company will test the next-generation Half Dome hardware in its own offices before distributing it to the public.
Rather than keeping DeepFovea solely for itself while it works on the next generation headset, Facebook releases an example version of the network architecture for researchers, VR engineers and graphics engineers. The company will present the underlying research document on Siggraph Asia tonight and will make the samples available thereafter.