Building the ultimate display

Stanford’s Bernd Girod lays out a research agenda for immersive video

Justin Hendrix
Haptical

--

Last month Bernd Girod, Professor of Electrical Engineering at Stanford University, visited NYU Tandon School of Engineering and delivered a talk on technical challenges for delivering fully immersive virtual reality experiences. Girod’s research interests are in the area of image, video, and multimedia systems, and as an entrepreneur, he has been involved in numerous startup ventures, among them Polycom, Vivo Software, 8x8, and RealNetworks.

Ivan Sutherland’s Head Mounted Display, 1968

Harking back to virtual reality pioneer Ivan Sutherland, Girod points out that the idea of the “ultimate display” has always been the goal of virtual and augmented reality researchers. The goal of such a display would be total immersion- a photorealistic reality in full spherical view. As anyone working in VR knows, right now we are a long way from that vision.

First and foremost it’s about pixel density. To get to a retinal resolution in the central fovea, the region of the sharpest vision for the human eye, requires about hundred pixels per degree, Girod notes. If you multiply this number and spread this out over a full sphere, then it’s about 650 million pixels for a sphere.

“Imagine having to send 60 frames 60 times per second- then you get into the gigapixel per second range,” said Girod. “Thus the hardware challenges, transmission challenges, rendering challenges, processing challenges etc. Right now we are at 10 pixels per degree. Right now we are at about 6.5 million pixels in the sphere.” So to get to fully immersive video, we need 100x times the resolution.

It’s all about the pixels

Resolution is one challenge, says Girod, but there is a ‘wish list’ of other technical capabilities necessary to create fully immersive experiences. These include:

  • Cover full field-of-view with retina resolution
  • Stereo in all directions
  • Head motion response with low latency
  • Defocus cues for accommodation of the eye
  • Compact high-quality representation for storage and streaming that supports all of the above

The promise of fully immersive VR is enormous, argues Girod. The potential will take the technology to applications far beyond entertainment and gaming. In the near term, head-mounted displays will piggyback on the evolution of smartphone technology; thus spatial display resolution will be insufficient for a few more years.

Girod has some direction for researchers working on improving display technologies to achieve immersion:

  • Forget equirectangular: use cube mapping or content-adaptive for sphere to plane mapping;
  • For streaming, use viewport-adaptive representations: tiles or multiple 360 representations or something in-between;
  • Omnistereo stitching: slit camera viewpoint interpolation needs good depth maps;
  • Depth-augmented omnistereo can support limited lateral head motion and future near-eye lightfield displays.

“For engineers and researchers it’s great news because we can still do a lot of work until we reach that point where we can expect this to take us as a computer/consumer technology,” said Girod. “My thesis is until we reach that point virtual reality is still going to be a niche thing, a curiosity application, but after that it’s really going to take off.”

--

--

CEO and Editor of Tech Policy Press. Associated Research Scientist and Adjunct Professor at NYU Tandon School of Engineering. I live in Brooklyn, New York.