Is Live 360 Video the Killer Application for VR?

At 360 Designs we think a lot about the future of VR, and lately we’ve come to some pretty startling conclusions.

We recently estimated that the potential market for live 360 video could be in the trillions of dollars (with a “T”), in about 10 years time, $5 trillion to be precise.

As far as we know there are few market analysts who are covering live 360 video, and awareness of its potential seems a little lacking.

Our article sets out the case for live 360 video, and why we think it could emerge as one of the largest industries on the planet, in just a few year’s time.

What is Live 360 Video?

Live 360 Video… is basically teleportation.

When you can put on a VR headset, and connect live to a 3D 360 video camera anywhere in the world, you virtually remove yourself from where you are, and place yourself exactly where the camera is. You experience exactly what it would be like to be there, without actually being there.

The opportunity to stream live 360 sports, concerts, festivals, awards shows, meetings, news events, family gatherings, other people’s lives, is literally endless.

We believe people will ‘pay to teleport using live 360 video’, similar to how they pay for premium entertainment content now.

Any event where people currently pay a ticket price for entry, can expand its potential (paying) audience to the entire world, using live 360 video.

With live 360 video, there becomes an infinite potential audience… for anything.

$5 Billion for a single tennis match?

Imagine the 2025 Wimbledon Tennis Championships, ten years from now…

100 million tennis fans worldwide (1 in 10 tennis fans) pay $49 pay-per-view VR to ‘virtually’ attend the Men’s Singles Final, teleporting live into Centre Court, using connected VR headsets. They get to choose from the best seats in the house, in hyper realistic live 360 video. It’s like they’ve teleported there, but they haven’t had to leave their couch.

We think it’s not unrealistic to imagine tennis fans from around the world paying £29 to purchase a ticket on their smartphone to travel there instantly, virtually, and immerse themselves in the event live. To get a seat they could never have in real life… up close and personal with the players, with super slow motion 360 replays on tap, minority report-style in-depth stats windows, and the ability to ‘go for a walk’ outside the courts. It might even be ‘better than the real thing’, in some ways.

We think it’s within the realms of possibility, and we’re developing technology to make it a reality.

Wimbledon is just one event of course… How many other events like Wimbledon are there? Where ticket demand exceeds supply, where geography and physical (arena) size limits the potential audience, where cost is prohibitive for large swathes of people, or where the effort to experience something exceeds people’s willingness or ability to do so.

It’s estimated there will be 8 billion smartphones by 2025, basically every person on the planet. Average income per person globally in 2025 will be an estimated $1,175 per month, including children and seniors. Average spending on entertainment by individuals is currently around 5%. We think ‘teleportation’ could represent 4% of people’s income, or around $50 per month in 10 years time, which would make live 360 video a $5 trillion industry.

How does Live 360 Video Work?

The beauty of live 360 video is that the technology you need to experience it is already in your pocket. Your smartphone’s screen, and the computer and motion sensors beneath it, are the only electronics you need for live 360 video.

A mobile VR headset, which places a couple of lenses in front of your phone, and makes it easy to wear on your head, rounds out the package needed. The cost of these mobile VR headsets will trend towards zero, and before long they’ll be giving them away for free (they already are…)… and whenever you upgrade your phone.

For ‘believable, immersive VR’, where you stop noticing the pixels, we feel you need around 2k x 2k per eye resolution. That’s a 4k smartphone screen, which are just coming to market now.

In 10 years time we’ll likely be at 16k or 32k screen resolution, the quality of which will be virtually indistinguishable from real life, even if it is blown up to fill your entire field of vision at distance, which is effectively what a VR headset does.

Delivering a believable ‘teleportation VR’ experience will not be without its challenges, but the component technologies are already in place:

“3-Axis” 3D 360 Video Capture

Illustrative renders. No partnership implied.

Illustrative renders only

We’ve developed the world’s first professional 360 VR camera system that is capable of capturing sufficient 3D visual data, for recreating a full 3D image in all 3 axes of head movement – i.e. pitch, roll and turn – and with some degree of positional movement (6 degrees of freedom – 6DOF). Basically you need a camera that can ‘shoot in all directions’, and from sufficient points of view, that a computer can recreate left and right eye positions anywhere within its ‘sphere’. Other cameras are either monoscopic 360, or are not full 360 x 360 (called 360 x 180), or do not have sufficient coverage to recreate 3D in all 3 axes of head movement, let alone 6DOF. In the future, we can expect to see broadcasters and Internet companies get involved in broadcasting live VR video events (aka teleportation). Their cameras might look like these above.

Availability: December 2015

“3 Axis” Full 3D Stitching

The video from all the different camera heads within a 360 camera needs to be stitched together and presented to the viewer realistically. This is known as stitching. We feel stitching for full 3D should be at the headset itself, although other methods may be possible.

We’ve developed a patent pending approach to solving the problem of stitching full 3D 360 video, i.e. 3 axis 3D, others are working on similar technology.

The answer lies with computational photography, similar to the “Bullet Time” technique popularized in The Matrix movies, but which appeared 4 years earlier in the (astonishing for 1995…) Like a Rolling Stone video.

Essentially our approach is to keep the footage from all the cameras in a VR rig separate, all the way to the VR headset. Once it gets to the VR headset, the VR headset computes the necessary interpolated left and right eye images, based on where the user is looking, using predictive smarts and head tracking, and computational photography ‘bullet time’ type approaches. This differs substantially from existing approaches which pre-stitch the 360 view, and which can’t adapt for parallax, or handle 3D 360 video in anything other than one axis.

Doing this intensive processing will require insane calculations at the client end, but processors get faster.

In the meantime, live mono 360 video and live single-axis 3D 360 video is possible now.

Availability: Within 2-4 years (on high end equipment)

Live 360 Video Streaming

Streaming retina-grade live 360 video is not much different technology-wise to streaming ordinary Internet video, aside from the fact it takes a lot more bandwidth. By 2025, fibre to the home will likely be prevalent, and we feel even 100+ Mbps connections, which some people have already, would give an astonishingly life-like teleportation experience. Streaming forty-two 2.5 Mbps HD streams from the 42 cameras in our rig would represent 105 Mbps. This is without optimizations that come from feeding back orientation data along the pipe (where the user is looking), and employing low res streams in ‘out of sight’ areas.

Availability: Now

Smartphones with 4k+ screens

As discussed above, there will be 8 billion smartphones in use by 2025, according to Huawei, basically every person on the planet. The smartphone receives the live stream, handles payment, and presents it to the user.

From a development standpoint, rights owners and content producers should be thinking about delivery of live 360 video through mobile applications. Doing live 360 VR as mobile applications allows use of the existing payment mechanisms in the App Stores. Other channels for distribution will develop and co-exist too.

Availability: Now (quality will improve)

Mobile VR Headsets

These already exist such as Samsung Gear VR, Google Cardboard and Zeiss VR. By 2025 expect these to be highly refined, comfortable to wear for long periods, and far more tightly integrated with the (then) current phones.