The New Yorker recently published an article about virtual reality, and I felt the need to publish a response. I did so via the Cinematography Mailing List‘s VR listserv, and I feel that an edited version of those comments are coherent and meaningful enough to republish here.

First, please take a look at the article in question.

The end of the article casts the filmmakers as, perhaps, a little arrogant about their new medium:

“For all of human history, art, music, storytelling, religion—those have been our modes for communicating the incommunicable,” she said. “But what if there were a way to know not an abstract version of my experience but what I’m actually feeling?”

This is impossible, with or without VR. You can never know exactly what another person is feeling. You can empathize, but that’s an emotional reaction to what you perceive another person’s feelings to be based on physical and verbal cues. That works just fine in 2D.

And, when you do want to communicate an emotion, it is—so far—much easier to do this via current 2D filmmaking processes, where the communication happens not just via acting and dialog but through lighting, composition, editing… there are so many creative ways to tell stories through the traditional filmmaking process, and I’m seeing a lot fewer options in VR.

So far the live action VR projects I’ve seen have been largely unsuccessful. The only one I’ve liked is Jaunt’s “Black Mass,” but it moves very quickly, has lots of lighting gags, simple but good performances, and “edits” through the use of blackouts (both lighting and camera). It’s also very, very short. That’s the other bit: VR movies tend to be short, because viewers get motion sick after a while. Also, looking around while wearing heavy goggles tends to be a bit wearing.

http://www.youtube.com/watch?v=vZ8t6IgPd88

I’ve seen attempts to edit in VR that fail miserably. A lot of VR puts all the action into a “TV safe” area directly in front of the viewer, and there’s very little else of interest to see elsewhere, and often no time given to the viewer to explore. Many of the actors in the live action VR pieces I’ve seen deliver marginal performances at best, and it’s clear that this is a medium not just for stage actors but for exceptional stage actors because we’re seeing the their performances in our new role as passive participants, not from a distance as an audience.

Lastly, I’d argue that the one thing people DO want to see in their media is abstraction. Reality is often boring, or at least not exciting as an entertainment experience. 2D filmmaking tells you exactly what it wants you to look at—or hides it from you to create tension—and frames and lights it in such a way that creates powerful subtext to the dialog and performances. Live action VR is often unlit, using cameras that largely function in an “auto” mode, and the performances, delivery, blocking and story all have to be tight and delivered perfectly in order to create a captivating and interesting experience.

If humans didn’t like abstraction then Instagram would not be the hit that it is.

I think that VR storytelling is going to be a combination of live action performances and video game technology. There has to be some way of showing interaction between characters while also pausing and letting the viewer look around the rest of the scene for clues that fill in the rest of the story. Nearly every live action VR project I’ve seen uses about 90 degrees of the field of view and neglects the rest, and at that point we might as well focus on traditional cinema. It’s clear that I’m supposed to look in one direction all the time, and if I explore in other directions then I miss out crucial bits of the story.

I also think that one of the strengths of VR storytelling is repeat experiences: as you can’t see everything at once, why not build a scene with so much to it that you can’t see it all at once? And, like reality (but exaggerated) maybe everything that happens within the frame has meaning for the story. Maybe the viewer looks at different things in each pass, so the story’s meaning changes with each viewing. I love movies that make me watch them more than once to REALLY understand what’s going on, so why not a five or ten minute VR experience that does exactly the same thing?

The last thing VR will be is “a way to know not an abstract version of my experience but what I’m actually feeling.” I can’t feel someone’s emotions just because I feel like I’m sitting in the same car with them while they’re acting (and, based on the VR I’ve seen so far, often badly).

Back to The New Yorker:

She looked directly into the camera. “Under your seats is a headset that will change the very nature of what it means to be human. Under your seats is the end of your individuality. Put it on and you’ll never want to take it off. Good luck.”

Yes, good luck changing human nature and convincing me that losing my individuality is not only possible but desirable, given that you’re presenting me with a VR experience that gives me some control—I can look around—but removes all other decision-making capabilities. As one reviewer put it, current VR provides the same experience as a “brain in a jar.”

I think there’s a market for VR, and I’m looking forward to exploring it, but so far I just keep seeing live action mistake after mistake after mistake. Right now it’s great for experiential material (virtually travelling to exotic places, or experiencing POVs of extreme sports) as well as video games (where both interaction with the environment and pausing to explore are possible), and there are certainly some uses beyond entertainment, but no one has cracked the storytelling aspect and I don’t quite understand why—other than the fact that VR will require much more planning and rehearsal than I see in any of my other work. VR performances will have to be perfectly believable, with impeccable dialog, a brisk pace, and clever blocking that directs the viewer’s attention without being obvious. And even then there will be people like me who will look everywhere but where we’re supposed to look, and there’ll have to be enough meaningful material in the environment to keep me interested.

This is where the artistic side of filmmaking comes into play. “Black Mass” used lighting brilliantly not just to set mood but to affect transitions between events and environments. Blocking and staging of set pieces is also important: as VR takes place in a 3D world it’s important to define the space and create an environment with which the actors interact, and the DP should participate in that process as part of our job is to create and enhance depth.

Color is probably more useful than ever in creating space and separating objects and people from foreground and background elements. It can also set the mood, both in the color of the lighting and set decorating. Shadowing a set on one side and lighting it on the other can guide the viewer through
a set, as can making one side cool and the other warm. Shifting that balance through lighting can help to redirect the viewer’s attention. (Long ago, when I lived and worked in Los Angeles, I learned that the worst parts of downtown were lit by sodium vapor lighting, and the safer parts were lit by mercury vapor lighting. It seemed the homeless population preferred living in areas of warm/yellow light rather than cool/blue light. That’s how powerful color can be.)

Framing takes a back seat to blocking, but otherwise lighting is more important than ever. It will have to be built into the set in some clever ways, or hidden, or rigged outside the set (through windows and bounced off floors, for example). A great example can be seen in (forgive my pop culture/geeky reference) Firefly, where Mal tries to steal the first laser handgun ever invented, “The Lassiter.” The setting is inside a room that is
open to the outside, and the only light in the room is bounce light from the floor and from the exterior patio. It appears that the DP (the brilliant David Boyd) decided to light the set for 360-degree shooting by smacking all his lights into the exterior of the set and using the bounce to light inside:

The first shot on the set is, indeed, a 360 that shoots out through the large doorway to the exterior, which is blown out but retains enough detail to show that it’s an exterior patio.

This is the kind of clever lighting that would work well in VR, especially as VR wants the environment to be as believable as possible, and lots of interior environments are lit by ambient light from a window or sunlight bouncing off a floor. The problem is that we’re currently shooting with GoPros and similar cameras, which don’t yet have the best dynamic range or exposure control options. On film this scene looked great, but I doubt there are many VR rigs that could handle this well. Still, that day will come.

VR may be one of the hardest-to-use storytelling mediums ever: it could have tremendous potential when done right, but also be insanely hard to do right.

Meanwhile, I’m dying to have the chance to shoot VR but do it in a way where I can be “abstract” by using my skills to impose mood and meaning upon the image, rather than simply capturing it as most VR demos do. At a recent VR expo I ran across a live sports VR company that took pride in the fact that they send out their camera with a software engineer and a couple of techs, and no one with any photographic experience whatsoever. That may fly for live events but it’s not going to work for storytelling. Storytelling is, by nature, abstract: reality is often less interesting than an interpretation of reality, where meaning is emphasized or created in ways that don’t happen in real life. That’s where the artistry comes in, and if there’s one thing that current VR seems to lack it’s an appreciation of artistry over technology. So far, technology is the easy part.

Art Adams
Director of Photography