The team at Epic Games who are dedicated to finding better ways for filmmakers to work, share their experience working with filmmakers to test the in-camera VFX toolset in Unreal Engine 4.27 release.

Recently, Epic Games and filmmakers’ collective Bullitt assembled a team to test out the latest in-camera VFX toolset in Unreal Engine. A short film and a Behind-the-Scenes showed what is possible to achieve, and ProVideo Coalition shared the experience with its readers. Now the extensive virtual production team at Epic who are dedicated to finding better ways for filmmakers to work, share their experience and goals.

In the first video in the series, the filmmakers talk about the huge benefits they witnessed when working with these latest updates to Unreal Engine’s in-camera VFX toolset. The entire team, including the director, producer, production designer, director of photography, and first assistant director, all agreed that what they experienced represented a quantum leap for on-set filmmaking.

Ryan McNeely, who acted as Director on the production test, said then that “as a director, time is your biggest enemy. And the thing about this, the speed at which we’re able to move and the amount of setups, that’s honestly the dream. The ability to find new shots on the stage is just incredible. We can really get to the core of what a scene is about, and then use the volume to help tell that story.”

Jump from filmmaking 1.0 to filmmaking 5.0

Everyone agreed that the set of tools available in Unreal Engine 4.27 represented a new step in terms of workflow. Filmmaker Anthony Russo, said: “One of the things that excites us most, of course, about this is that the fact that we can do in-camera choreography where all the elements of the frame are actually in concert with one another and organically working on one another to create a more visceral experience”, and Director of Photography Matthew Jensen noted that “It’s so much easier now to be able to make the changes in real time, see their effects, and then make adjustments from there,”  adding that “It allows for a happy accident. It allows for more spontaneity in the creative process.”

While there was some skepticism at the start, in the end everyone agreed that with the tools available it becomes possible to “go from shooting to projection in the same day. That’s the technology at its finest,” as Sean Harner, First Assistant Director said while Producer Diane Castrup stated that “If it was up to me, I would shoot like this on every production. I love it. I absolutely love it.”

Production Designer Quito Cooksey added “I don’t think there’s any turning back. When silent films are going to speaking, you better learn how to start speaking” while Director Joe Russo revealed the whole experience felt “like we’re moving from filmmaking 1.0, jumping to filmmaking 5.0—and I don’t see any other path forward that is nearly as exciting, compelling, or freeing for artists as this path.”

Turnaround times of minutes instead of days

The second video in the series starts with Kim Libreri, CTO at Epic Games, explaining the goal with the new version of Unreal Engine. He says “the main push with 4.27 when it comes to virtual production is quality, efficiency, and ease of use. So that really is what’s driving us right now– the ability for filmmakers to get what they want in the moment and not be beholden to some external process that takes away the control and their creativity whilst they’re actually making their movie.“

One of the tools included in Unreal Engine 4.27 release is GPU Lightmass (GPULM), a light-baking solution that precomputes complex light interactions from lights which have their mobility set to Stationary or Static, and stores that data in generated lightmap textures that are applied to scene geometry. GPULM significantly reduces the time it takes to calculate, build, and generate lighting data for complex scenes, offers new workflows with interactivity that enables you to edit the scene, then recalculate and rebuild lighting on-the-fly. This workflow is not possible with the CPU-based Lightmass system.

“GPU Lightmass is a new generation of our light making software,” says Jonathan Litt, Senior Product manager at Epic Games, “that runs entirely on the GPU, which has been a game changer. Where previously turnaround times might have been a matter of days or even longer, we’re aiming for turnaround times of minutes. We want you to be able to run these operations on set with interactive feedback as you’re changing the lighting.”

OCIO color management inside Unreal

“One of the big issues with color is consistency. And it’s critical to maintain that detail all the way through the virtual pipeline,” says Rod Bogart, a Senior Software Engineer and Lead Color Scientist at Epic Games, adding that “with the new workflows, the aim is to make turnaround times go from days to minutes, while OCIO color management in Unreal Engine allows to predict how the film will look. We’ve been using OCIO color management to help us do that.”

According to Rod Bogart, “We can apply different OCIO looks that you might do to footage, but we can also do that inside of Unreal. And so we’re able to do that to the viewport. People can get predictive about the way the thing is going to look. And at the same time, when they finally get the final product, that that also looks that same way because they’ve used the same lens.”

The set of tools present in the VFX toolset in Unreal Engine 4.27 release includes a tool called Switchboard, “used to ensure that every single one of the devices you see running this LED wall is able to remain in sync” and also a new tool called Stage Monitor, “which allows you to monitor for hitches or other notable events that may be occurring on the day, all stamped with time code”, according to Thomas Kilkenny, Lead Technical Artist at Epic Games.

Control everything from the palm of your hand

The team also developed a tool called Remote Control, a customizable web interface that connects directly to the Unreal Engine. Alejandro Arango, Virtual Production Lead Engineer at Epic Games says: “You can have it on any mobile device. You can have your iPad on set walking around, and you can be controlling the sky and changing the illumination on the live action on your set. You can be starting a playback of some animation. You can do anything you want from the palm of your hand. In fact, multiple people can be doing this at the same time. That is the power of Remote Control.”

“We’re developing tools quicker and quicker, adapting specifically to the needs of filmmakers so that we’re able to take the visions and dreams of any aspiring filmmaker from the biggest names we could possibly imagine to a film student looking to create their next short film, and execute that in a way that it’s accessible to really anybody. And that’s extraordinarily exciting” says Thomas Kilkenny.

If you’d like to try these features out for yourself, Unreal Engine 4.27 is now available in Preview. You can download the latest release from the Epic Games launcher.