Shooting a dawn scene for 11 hours or moving the mountains behind the actors are some of the magic tricks introduced by Virtual Production in filming The Mandalorian. Here is how it’s being done!

A peek behind the scenes of The Mandalorian, in February 2020, revealed how Virtual Production was being used to shoot the saga. Over 50 percent of The Mandalorian Season One was filmed using this ground-breaking new methodology, eliminating the need for location shoots entirely. Instead, actors in The Mandalorian performed in an immersive and massive 20’ high by 270-degree semicircular LED video wall and ceiling with a 75’-diameter performance space, where the practical set pieces were combined with digital extensions on the screens. But it all started almost half a century ago, when George Lucas imagined a space-opera in the sky.

When George Lucas returned to the Star Wars saga, in 1997, with what was to be The Star Wars Trilogy Special Edition  a theatrical anniversary edition of the original trilogy, in order to celebrate the 20th anniversary of the release of Star Wars: Episode IV A New Hope, he could not have guessed that decades later the Star Wars universe would still be the test bed for some of the most exciting developments in Virtual Production: The Mandalorian is the new playground for all the magic Hollywood can deliver.

The 1997 renewal of the original saga – which started in 1977 – was done by digitally remastering the image and sound to, as George Lucas said in a September 2004 interview with the Associated Press, “finish the film the way it was meant to be”.  The Special Edition versions introduced special effects which were not viable when the films were first released and added new digital sequences made possible by the improvements in CGI technology and other production techniques that occurred in the twenty years that passed since the original trilogy was released.

From the Dykstraflex to Virtual Production

A name always connected with the continued quest of “what’s possible” by George Lucas is Industrial Light & Magic (ILM), the motion picture visual effects company that the filmmaker founded in 1975. A division of the Lucasfilm film production company, the birth of ILM coincides with the start of production of Star Wars. ILM is responsible for many of the developments that came after, from the computer-controlled motion camera Dykstraflex — named for lead developer John Dykstra – to THX sound quality or the evolution of the original blue screen composite shot technology, used for so many of the space battles in the saga.

Industrial Light & Magic (ILM) is now part of Disney, as the company bought Lucasfilm in 2012, and acquired ILM in the process. Through that process, ILM continues to be involved with the Star Wars saga, in its new chapter, The Mandalorian, which is a test bed for the technologies now available to filmmakers. The video published last year showed a behind the scenes look at the groundbreaking virtual production technology used on The Mandalorian, Season One.

Digital 3D environments created by ILM played back interactively on the LED walls, edited in real-time during the shoot, which allowed for pixel-accurate tracking and perspective-correct 3D imagery rendered at high resolution via systems powered by NVIDIA GPUs. The  environments were lit and rendered from the perspective of the camera to provide parallax in real-time, as if the camera were really capturing the physical environment with accurate interactive light on the actors and practical sets, giving showrunner Jon Favreau, executive producer and director Dave Filoni, visual effects supervisor Richard Bluff, and cinematographers Greig Fraser and Barry Baz Idoine, and the episodic directors the ability to make concrete creative choices for visual effects-driven work during photography and achieve real-time in-camera composites on set.

A game engine, Unreal, and The Mandalorian

Four years ago, in 2018, Industrial Light & Magic introduced the use of real time rendering (with Unreal Game Engine from Epic Games) and digital LED displays as a virtual set in the production of The Mandalorian. The end-to-end virtual production solution, named ILM StageCraft, is a production-hardened technology that provides a continuous pipeline from initial exploration, scouting, and art direction, traditional and technical previsualization, lighting, and of course, real-time production filming itself, with the innovative StageCraft LED volumes.

As ILM noted at the time, “the technology and workflow required to make in-camera compositing and effects practical for on-set use combined the ingenuity of partners such as Golem Creations, Fuse, Lux Machina, Profile Studios, and ARRI together with ILM’s StageCraft virtual production filmmaking platform and ultimately the real-time interactivity of the Unreal Engine platform.”

“We’ve been experimenting with these technologies on my past projects and were finally able to bring a group together with different perspectives to synergize film and gaming advances and test the limits of real-time, in-camera rendering,” explained Jon Favreau adding, “We are proud of what was achieved and feel that the system we built was the most efficient way to bring The Mandalorian to life.”

The Mandalorian, Season Two, and StageCraft 2

This April, Industrial Light & Magic returns with another video, this time revealing the key aspects of the Virtual Production of The Mandalorian, Season Two. For the second season of Lucasfilm’s hit Disney+ series, The Mandalorian, Industrial Light & Magic reengineered their StageCraft virtual production platform rolling out version 2.0 in which ILM introduced among other things, Helios, Industrial Light & Magic’s first cinematic render engine designed for real-time visual effects.

According to ILM, “engineered from the ground up with film and television production in mind, Helios offers incredible performance, high fidelity real-time ray tracing, the ability to rip through scenes of unparalleled complexity, all while leveraging ILM’s unrivaled color science, and was designed from the start to work seamlessly with ILM StageCraft. The purpose-built, production-hardened platform allows filmmakers to explore new ideas, communicate concepts, and execute shots in a collaborative and flexible production environment.”

Also, according to ILM, “the new stages are vast improvements over the original ground-breaking LED volume developed for the first season of The Mandalorian in 2018. Physically, the new stages are larger, utilizing substantially more LED panels than ILM’s original stage and also offering both higher resolution and smooth wall to ceiling transitions – this directly results in better lighting on set as well as many more in-camera finals. ILM’s proprietary solutions for achieving groundbreaking fidelity on the LED walls at scale allows for higher color fidelity, higher scene complexity, and greater control and reliability.”

5,000 visual effects shots

The tools StageCraft offers are only part of the process of filmmaking, and that’s the reason why ProVideo Coalition shares with its readers a third video, Behind the Magic: The Visual Effects of The Mandalorian Season 2. This new video, published last March, shows visual effects work on The Mandalorian created in all five of ILM’s studios (San Francisco, Singapore, Vancouver, London, and Sydney) as well as a contingent of other vendors under ILM’s supervision.

Industrial Light & Magic states that “the season’s 8 episodes encompassed nearly 5,000 visual effects shots in addition to all of ILM’s real-time effects work done for use during principal photography. The effects team leveraged virtually every trick in the book, from miniatures and motion control to traditional puppeteering, advanced animatronics, spectacular special effects, and photo-real CG. Here’s a sampling of ILM’s post visual effects work done for the series that garnered ILM 13 Visual Effects Society Award nominations.”