The first thing to understand is that “Virtual Production” comprises several different workflows, not just the LED wall made famous by the production of The Mandalorian. Most people are now defining Virtual Production as “production that combines virtual and physical filmmaking techniques.” Some of these workflows have been in play for decades, others are just emerging. Let’s take a look at some of the key areas
Previsualization (“Previs”)
Previsualization (“Previs” for short) is the process of creating a virtual version of a film before an actual shoot. Previs allows filmmakers to stage out action, block camera angles, and potentially create an edited animatic of entire scenes. It can significantly help determine if a scene “works,” providing visuals where a table read would only offer dialog.
Techvis
Techvis blurs with previs, and many filmmakers don’t even make a distinction here. Techvis is the process of planning a shoot virtually to ensure that it’s actually possible to film what the director wants. While previs in the strictest sense is focused on visualizing the storytelling, techvis informs on what it’ll take to get the shot. By simulating lenses and framing, techvis can determine whether a 28mm lens will be able to capture the full scene in a 12 foot wide room, whether a 3 foot wide dolly will actually fit through the door of the filming location, and what height of crane needs to be rented to achieve the desired establishing shot.
Postvis
Postvis uses similar filmmaking techniques to previs, but is designed to generate temporary visual effects. It comes into play after principle photography, compositing stand-in effects over the source footage. The purpose is twofold: to provide VFX studios with a better understanding of the director’s intent, and to provide temporary footage for the editor to work with while waiting for finals to come back from those VFX studios.
ICVFX
The term, “In-Camera VFX” has been proposed in an effort to differentiate the “Mandalorian”-style LED wall production from all the other types of virtual production. This is what most people think of when they hear, “virtual production.” ICVFX includes both LED volume and greenscreen workflows. It points to the idea that the effects are done in-camera rather than as a separate post-production phase, and that once the cameras stop rolling the visual effects work is done.
Theory and reality can be quite different here of course, and there are many situations where additional post-processing will need to be performed to correct artifacts or issues with the quality of the VFX captured during principle photography. These issues are driving the development of frame interleaving—aka Ghostframe—and other techniques (see below), allowing the flexibility of correcting issues more easily in post.
In either the case of an LED volume or a greenscreen, visual effects work (world creation, particle, fluid, and destruction effects, character animation etc.) is done before the day of the shoot. The effects are then projected on the LED panels, or—in the case of a greenscreen—composited in real-time and viewed via on-set monitors.
LED Volume workflows, aka “The Volume”
In an LED volume workflow, the CG environment is projected behind actors and foreground set pieces in real-time. This serves a threefold purpose: 1. The actors are illuminated by the virtual environment, creating a natural integration of foreground and digital background; 2. Reflective surfaces in the foreground receive a somewhat accurate reflective source from the panel images; 3. The actual virtual environment background can be filmed behind the actors and set pieces without requiring a greenscreen and the arduous compositing work involved in perfecting a final greenscreen composite.
Greenscreen ICVFX workflows
While greenscreen ICVFX lacks the automatic creation of reflections and lighting sources—and the ability for actors to feel immersed in the virtual environment—it does allow for post-production modification, since the background is not baked into the final image. Additionally, reflections can be applied either on-set using a tracked, adjustable UHD display or mini LED panel, or digitally in post by projection mapping the reflections over proxy geometry. Furthermore, DMX set lighting can provide a match to the virtual environment’s light sources by converting virtual camera samples into DMX signals.
It’s worth noting that while greenscreen ICVFX sets provide a cost-savings up-front, the continued cost of fixing green spill and general compositing issues (accurately matching foreground to background) can quickly outpace those savings.
The Camera Frustum and Spherical Projection
Parallax is how objects appear to shift against their background when viewed from different positions. It’s not possible to show a single image on an LED wall that looks perfect from every camera angle on a set. The compromise then is to generate a spherical projection from the center of the stage space and display the resulting image on the LED walls. This provides the desire result of illuminating subjects with the background lighting and providing a source for scene reflections.
However, the background image from this method won’t match the exact perspective of the camera filming the scene. To fix this, another video of the virtual world is generated specifically for filming the camera’s point of view. This is typically called the camera frustum view.
A camera frustum (not frustrum as it’s often misspelled) is a term used in computer graphics and 3D rendering to describe the 3D space or volume that a camera can “see” or capture within its field of view. The filming camera’s frustum is projected a little wider than the actual area captured by the camera lens, and feathered into the main spherical projection on the screen. The wider area provides a better continuation of lighting, and the feathering prevents any harsh changes that might be picked up in reflections on set.
The two image sources are combined. It looks a little weird on set, with the mismatch of the spherical projection and the camera frustum, but in-camera the final framing only sees the frustum image as a background behind the actors, while the actors and other set pieces are illuminated by and reflect the overall lighting provided by the spherical projection.
DMX Lighting and Pixel Mapping
DMX lighting is the cutting-edge virtual production lighting technology…that was invented in the 1980’s. It’s survived (like it’s sister technology MIDI) thanks to it’s bulletproof design—it just works. Most professional production LED light fixtures include DMX controls.
As you can imagine from it’s 80’s origins, the protocol is extremely primitive, supporting only 8-bit intensity changes (i.e., 8 bits-per-channel color) and 512 total channels of discrete data. Modern systems can double up channels to produce 16 bit-per-channel light signals, and variant protocols Art-Net and sACN can support multiple “universes” (the name given to a block of 512 DMX channels).
Pixel mapping (an emerging technique that has a different monikers depending on a lighting manufacturer’s marketing material) is the process of taking a view of the virtual world and converting it to DMX signals to drive a bank of DMX lights. The effect is something akin to a low-resolution LED volume, albeit one that can be interactively positioned and contains much lower metamerism (i.e. more faithful color reproduction) than an LED wall.
Pixel mapping can provide the missing interactive lighting solution to a greenscreen ICVFX stage. However it’s worth noting that DMX lighting control in general—and pixel mapping in particular—are still of significance to LED volume ICVFX. LED panels are designed to generate images, not to operate as light sources. So their particular flavor of LED wavelengths aren’t equipped to handle the lighting requirements of a film set. The result is unpleasant metamerism if your subjects get too close to the walls. Augmenting with DMX-controlled, high-quality production lights can augment the color of the scene coming from the LED panels.
Exhibition Servers and Real-Time Game Engines
Until the last few years, driving high resolution LED wall displays required niche and expensive dedicated playback servers with proprietary software. All this has changed with the advancement of real-time rendering in generalized game engines, like Unity and Unreal Engine. More specifically, it’s been Unreal Engine—the platform used to develop Fortnite—that’s been the driving force for this evolutionary sea change.
Epic Games has been aggressively pursuing filmmakers and expanding their game design platform, Unreal Engine, to become a generalized platform for digital content creation. The company’s CTO Kim Libreri has a legendary history as a visual effects supervisor at ILM, so it’s no surprise that Epic has understood the unique needs of the film industry when it comes to animation and rendering.
What’s just as interesting about Unreal Engine is that even with the massive investment Epic has made by developing custom software specifically for virtual production, the use of Unreal in filmmaking is completely free. That’s correct: you can use Unreal Engine on a $300 million blockbuster and not pay a dime in licensing to Epic. In fact, unless you are specifically creating a game with Unreal Engine you don’t pay royalties. (Even if you are making a game, you only start paying 5% royalties on gross earnings after the first $1,000,000.) The full version of Unreal Engine is available for free to download and start using to create virtual production, virtual humans, and all kinds of other weird and wonderful DCC applications.
It’s worth noting that Unity—the other big commercial game development platform—is also pursuing virtual production and was used on the seminal VP title, Disney’s The Lion King. Unity’s purchase of Weta Digital for over $1.6 billion, along with smaller acquisitions like Ziva Dynamics, was seen by many as an effort to catch up to Unreal’s lead in the sector.
That being said, the recent round of layoffs at Unity have reportedly had impact on the Virtual Production team, and it’s hard to know at this point whether Unity have given up on efforts to enter the market, or are simply regrouping. The former makes the most sense: Unity has a significant majority share of the lucrative mobile game dev market, and given that Unreal currently offers its virtual production tools for free, there is no obvious immediate ROI for all the investment required.
Unreal Engine
At some point Unreal Engine’s virtual production features require their own article, but here are some of the key points:
- Unreal has a visual scripting language called Blueprints that allows non-programmers (like 3D artists) to create custom behaviors and tools without engaging a full development team
- Unreal is based on C++ and the entire code base is open (not open-source in licensing mind you, but open to be read, modified, and compiled), so that developers can extend it with additional C++ and make changes to the base code if needed, without having to wait for Epic developers to address a critical issue.
- Unreal is constantly working to increase realism while rendering at 60 frames per second and beyond.
- The new combination of the Lumen lighting system and the Nanite polygonal streaming system in Unreal Engine 5 (UE5 for short) enables physically-based rendering (PBR) at quality levels previously unseen for real-time gaming
- Unreal includes software called nDisplay designed to drive multiple synchronized clusters of real-time rendering servers. This allows separate sections of an LED wall to be rendered at tandem in high resolution.
- For all its power (and “freeness”), Unreal Engine is first and foremost a game engine. As such, many of the workflows for Virtual Production are a little, “clunky.” This has led many systems integrators to build an additional interface layer over the top of Unreal Engine, simplifying workflow for end users. Keep this in mind if you expect to simply “plug in” Unreal Engine and start making movies.
Notch
Notch has been around for a long time in the live entertainment space, and became popular with performance artists, thanks to its node-based programming system (similar to Unreal’s Blueprint).
Notch’s main appeal over Unreal is its streamlined authoring environment. Unlike Unreal Engine that is first and foremost a game design tool, Notch is much more tailored to artists designing live experiences. The learning curve is much gentler and the artist isn’t fighting an interface cluttered with non-relevant toolsets and a workflow paradigm focused on creating packaged games.
On the flipside, Notch lacks Unreal Engine’s expansive features and almost limitless expansion capabilities. Notch tends to be run through TochDesigner or Disguise (below) rather than operating as a standalone media server environment.
TouchDesigner
Disguise
Technically, any workstation with the right CPU, RAM, and GPU specs can be used to power Unreal Engine and operate as a node in an nDisplay cluster to drive a portion of an LED wall, but Disguise servers are designed to exacting standards as something of a turnkey solution to quality rendering and playout.
Developed by the company Disguise (formerly known as d3 Technologies), the Disguise server is known for its ability to handle large-scale video processing and manipulation. It’s equipped with a combination of high-end GPUs, CPUs, and large amounts of RAM to efficiently handle real-time rendering and playback tasks.
Disguise can operate as a standalone content creation tool for virtual production, but in such cases focuses on “2.5D” workflows, where video and stills are projected on cards placed at different distances in the scene. The benefit of this is that when working with photoreal still and video assets, setup is not technically complex (i.e. anyone can do it) and modifications can be made in real-time without any offline rendering.
Disguise supports both Unreal Engine and Notch as the real-time engine to drive the hardware. Disguise also provides enhanced features like digital set extensions (see below).
StageCraft
StageCraft is ILM’s proprietary real-time engine for virtual production and specifically ICVFX. Unreal Engine was the used for the first season of The Mandalorian, but was replaced by StageCraft in season two. Evidentally ILM felt that an internal tool streamlined for virtual production was a better fit than using a generalized game development platform. It will be interesting to see if that investment pays off long-term. ILM have bet against commoditized general platforms in the past and lost; their internal compositing tool CompTime was replaced first by Shake, and then ultimately by Nuke. There have already been reports of compatibility issues with teams working on assets in Unreal and then needing to convert them for use in StageCraft.
Genlock
Part of the magic of displaying content on LED walls is distributing the workload to multiple workstations rending the scene. It’s critical that the frame of video being rendered from each computer is displayed at exactly the same as the others. If not, spatial lag, flicker., or tearing of the frame may be the result.
Each computer generating a portion of the LED wall image receives a sync signal from the master signal generator. When all frames of video are released at the same time they are said to be genlocked. Special sync cards are added (typically as Nvidia Quadro daughtercards) to the machines, while the cameras and LED display processors ((e.g. Brompton processor) also receive and are genlocked to the signal generator.
The timing alignment is so critical in an LED volume that the length of sync cables needs to be taken into account and delay compensated in order to ensure that everything fires in perfect synchronicity.
V.A.D.
Virtual Art Department is the name given to the crew responsible for designing virtual sets, as well as sourcing, scanning, modeling, and texturing the digital props that will appear in the scene. In many ways this is simply a fancy name for something that’s been taking place in animation and game design for decades: modeling, texturing, and scene layout. The “new name” is very much an effort to engage traditional art directors (here in Hollywood, the ADG, IATSE local 800) in the hybrid digital process of designing sets for virtual production.
Production designers and art directors have been dabbling in the digital dark arts for a few years now, using everything from Sketchup to CAD software to help design and layout live action sets. In some ways VAD is an expansion of these skills, though now the layout being created in the computer is the final product, not merely a representation of what will ultimately be built as a physical set.
Where VAD work departs from traditional animation and game design layout is in the blending of the virtual and the practical. Currently at least, virtual production sets include practical foreground stage props and these need to blend seamlessly with the virtual props. One way to achieve this is by using reality capture to create digital replicas of real-world prop pieces (think a barrel or crate for example) and then match the color and lighting of virtual background objects to the physical ones on the practical foreground set.
As AI and imaging tech continue to improve, we may find that physical props become less important, with digital foreground elements being superimposed over actors. Nonetheless, actors will always perform best with real set pieces to interact with, so it’s unlikely that we’ll ever see a complete shift to entirely virtual set design.
One last point worth making: the concept of the VAD is still very much in its infancy and Unreal Engine’s “packaged game” paradigm makes asset control and iteration very difficult. New versions of set pieces need to be manually reimported into a project rather than referenced. The Pixar-created open standard USD schema seems like it might be the long-term solution, but current production pipelines leverage version control systems like Perforce and Git (these are usually deployed in programming development environments rather than digital content creation) to help tame the asset creation pipeline. The takeaway: virtual production is still very much the Wild West with little standardization of workflow and virtual prop formats.
Edge blending color correction
To perfectly marry the practical set to the virtual one, the very base of the LED wall needs to be color corrected to perfectly match the live action foreground floor. (Note, while it is possible to use LED panels for a floor, this only works for surreal applications, since the ground typically isn’t supposed to generate radiating light.) Edge blending is used to dial in a feathered strip of pixels at the edge of the LED panels to match any local variation of color and contrast in the practical set.
Reality Capture
Reality capture is the process of converting a real, physical object into a digital representation that when filmed by a virtual camera looks indistinguishable from the real thing. In other words, you want to create the physical object’s “digital twin.” The audience should be unable to tell the difference between the physical prop in the foreground and a digital version of the same displayed on the LED wall behind.
There are several methods for capturing real-world objects:
Photogrammetry
In photogrammetry, multiple still photographs are taken from different angles around the object to be captured. Software then identifies the same features found repeatedly in the different photos, and by comparing the parallax differences in the photos comes up with a cloud of points in space representing those features. The software then “skins” (or “meshes”) the cloud of points to create a continuous surface that hopefully matches the surface of the physical object. Continuous textures are extracted from the photographs to create virtual materials matching the detail in the original object’s properties: its coloration and reaction to light.
Modern photogrammetry can be highly detailed and is a popular method for photogrammetry. It struggles to capture reflective surfaces and transparencies.
NERF (Neural Radiance Fields) object capture
People have been wowed by the results of a recent form or reality capture: NERF, or Neural Radiance Fields. NERFs use similar source data to photogrammetry (like a collection of photographs) but use AI to generate novel views of a scene that weren’t photographed during the initial photo shoot. It captures all the surface properties (glossiness, bump detail etc.) of the scene and enables an artist to “rephotograph” the scene from any angle with the lighting reacting naturally to the change in camera angle.
What’s not to love? Well, it turns out that at least in its current form, NERF has limited usefulness for virtual production. NERF “scans” don’t actually capture scene geometry. They’re more like a glorified VR still; a better version of those online real-estate VR tours. (OK, it’s actually way more sophisticated than that and involves some heavy neural network training.) Very impressive if all you want to do is move around a pre-built space, but less useful if you want to interact with scene elements and rearrange the props. There are ways to convert NERF captures into mesh, but at that point photogrammetry will often be a more efficient choice of capture. Where NERF excels over photogrammetry is in scanning featureless objects, something that photogrammetry does poorly.
LIDAR
LIDAR (Light Detection and Ranging) uses a spinning laser (actually an array of lasers) to measure the distance of a scene in all directions. It uses this to build a point cloud like photogrammetry, and then meshes the result.
LIDAR is typically used to capture entire scenes and can capture anything from a small kitchen to several miles of landscape. LIDAR systems can either be mounted on tripods or fitted to drones, the latter for extensive terrain capture.
Modern LIDAR systems supplement the laser scan with conventional photography to capture high resolution surface textures simultaneously with the point cloud data. Systems for virtual production application start at around $20,000 and can range to 6 figures for extreme quality and precision of capture.
It’s worth noting that there are LIDAR scanners built into modern iOS devices, although their precision is much more limited than the expensive commercial systems.
Desktop Scanning
Desktop scanners take a similar approach to LIDAR, albeit on a much smaller scale. In addition to laser scanning of objects (usually with the object on a turntable) there are also mechanical scanners that use an armature to measure the surface of an object at specific points.
3D Wrapping
Enter 3D Wrapping. Popularized by a software called (appropriately) “Wrap” (russian3dscanner.com), this technique can now be found in Houdini, and even Epic’s Mesh to Metahuman plugin. Essentially it uses a fitting algorithm to “shrink-wrap” a clean, neat mesh to a dense mesh generated by a photogrammetry or LIDAR scan. In the case of an animal or human, that new, clean mesh is typically designed to be animated, with edge loops in the geometry arranged to match the muscle directions in the anatomy of the creature.
Other terms
Below is a list of other terms important for understanding the virtual production landscape.
Brainbar
Brainbar is the name given to the collection of computers used to operate the system, and is where technical artists can make changes to the virtual scene as creative choices are made on-set.
Digital Set Extension
Very few productions can afford LED walls of the size used for the Mandalorian, so what happens if the creative vision expands beyond the edges of the wall? Are you always limited to close-ups, mediums, and fulls? What if you want to crane down from post-apocalyptic New York to your actors huddled with gas masks behind a dumpster? This is where digital extensions come in.
Digital extension technology blends the outer edges of an LED wall into a fully-CG version of the virtual world. It composites a full digital render over the top of the rigging, lighting equipment and surrounding stage that would otherwise appear “in-camera.” This requires significant magic in terms of edge blending to make the transition work from pure CG to CG being filmed as it’s displayed on an LED wall. In addition, lens distortion and chromatic aberration in the physical lens have to be precisely matched and simulated in the CG in order to match the pixels filmed by the camera sensor.
Ghostframe / Frame interleaving
Ghostframe is an emerging system for capturing alternate backgrounds simultaneously. By alternating the backgrounds displayed on an LED wall in fractions of a second, it can capture actors illuminated by the virtual scene and greenscreen at the same time. Thanks to the high refresh rates of modern LED panels (specifically ROE panels) Ghostframe can display green pixels to the wall for just a few milliseconds. A high frame rate, fast shutter camera can be perfectly synchronized to record interleaved frames of the LED walls displaying the virtual CG background and then the LED walls displaying the greenscreen.
This sounds like the best of both worlds: a final, in-camera image ready to go the editorial, and a greenscreen version in cases where further visual effects finessing needs to be done. As with all things, it’s not that simple: The requirement of fast shutter timing means that scenes need to be either shot with extremely fast glass (and thus a narrow DOF) or blinding lighting (something that actors are going to struggle with). Currently there are also some artifacts in the frame sync, although this will no doubt get ironed out over time.
For these reasons, the technology turns out not to have great potential for feature and episodic use. It does, however, offer great value in the broadcast graphics space, where multiple screens can be captured simultaneously, including graphics and teleprompt cue projections onto wall surfaces that the TV personality can see—but final audiences will not in their version of the feed.
LED Panel Pixel Pitch
The density of pixels on an LED panel is called the panel’s pixel pitch. Pixel pitch is measured in millimeters between pixels. Pixel pitch for virtual production typically ranges from around 0.9 at the tiny end, up to 2.9 or more. For large outdoor shoots, sizes much larger than 2.9 may be adequate. (Season one of the Mandalorian was shot on a 2.8mm pixel pitch volume.)
While more resolution (lower pitch) is obviously desirable, cost of panels rises dramatically as the pixel pitch lowers.
Lens Encoder
In order to align the exposure, focus, and focal length of the virtual camera “filming” the virtual world to the real-world camera, these properties need to be tracked. A simple rotary encoder attaches to the aperture ring, the focus ring, and (optionally for a zoom lens) the zoom ring of the lens to capture this data. Think of a rotary encoder like one of those old-fashioned clicker wheels that surveyors use to measure distances along the ground. Every so many feet or meters the wheel clicks to count another unit traveled. A rotary encoder does the same thing by sending a digital pulse when the encoder dial is rotated slightly. These things are highly precise, able to measure tiny fractions of a degree.
For focus, measurements of the encoder’s value are calibrated with the distance of a focus object from the camera. By sampling several pairs of encoder value and focal distance readings a curve can be created that estimates the focal distance for any given value coming from the encoder. Similar calibrations can be done for aperture stops and for camera zoom.
As simple as this all sounds, these encoders have not been standardized and currently companies charge exorbitant prices for these units. This is sure to come down drastically in the next year or two, since the actual electronic components to make one of these systems retail at under $50 USD.
Bear in mind that lenses “breathe” as they change in focus and zoom, so lens distortion needs to be measured and computed at each of these distances to create and accurate match in the case of greenscreen composites and digital extension work.
Lumen vs baked lighting
Lumen is Unreal Engine 5’s (UE5 for short) new system for generating global illumination in real-time. This means that UE5 simulates the natural way photons bounce around a scene, combining the colors and light scattering properties of the surfaces they collide with in order to produce more natural-looking lighting compared with traditional game engine “cheats.”
Before Lumen came onto the scene, all virtual production sets used “baked lighting.” In baked lighting, intensive lighting calculations are made offline (before shooting on the set begins). These calculations could take a single computer weeks and are typically distributed between dozens of computers in a render farm to complete the process in minutes instead. Once all the lighting is calculated, the light and shadow detail is “baked” onto object surfaces. Imagine buying wallpaper that includes image details of picture frame shadows in the wallpaper artwork. That’s essentially what happens with baked lighting: The brightness and shadow detail is painted onto the surface so that as the camera films the virtual scene Unreal Engine doesn’t need to recalculate the lighting, it just films surfaces with the shadows “painted on.”
Of course the downside of this is whenever a light source or an object casting shadows is moved, the entire scene needs to be recomputed and new shadows and highlights redrawn over the scene surfaces. This means a director who likes to change their mind on set frequently can become very expensive very quickly: Each change may require hundreds of computers to recalculate the lighting at great expense if the production schedule is to stay on track.
Because Lumen calculates the lighting information in real-time, the need to bake offline is eliminated. It’s important to note that UE5 still “cheats” to achieve global illumination in real-time. It uses tricks like Signed Distance Fields (SDFs) to approximate surface geometry and simplify the calculation load required to simulate lighting effects. This inevitably produces artifacts compared to the more complex rendering produced by offline renderers (like Arnold and V-Ray). As a result many virtual production stages still use baked lighting for the improved accuracy of surface details (and for the faster performance of the real-time render nodes). Nonetheless, with a bit of skill and experience, artists have been able to produce surprising realism with the Lumen system. Combined with the ability to modify scenes on the fly without having to kick off massive render farms to make changes, it’s clear that Lumen and systems like it are the future.
Nanite
Lumen’s counterpart in the real-time revolution is Nanite. Where Lumen improves the detail of light rays, Nanite improves the detail of the scene surfaces illuminated. In traditional game engines, every triangle that makes up the polygonal mesh of an object’s surface has to be loaded into a computer’s GPU ready for rendering. Since even powerful GPUs have a limit to the amount of fast VRAM memory available to them, designers have had to limit the number of triangles used to represent objects, resulting often in simplistic, angular surfaces that should look smooth.
Nanite solves the problem by intelligently streaming only the triangles needed for the current camera angle off a fast NVMe solid state hard drive. This effectively allows an unlimited number of triangles to describe an object and therefore removes the polygon count limitation when designing virtual production sets. Now before everyone goes crazy dragging unedited scan data into Unreal Engine, know that there are still practical limitations to just how much data can be streamed. Nonetheless, Nanite dramatically improves the workflow and makes the kind of detail needed for feature film work possible.
Virtual Location Scouting
Virtual Lighting (DMX Simulation)
In addition to controlling video production lights in real-time, another application of virtual production is the planning of lighting for live events. Unreal Engine can previs a full stadium concert lighting setup, complete with DMX lighting messages and multi-universe signaling.