On a film set, a director can command a technical artist to replace desert dunes with a glowing alien landscape, fill the sky with stars, or set the sun—all in real time, while the camera rolls. This real-time capability is the essence of virtual production and real-time technology in filmmaking, merging physical and digital worlds to give creators unprecedented control. Unlike traditional methods that add visual effects months after filming, this technique captures fantastical environments directly in-camera.
What is Virtual Production in Filmmaking?
Virtual production is a filmmaking technique that combines live-action filming with real-time computer-generated imagery (CGI), directly joining the digital world with the physical as filming happens. This integration allows directors, cinematographers, and actors to see a close approximation of the final visual effect shot live on set—a significant departure from the traditional linear pipeline where visual effects are added in post-production.
The core of this process is powered by real-time game engines, such as Unreal Engine, which are capable of rendering photorealistic digital environments instantaneously. While the concept has been used in the broadcast industry for years to create live graphics for sports and news, its adoption in narrative filmmaking was popularized by productions like Disney's The Mandalorian. According to a report from ftrack, the worldwide market for virtual production reached $1.82 billion USD in 2022, signaling its growing importance in the industry.
How Virtual Production Works: A Step-by-Step Guide
The virtual production workflow fundamentally changes filmmaking's order of operations, emphasizing extensive preparation before cameras roll. A typical in-camera visual effects (ICVFX) shoot requires specific steps to come together.
Step 1: Digital Asset CreationLong before the cast arrives on set, the process begins in the digital realm. 3D artists and environment designers build the virtual world inside a real-time game engine. This involves creating everything the camera will see: landscapes, buildings, skies, and digital props. These assets must be highly detailed and optimized to be rendered instantly without sacrificing visual quality.
Step 2: Previsualization and Virtual ScoutingOnce the digital environment is built, key creative stakeholders can enter it virtually. Using VR headsets or specialized software, the director and cinematographer can "scout" the digital location. They can walk through the scene, plan camera angles, test lighting schemes, and block actor movements in a process called previsualization, or "pre-viz." This digital rehearsal phase allows for creative decisions to be made early, saving valuable time and money once physical production begins.
Step 3: The LED Volume SetupThe physical set, often called "The Volume," is typically a stage surrounded by large, high-resolution LED video panels. These panels form walls and sometimes a ceiling, creating an immersive space. The digital environment created in Step 1 is projected onto these screens. In the center of the volume, physical set pieces and props are placed to blend seamlessly with the virtual background, grounding the actors in a tangible space.
Step 4: Camera and Motion TrackingThe film camera is outfitted with motion tracking sensors. These sensors constantly communicate the camera's precise position, orientation, lens focal length, and focus distance to the game engine. This data is the critical link between the physical camera and the virtual world. Multiple tracking systems exist, from optical solutions that use infrared cameras to inertial measurement units (IMUs).
Step 5: Real-Time Rendering and Parallax EffectThis is where the technology's true power is revealed. As the physical camera moves, the tracking data is sent to the game engine, which then adjusts the perspective of the digital background in real time. This creates a perfect motion parallax—the visual effect where objects in the background appear to move more slowly than objects in the foreground. This makes the 2D image on the LED screens appear as a convincing 3D space from the camera's point of view, eliminating the flat look of a traditional backdrop.
Step 6: In-Camera VFX (ICVFX) FilmingWith everything synchronized, filming begins. Actors perform on the physical set, surrounded by the dynamic virtual environment. A key benefit, as explained by resources from Unreal Engine, is that the light from the LED panels illuminates the actors and physical props. This creates realistic, interactive lighting and reflections that are notoriously difficult to replicate with a traditional green screen. The final composite image—actor, physical set, and virtual background—is captured together in-camera, resulting in a nearly finished shot on the day of filming.
