Line producing virtual production: What you need to know

The experts from MESH – James Blevins (post supervisor on The Mandalorian) Ben Baker (Epic Games Unreal Fellow and previously vice president of Studio Services at HBO)- share their VP insights and experiences in the burgeoning world of virtual production

Byline: Adrian Pennington

Virtual production is upending the century-old filmmaking process with a suite of technologies and techniques converging around digital tools and nonlinear workflows. While elements of virtual production such as previz and the creation of VFX assets and LED backdrops have been around for a while, the speed at which they are being combined into a unified workflow for the whole of production can be overwhelming to producers and crafts technicians making the transition. In particular, the difficulties of finding talent with virtual production experience and an understanding of how to work with vendors to get the best out of everyone in this new collaborative environment calls for cool heads and specialised line production skills.

The MESH co-founders walked us through some key steps.

How does pre-production help with success?

Nothing succeeds like prep. Prep reveals efficiencies and forces people to collaborate much earlier on the project. It affects every department. You can do pretty much anything you want on a virtual production stage provided you plan for it.

For example, the camera department needs to better understand what they can and can’t do. That requires a lot of testing which is part of a bigger approach that enables a collaborative conversation between department heads.

“You can do pretty much anything you want on a virtual production stage provided you plan for it”

Traditionally, arriving on set was the point of collision where the camera and production design finally met. That’s why people talked about ‘happy accidents’ in the creative process. Now those ‘happy accidents’ need to be baked-in in planning. There is so much going on in a virtual production that winging it is not an option.

Scheduling is a main element. The principles are the same but with virtual production you drive a conversation by creating a schedule that creates a cadence of delivery and involvement and note giving for all assets. This requires that people come together in prep in a way that will be unfamiliar to many and may be outside the comfort zone. It also requires that you pay them for their time.”

What is the impact of pre-production planning on previously siloed crafts?

Cinematographers now have the power to make decisions in pre-viz that directly impacts other crafts, like production design. They can make decisions on designing light sources (windows, perhaps) into the virtual assets. The director must start making hard wiring decisions too. Shooting on green screen is only kicking the can down the road. The discipline required to make decisions and to stick to them can be hard for some directors and crews to get to grips with.

“Pre-viz forces you to come to the table with a collaborative approach”

This whole process is beneficial to the end product because it is far more proactive than reactive. Pre-viz forces you to come to the table with a collaborative approach. There is no time during principal photography to move things around because of the asset light baking. Moving an asset on set can mean hours of light baking. You want to avoid that if you can.

You’re essentially asking your whole team to step up to what Spielberg and Zemeckis have been doing for years: planning the hell out of it, costing and re-costing, iterating and re-iterating so on the day you just shoot. You can’t get onto a volume without having made some hard decisions a month prior. Filmmakers have to be into that.

Some things you can move up to the point of being on set and some you can’t. The trick – or the skill – is to know the difference between a reasonable and an unreasonable request.

What are some of the things to check when selecting a volume stage?

“All walls are not the same. We view them as instruments that need to be calibrated. If a vendor says their volume stage has been calibrated, we always want to measure it. For example, that means making sure that the colours that are expected to be sent to the wall are being received by the camera properly. You can’t rely on your eyes. The light emitting from the LED reacts differently with organics to that of a chip.

The size of the pixel pitch is not a main issue. The relationship of the grid to the camera chip is. This is a mathematical problem related to how light waves from the wall are transmitted to the camera and, if not checked and properly calibrated, can result in artefacts like moiré. This calibration is different for every single camera set up and every wall.

What role does remote collaboration play in virtual production?

Triple ‘A’ games have been developed collaboratively for some time using high bandwidth connectivity to contribute Unreal assets remotely. Downloading assets, from Quixel for example, is entirely cloud-based today and only enabled by connectivity. We used the Sohonet’s full collaboration toolkit on The Mandalorian – with multiple ClearView Pivot, Pivot Lite, and Flex boxes being utilised across the post, and I was able to colour time it entirely remotely.  Cutting on decentralised edit stations or sharing review sessions is so routine now that it’s almost second nature.

“I was able to colour time The Mandalorian entirely remotely”

So, connectivity is an integral part of the virtual production workflow and one that will only become more important as a means of solving the shortage of talent in media and entertainment. A connected multi-cloud solution will be key to finding the talent to work with around the planet and connecting them in real-time to work with OCF. We are not there yet, but the building blocks are in place.

“We used the Sohonet’s full collaboration toolkit on The Mandalorian”

One of the benefits of virtual production are the economies it delivers over conventional shoots, but can that be quantified?

The equation is project dependent and not quite as simple as removing travel from the bottom line. Certainly, transport to location, accommodation, per diems etc. are reduced. There’s often a small reduction in the cost of production design. A lot of VFX are eliminated and there’s little to no green screen work, but a lot of the budget is reallocated from post into prep. Overall, if you observed all these efficiencies, you could net around 10% savings.

“If you observed all these efficiencies, you could net around 10% savings.”

The biggest reduction comes from needing fewer shoot days and ideally, no reshoots. For example, with virtual production you can control the weather. By identifying efficiencies at script breakdown stage, you can maximise everyone’s time on the virtual production stage. If you’re looking at $150k per shoot day and you knock two of those off a 20-day schedule you have saved some pretty serious cash.

Can you outline what you mean by savings that are project dependent?

There are particular types of shows that benefit from virtual production. These include all shows that would previously have shot elements on green screen. The superior quality of being able to shoot in camera VFX should relegate green screen to history.

“If you’re looking at $150k per shoot day and you knock two of those off a 20-day schedule you have saved some pretty serious cash.”

Many shows with a low to mid-range budget can save money by not paying licences for and setting up at locations – a hospital, a fire station, a mansion – when all can be efficiently shot in the volume. Similarly, those shows can maximize time with a major actor – an A list star perhaps – by shooting several pages over one day at different locations in a volume.

For tentpole series and films, the cost savings of a few hundred thousand dollars here and there is less important than having their $200m show up as production value on screen. The imperative is ‘let’s see it on-screen.’ That’s the drumbeat at this point, not cost efficiency.

How will technologies and techniques change in the next 12 months?

In the next year, heads of production will try to enable a certain level of virtual production through their art departments because that is where a culture of spending money up front exists. It’s a very difficult thing for a producer to get their head around managing and spending the VFX money upfront and giving a VFX supervisor so much latitude. Going forward, the art department will get more control and more access.

“VFX supervisors are in charge of the conversation at the moment”

VFX supervisors are in charge of the conversation at the moment because they are really the only ones who can recover a show if there are any issues. Until the art department has access to the same level of vendor and talent relationships, VFX supervisors will remain the trusted partner in delivering the final image.

That said, there is also a sea change in the way the art directors request assets. A certain quality of asset will have to be delivered much faster than before and certainly a lot faster than what VFX are used to. This is currently a source of some resistance, but artificial intelligence can ride to the rescue.

The rapid advance of generative image making tools (like Stable Diffusion, DALLE-2) will increasingly play a role here as will NeRFs (neural radiance field), a method of generating 3D objects or scenes from 2D images.

We have worked on projects that would have had a completely different cost, had AI tools been available. Today, you can build an asset for a proof of concept in under two weeks using tools such as geophysical survey data, Google Earth, high range satellite photos, LIDAR scans and photogrammetry. The issue is whether you light the final set convincingly, so you sell the story you are telling.