Category: Mo-Sys Academy

What you can achieve with Virtual Production – Mo-Sys Engineering’s commercial director Mike Grieve on how virtual production can elevate creativity and save resources

In the virtual world, possibilities are endless. Unlike real sets where you’re limited to the physical attributes of set design, virtual sets are built in Unreal Engine where you can be anywhere and have anything. Creatively, it breaks you free from budget, time and location limitations.

Stop Fixing in Post, Solve It in Pre

One of the key attributes of virtual production is the ability to pre-visualise everything before you even get to set. You can “walk” through your virtual scene and make decisions on where the best camera angles are, change lens types and adjust lighting. And with everyone from the director and producer, to the cinematographer and VFX supervisor having the ability to be together, looking at the same 3D scene from anywhere in the world, decisions can be made far more quickly and easily. So when you turn up on the day, all you need to do is light and shoot.

You don’t get that level of foresight on a physical shoot. Virtual production swaps basic preparation and fixing things in post, for high level prep by solving things in pre-production.

Not only that, but now that talent can actually see the virtual set around them – using an LED volume – rather than imagining where they need to look and interact using a green screen, you can shoot far more accurately. This helps avoid errors on things like eyelines between talent and virtual elements.

When you look at the whole production process, from pre-production to the actual deliverable, virtual production shrinks the overall production time and costs by reducing post-production needs. The bottom line is, it’s better to solve problems in pre than try to fix them in post.

Shoot In-Camera Effects in Real-Time

The quality of the 3D scene created for a virtual production shoot, is always very, very good. But when the scene is loaded into the computer stack running Unreal Engine, and camera tracking is attached, the scene, more often than not, doesn’t play back in real-time. This is because the scene can’t be processed fast enough.

When this happens, the scene needs to be ‘optimised’, which is a bit like video compression that shrinks down the file size. When the processing load goes down, the frame rate comes up allowing the scene to play back in real-time and the real-time VFX shoot to happen.

The problem then is that the quality level of Unreal scenes is fixed. Because if you try to add any more quality, the frame rate drops below real-time and you can’t shoot in-camera effects. This is a well known problem.

What normally happens is that a director or producer will then need to decide which shots will need to go to post-production for compositing to increase the quality of the background. That takes time and money. But not only that, it actually goes against the whole principle of virtual production which aims to cut down compositing time as much as possible.

At Mo-Sys, we’ve patented a solution to this called Neartime. It’s a service that runs in parallel with a real-time VFX LED shoot, that auto re-renders the background virtual scene at higher quality, enabling it to be composited back together with the keyed talent, so you can deliver a much higher quality product in the same delivery window.

So as soon as you start the camera to do the first shot, all of the tracking and lens data from the camera is thrown up into the Cloud, where that same Unreal scene that you’re shooting exists on 50 to 100 servers. Then, all the quality dials are wound up and each take is re-rendered out sequentially as the real-time shoot goes on. It allows you to deliver higher resolution background graphics, faster and automatically, to save money and time.

Embrace Change and Dive In

As virtual production is still fairly new for most creatives and directors, there is an element of getting used to new ways of working. Things like lighting are handled differently on a virtual set, for example. When you’ve got real talent lit by hard and soft lighting, and the LED wall with different lighting characteristics displaying the background scene, it all needs to match in order to look like part of the same set viewed from the camera perspective. Fortunately on-set colour grading is about to get a boost, which will be ‘music’ to cinematographers who have already shot in a LED volume.

At the moment, the biggest challenge lies in the quality of the Unreal scene. When you go into a virtual set, there are two types of video that you display on the LED wall. One of them is video plate playback which is used for things like car scenes where the vehicle is moving quickly down a street. The car is static in the virtual studio but the video is moving. Those scenes are very high quality because they are shot with multiple high quality cameras on a rig designed to capture a rolling 360 degree view.

But then you have the Unreal scene using virtual graphics. This is where you need camera tracking on the real camera to match it to the virtual scene displayed on the wall. The quality of these virtual graphics is very good but it’s not quite as good post-production compositing just yet. This is where our NearTime technology can help.

And finally, you’ve got the challenge of continuity when changing elements or editing Unreal scenes live on set. Imagine you’re on a virtual set and suddenly you decide that you want to move one of the objects on the LED volume to the other side of the screen. When you change something, you need to log what you’ve changed as it always has a down-stream impact on the shoot, and it can cause issues if you need to then remember what other scenes need updating as a result. This is something Mo-Sys is working on solving very soon, with technology that allows on-set real-time editing of Unreal scenes that automatically captures and logs the revisions. Watch this space!

When the unmistakably quirky Gollum first appeared on our screens back in 2001, audiences were taken by his unique character conveyed purely through CGI. Despite the fact that MoCap had already been around for a while, this moment signified the shifting of Motion Capture acting into the mainstream. 

Andy Serkis’ take on Tolkien’s now infamous Lord of the Rings character provided the movie industry with a true taste of the potential of Motion Capture Acting. The result was a complete revolution. Not just for the film industry, but for creating realistic and cinematic gaming experiences as well as for use in sports therapy, farming and healthcare.

From Middle Earth to Pandora: How motion capture works

In the last two decades we’ve watched numerous ‘behind the scenes’ images of Serkis jumping around in a MoCap suit equipped with retroreflective markers. During the filming of Peter Jackson’s Lord of the Rings (2001), these retro reflective optical markers allowed motion capture technology to accurately record his facial and body movements through a series of motion tracking cameras. This data was then transferred to a graphics engine to create a ‘skeleton’ which acted as a base for the animated body of Gollum. 

This early MoCap technique, which is still used today by some, is known as an ‘outside-in’ system; this means that the cameras look from a perspective ‘outside’ of the MoCap environment, and into the movement of the actor. The second, more recent technique (which we’ll explain further below), involves the use of Inertial Measurement Units (IMUs) to capture an actors movement regardless of any space or location (Xsens’ MVN system is an example of this type of MoCap setup). 

Performance capture  

Seeing the potential of the technology to enhance productions, a number of companies have since  invested in technology that more accurately records facial, hand, and finger motion. Known informally as ‘performance capture’ tech, these more targeted systems give motion-caption actors a greater degree of creative freedom whilst improving the credibility of their CGI digital characters. 

And their increased use in film production has not gone unnoticed. James Cameron’s Avatar (2001), for instance, was highlighted by critics for its innovative use of performance capture tech when creating the ethereal Na’vi. Matt Reeves’s War of the Planet of the Apes (2017), furthermore, was praised for its use of facial motion capture; Andy Serkis, who played the leading role, wore 132 mini retro-reflective markers on his face in order to record the exact movement of his facial muscles. 

Free-environment motion capture 

In 2011, Mo-Cap was upgraded again so it could be taken out of the studio and used on location. This began with Rise of the Planet of the Apes (2011), and allowed actors to fully immerse themselves in mixed reality environments whilst giving producers unparalleled creative freedom. Shooting outside also meant adapting the technology to varying climates. Mo-cap motion capture suits were made much more robust with the reflective markers upgraded to infrared pulse so directors could film in bright light (an example of active optical motion capture). 

The production teams of Dawn of the Planet of the Apes (2014) and War of the Planet of the Apes (2017) took the tech even further, filming in humid conditions and at night. This came alongside advancements in the rendering textures of fur, skin and eyes, allowing audiences to enjoy the most cinematically-gripping visuals of photo-realistic simulations. 

The magic of Motion Capture Acting

The beauty behind motion capture as a form of acting lies in the emphasis on physicality and embodied movement and expression. In an interview with WIRED Serkis explains, “It’s not just about mimicking behaviour. This is about creating a character.” This includes developing the psychological and emotional journeys to pour into the character. “It’s not just about the physical build of the character, but also the internalisation,”explains the actor. “That’s why the misconception that performance capture acting is a genre or type of acting is completely wrong, It’s no different to any process you go through to create a role… the actor’s performance is the actor’s performance.”

The combination of professional acting skills with advancements in Mo-cap technology has led to the development of many memorable CGI characters in recent years. From Serkis’s Portrayal of Caesar in Planet of the Apes to Benedict Cumberbatch and his unique take on the dragon Smaug in The Hobbit, motion capture is giving actors more powerful tools to portray characters and ultimately, enhance storytelling. 

If you’re interested in learning more about the future of the film industry, check back regularly for more articles from Mo-Sys Academy. Drawing from years of experience in virtual production for film and tv, and as one of the UK’s leading camera tracking suppliers, we’re aiming to educate the next generation of producers.  

A jib camera is simply a camera mounted on a jib, which is a boom or crane device. On the other end of the jib, there’s a counterweight and either manual or automatic controls to direct the position of the camera. The main benefit of using a jib is that it allows camera operators to get shots that would otherwise be difficult to obtain due to obstacles or awkward positions. 

Different types of jib 

Jibs come in many sizes; from small jibs for handheld cameras to enormous booms that can pan over the heads of huge festival crowds. Regardless of the size, however, the purpose remains the same. A camera jib is there to help provide producers with stable crane shots. 

What is a crane shot? 

A crane shot is shot taken from a jib camera. Whilst most jibs can move in all directions, they’re valued primarily for their ability to move on a vertical plane. This gives producers the opportunity to emphasise the scale of a set, and is often used either to introduce or close a setting in a film. 

Crane shot examples 

La La Land (2017) 

The opening scene of Damien Chazelle’s Oscar-nominated La La Land was shot with the use of a camera jib. The scene presented various challenges to camera technicians as the shot weaves around stationary cars and dancers. An added complication was that the freeway it was filmed on was actually slanted, creating panning perspective problems. Regardless, the end result was a success –  the scene set the tone for the rest of the film whilst introducing Los Angeles, the central location of the narrative. 

Once Upon a Time in Hollywood (2019)

Quentin Tarrantino is well-known for his use of jibs for panoramic and tracking shots. Most recently, he used them in Once Upon a Time in Hollywood (2019) to place characters in context and add atmosphere. At the end of the ‘Rick’s house’ scene, a large jib camera slowly pans out from across the top of a Hollywood home to reveal the neighbourhood’s quiet night time roads. 

Camera heads for jibs 

To achieve shots like these, operators need to be able to move and adjust the camera at the end of the jib. This can be done either manually through a series of pull-wheels, or automatically with a controller. Either way allows operators to pan, tilt, and zoom the camera.

Camera jibs for virtual production 

Jibs used for virtual production either need to have all axes encoded, or have a tracking system attached to them. This is required in order to capture camera movement data in order that the virtual elements of a shot can be made to move in exactly the same way as the real camera shot. When it comes to virtual production, which jib you decide to use is extremely important. This is because any unintended movement (i.e. any unencoded or untracked movement) caused by the jib can cause virtual images to ‘float’ and break the illusion. To counter this, VP jibs need to be heavier, sturdier, and more rigid. Mo-Sys’s e-Crane and Robojib were designed specifically with these needs in mind – catering to a growing trend in Virtual Production (VP), Extended reality (XR), and augmented reality (AR). 

Mo-Sys Academy is committed to sharing the latest developments in film and broadcasting technology with those looking to enter the field. If you’re interested in learning more, check out our previous articles explaining what a film crew does, and the difference between AR and VR broadcasting.