Month: October 2020

Extended reality (XR) is a term that is commonly used to describe all environments and interactions that combine real and virtual elements. Whilst XR usually encompasses AR (augmented reality), MR (mixed reality) and VR (virtual reality), it has a more specific meaning when used in relation to film, broadcast, and live entertainment production. In this article, we explain how, and why it’s on course to become a studio staple. 

Extended Reality meaning 

When used as an umbrella term, XR denotes all AR, MR and VR technologies –it’s the overarching label given to systems that integrate virtual and real worlds. In this sense, XR can be applied equally to certain motion capture techniques, augmented reality applications, or VR gaming. 

In the production world, however, it means something much more specific. XR production refers to a workflow that comprises LED screens, camera tracking systems, and powerful graphics engines. Often the LED wall operate with set extensions, which are tracked AR masks that allow the virtual scene to extend seamlessly beyond the LED wall.

How does XR production work? 

In XR production, a pre-configured 3D virtual environment generated by the graphics engine is displayed on one (or across multiple) high-quality LED screens that form the background to live-action, real world events.  When combined with a precision motion tracking system, cameras are able to move in and around the virtual environment, with the real and virtual elements seamlessly merged and locked together creating the combined illusion.

The benefits of XR production 

Immersive production and real time edits 

Immersive technology enables actors, hosts, and producers to see the virtual environments whilst shooting. This means that they can adapt their performances or make edits live on set, which reduces time (and budget) spent in post-production.  

Lighting 

Lighting is provided by the LED screens on an XR set. Common illumination helps real-world people and objects blend seamlessly into virtual environments, and further reduces time on set adjusting lighting. 

No Colour Spill or Chroma Key Compositing 

On certain green screen setups, colour spill and the need for chroma key compositing can increase the time spent in post production. Neither is required for XR screens which, again, reduces any time needed in post-production. 

Rapid calibration 

The calibration of camera tracking systems on xR sets takes minutes rather than hours (as can happen with green screen sets). This allows for scenes to be shot across multiple sessions with minimal disruption and preparation. However, where set extensions are used or AR objects are added, more precise calibration would still be required.

Examples of XR in production 

The Mandalorian 

Having experimented with XR sets in the making of The Lion King (2019), producer Jon Favreau used them to complete half of all scenes for his production of Disney’s The Mandalorian (2019). With a twenty-foot tall LED screen wall that spanned 270°, Favreau’s team filmed scenes across a range of environments, from frozen planets to barren deserts and the insides of intergalactic spaceships. Apart from giving the cast and crew a photo-realistic backdrop to work against, the XR set saved a significant amount of time in production. Rather than continually changing location and studio setup, the design team could rapidly switch out props and partial sets from inside the 75-foot diameter of the volume. 

Dave at the BRIT Awards 

At the 2020 BRIT Awards, an XR setup was used to enhance Dave’s performance of his single ‘Black’. In a collaboration between Mo-Sys, Notch, and Disguise, a 3D animation was projected onto Dave’s piano, giving live audiences around the world an engaging visual experience. With effective camera tracking provided by Mo-Sys Startracker, camera operators could move freely around the stage with zero movement or disturbance in the piano’s moving images. 

HP OMEN Challenge 2019 

With the rise of eSports, developers are exploring new ways to enhance gaming and bring immersive experiences to ever larger audiences. In 2019, HP did this by broadcasting their OMEN eSports tournament live from an XR stage. There were two main benefits of using extended reality: firstly, audiences around the world could immerse themselves in the virtual environments of the game; secondly, gamers in the studio could review their gameplay from ‘within’ the game. The end result was an interactive, immersive experience that blurred the lines between the real and virtual world. 

Mo-Sys Academy is committed to sharing the latest developments in film and broadcasting technology with those looking to enter the field. If you’re interested in learning more, check out our previous articles explaining what a film crew does, and the difference between AR and VR broadcasting. 

As film-making has got more technologically-advanced, film crews have changed in composition. Whilst the core remains the same (directors, producers, technicians, and camera operators) – there are many more roles which have opened up as a result of virtual production (VP). If you’re considering a career within the film industry and are wondering, ‘what does a film crew do?’, read on to learn about the main positions, and how they’re changing as VP progresses.  

Film crew roles explained 

Producer 

Producers are the driving force behind productions. They oversee projects from start to finish, making decisions about key concepts, creative processes, and finances. Whilst they’re mostly focused on organisation and operational functions, they also hold sway over script selection, directing, and editing. 

Director 

Directors, on the other hand, are much more involved in the creative side of filmmaking. They set the artistic direction of the production, and guide the technicians on how to achieve it. Alongside deciding on shots and angles, directors oversee casting, set design, and musical score. 

Scriptwriter 

Scriptwriters are often the starting point of film productions. They provide the initial idea of a story, and craft it into a compelling narrative. They shape characters, give them a voice, and make them believable. Although scriptwriters are often in the shadow of producers and directors, they’re a fundamental part of the filmmaking process. 

Production designer 

Working closely with producers and directors, production designers are responsible for the visual concept of a film. This covers location spotting, set design, costume design, lighting, and visual effects. As VP becomes more popular, moreover, production designers are collaborating more with graphic designers and VFX artists in the pre-production phase – ultimately, shortening time and money spent in post-production. 

Production manager 

Reporting to the producer, production managers deal with the day-to-day running of film projects. They’re responsible for hiring film crew members, managing running budgets, and organising shooting schedules. You might consider them the on-the-ground project managers of the filmmaking process. 

Cinematographer/ Director of Photography (DP)

The DP transforms the director’s vision into reality through their technical knowledge. Traditionally, they advise on which cameras, lenses, filters, and stock to use to achieve the desired shots. Similarly to production designers, their role is increasingly influenced by VP. They need to be aware of how virtual production systems integrate with standard studio equipment such as lighting and rigging, and how to set up and calibrate VP gear. 

Focus puller 

In traditional filmmaking, a focus puller works alongside a camera operator to manually bring actors and objects into focus at the right time. Moving with the camera, they adjust the lens to according to the distance between themselves and the object they want to focus on. They may put markers down before filming (like sellotape on the floor of the set) to help them, or they may just rely on their own spatial awareness as the camera is rolling. Whilst their role is likely to change with the increasing use of VP methods, they’ll remain incredibly important on set. The production team of Jon Favreu’s The Lion King (2019), showed how focus pullers can be integrated into a new way of multi-track filming methods. 

Director of Virtual Production (DVP)

As VP progresses, however, it’s becoming more common to hire a separate Director of Virtual Production (DVP), who concentrates solely on VP during a project. Although the role is developing along with the tech, DVP’s are generally responsible for managing virtual props, dealing with VFX vendors,  overseeing the pre-production phase, and transferring assets to graphic engines.

Digital Image Technician (DIT)

In a broad sense, the DIT ensures that the highest technical standards are maintained when filming. As experts of the latest camera technology and associated software, they advise the DP and DVP on any issues relating to digital (rather than film) recording, including contrasts, exposure, framing, and focus. They’re also responsible that all film data is stored and managed correctly, making regular backups and transferring into file types that are accessible to other departments in post production. 

Films crews and changing technology 

As VP technology advances, the traditional roles within film crews are adapting. Whilst directors, producers, and DPs are becoming more aware of the advantages of VP – including greater creative freedom, fewer resources required, and less time needed – new roles are emerging to operate highly specialised VP tech. Ultimately, the days of large film crews moving between locations like travelling circuses are gone – replaced with remote, more agile and technically-enabled teams. 

Want to learn more about virtual production and the film industry? Take a look at articles from Mo-Sys Academy. We’re here to help educate and inform those looking to make their first steps into the world of filmmaking.

Motion capture (Mo-cap) refers to a group of technologies that records the movements of people and objects, and transfers the corresponding data to another application. It’s been used for many purposes, from sports therapy, farming, and healthcare, to film and gaming. By mapping real-world movement on computer generated frames, motion capture allows for photorealistic dynamics in a virtual environment. Here’s how it developed and how it works. 

The birth of mo-cap 

The first major step in the development of mocap was brought by American animator, Lee Harrison III, in the 1960s. Using a series of analogue circuits, cathode ray tubes, and adjustable resistors, Harrison devised a system that could record and animate a person’s movement in real-time. 

Harrison’s Animac and Scanimate technology was developed in the late 1960s and allowed real-time animations to be created and processed by a computer. With Animac, actors would wear what was described as an ‘electrical harness’ or ‘data suit’ wired up to a computer. Using potentiometers that picked up movements attached to the suit, an actor’s movements could be translated into crude animations on a monitor.

Though the result was fairly basic, it was soon being utilised in various TV shows and advertisements across the States. The abstract images that could be produced with the rudimentary mocap technology of Animac and Scanimate, however, just weren’t good enough to attract mainstream attention.

The development of mo-cap 

The following decades saw improvements on Harrison’s designs, with bodysuits more accurately recording movement. They were also helped by the development of large tracking cameras; as useful as they were, however, each was about the size of a fridge. 

While mocap had been used sparingly in the 1980s and 1990s with films like American Pop (1981) and Cool World (1992), the first film to be used entirely using the technology was Sinbad: Beyond the Veil of Mists (2000). The film was a flop, but its use of mocap was picked up and expanded on by Peter Jackson in his making of The Lord of the Rings Trilogy in the early 2000s. 

For the first time ever, actors wearing their bodysuits (complete with retroreflective ping-pong balls) could perform alongside their non-animated colleagues in the same scene. Among CG-created characters, The Lord of the Rings’ Gollum is recognised as one of the most impressive Hollywood has ever produced. The combination of the character’s voice and intricate facial expressions performed by Andy Serkis resulted in an unforgettable motion-capture performance. The character and technology were created on the fly by Weta Digital’s Bay Raitt.

Facial capture 

With an increasing awareness of how motion capture techniques can enhance productions, more attention has been given specifically to facial capture. A number of companies have developed highly accurate systems, which, when paired with powerful graphics engines, result in like-like, photo-realistic facial images. Cubic Motion (now partnered with Unreal Engine) is one of these, and has worked on a number of high-profile games and virtual experiences, including Spiderman, League of Legends, Apex Legends. 

Motion capture techniques 

Nowadays, there are four main motion capture techniques: 

• Optical (passive) – With this technique, retroreflective markers are attached to bodies or objects, and reflect light generated from near the camera lens. Once reflected, the light is used to calculate the position of the markers within a three-dimensional space, and recorded. 
• Optical (active) – This technique is exactly the same, but the markers emit light rather than reflect them. The markers therefore require a power source. 
• Marker-less – This technique doesn’t require markers of any sort. It relies on depth-sensitive cameras and specialised software in order to track and record moving people and objects. Whilst more convenient in some ways, it’s generally considered less accurate than its optical or mechanical-tracking alternatives. 
• Inertial – This technique doesn’t necessarily need cameras to operate. It records movement through IMUs (inertial measurement units), which contain sensors to measure rotational rates. The most common sensors used in IMUs are gyroscopes, magnetometers, and accelerometers. 

What is motion tracking used for? 

Motion tracking and capture has a broad range of uses across various industries, including: 

• Film and Gaming – Motion capture is used to record the movement of actors and transfer them onto virtual or computer-augmented characters.
• Sports Therapy and Healthcare – Health professionals can use motion capture to analyse the movement of patients and diagnose problems i.e gait analysis. 
• Military – When combined with virtual reality, motion capture technologies have been used to enhance military training experiences. 

---

If you’re interested in learning more about the future of the film industry, check back regularly for more articles from Mo-Sys Academy. Drawing from years of experience in virtual production for film and tv, and as one of the UK’s leading camera tracking suppliers, we’re aiming to educate the next generation of producers.