Virtual Reality (VR) is the term used to describe a three-dimensional, computer-generated environment, which can be interacted with by a person. That person becomes part of this virtual world or is immersed within this environment and whilst there, is able to manipulate objects in a way that he feels he is a part of it. Augmented Reality (AR), on similar lines, is a technology that superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view. VR and A have been used to great success in movies and documentaries, owing to its ability to simulate an environment for a user to feel he’s right in the middle of the scene going on in a movie. This helps the user empathise with the subject matter—especially a sensitive one, which otherwise, might be difficult to convey by traditional flat screens.
Behram Patel, head of training, Emerging Technologies, Famous Studios, said, “Film productions in Hollywood are making tremendous use of VR for previsualisation. Entire digital sets are created in 3D and then seen in VR, so that the filmmakers can get a clear sense of space and location. Digital set construction helps all stakeholders make informed decisions when the time comes to actually construct a set.” VR is also used in digital location scouting, which falls under virtual production. Famous Studios has the first ever virtual production setup in Bollywood.
“The rise of AR and VR has been growing exponentially over the years and has become a real part of the production workflow. In studio environments nowadays, using both has become a new trend, and is called Mixed Reality. Entertainment shows (The Voice, The VMA, NASCAR, Super Bowl), sports and esports (tournaments) are bringing more interactive graphics to their productions. Creating more realistic and animated graphics and backgrounds to their workflow has really given the viewer a new perspective of watching television,” explained Richard Villhaber, VP and sales director, Egripment.
VR has seen considerable investment in past decade, thus, advancing steadily as a technology that will gain mass adoption in the years to come. Each year is brings more innovation on hardware and software—accompanied by price affordability. These spatial computing tools have wearable variants to, which are extensively being used by R&D companies.
We’re witnessing large strides forward in graphics rendering technology. “Most of the commercial solutions available to content creators are now based on the Unreal4 engine from Epic Games, because it enables incredibly life-like, photo-realistic results. This is a large step forward for those working in the graphics field,” said Vivian Yu, marketing manager, APAC, Ross Video.
“AR gets an additional push from the render engines from the gaming industry, namely Unreal and Unity. Both have breathtaking quality and are very affordable, compared to the traditional render engines. But on the other hand, they require much more effort to create virtual sets, thus, requiring more human labour,” said Thorsten Mika, Thorsten Mika, GM and founder, TrackMen GmbH.
“Egripment’s mechanical tracking giving the best tracking one can offer for integration with an AR/VR setup, without any glitches. How? The position data is always there and therefore, will be available at all times. This, in comparison to other techniques such as optical tracking—where you need to make sure you are always in the line-of-sight of markers, or camera’s eye, to make sure you will receive the tracking data, is way better,” added Villhaber.
In certain segments, especially broadcast and live sports production, the use of AR/VR has risen significantly. The key objective behind using these tools is to curate powerful stories. Broadcasters leverage the technology to enhance their content, along with making it immersive and interactive. This is particularly important in a scenario, where the competition is strong and stakes are high. “The sports world has seen a new emphasis on fan engagement, as sports teams and stadia try to tempt audiences off their sofas and into the stadium for the live game experience. In both of these domains, there has been a welcome emphasis on creating a better ‘product’ through creative tools that are centred on audience engagement. We’ve all seen video clips of mythical animals or esports characters flying around the crowd on an enormous screen in the stadium or news programs displaying social media content during their bulletins on impressive LED display walls. These are just two examples of how content creators are trying to tell stories more effectively and, by association, differentiate their offering from those of their competitors,” added Yu.
“Imagine the ability to repair or install something with instruction from an expert in another part of the world. Also imagine learning Yoga from a digital yoga teacher demonstrating Asana’s right in your living room in front of you. Someone you can walk around to observe posture from any angle. There are many other use cases which eventually serve to make business better by providing clarity in communication—without the burden of being in each other’s physical locations,” explained Patel.
“Our marker-less tracking called VioTrack R, which can be applied to drone videos, only needs the live feed of the drone camera to determine all relevant parameters of the virtual camera model. Then you can immediately apply virtual objects to any place within the real scene. Augmentation was never so easy,” said Mika.
It will be some time before AR/VR can be leveraged to its full potential, for unique business needs. The prerequisite is to explore, test and incorporate AR VR as an extension of their services. Patel said, “The design rules for VR & AR content creation are very different from traditional content creation and production. The field is still evolving to form its own design language so different businesses will see drastically different uptakes in adopting AR VR.”
The latest trend is to integrate camera tracking data besides other metadata to an MXF stream. This brand new standard offers a container format for video and metadata, which can be stored or used live. “The costs will dramatically be reduced, due to the fact that AR and VR are replacing alternative technologies to a lower price (e.g. virtual sets replacing real scenario), while at the same time the components are getting inexpensive and create a photorealistic quality,” added Mika.
The way AR/VR can be optimised is subjective to each project’s unique needs, while considering the budget, time and roadmap. With a plethora of tools and services available at one’s disposal, how it can be presented and implemented still remains a people-business. “When we started about 10 years ago, we noticed that big companies offering the rendering engines, software and licenses, were quite expensive and in most cases the customers who already implemented that part also considered to buy camera setups that could be tracked and used in the AR/VR environment. Having the right skill set allows you nowadays to build something which is affordable and can do the same trick. The rise of the game engines, using Unreal and Unity, are becoming more popular and are more affordable since they are open source software,” elaborated Villhaber.
Yu explained, “While the capital expenditure is significant, there has always been a very solid business case for virtual in terms of return on investment and the ability to use the green screen space for multiple show formats/change virtual sets relatively quickly. Most recently, we’ve seen quite a few broadcasters using hybrid sets, where there is a traditional, physical component (e.g. furniture and seating for talent), but the set is then extended on either side to look larger than it really is using green screen and graphics – an approach that can look very impressive and be quite cost-effective.”
THE IFS AND BUTS
The biggest challenge in the face of AR and VR is its affordability and accessibility in a way convenient to the masses. “On the social front, AR poses a privacy challenge. A regulatory policy will have to be enforced as even more private data will be available to mega corporations, governments and mal-intentioned parties to exploit. VR on the other hand poses challenges of isolation. The more one gets used to being in digital simulated worlds the more they get disconnected with the present one. Everything in moderation as your grandmother would advise,” explained Patel.
Another challenge lies in the production workflow, ease of integration, and flexibility. Yu said, “I personally don’t see a downside to the technology; I believe it is empowering content creators to really improve the work they do and to engage more deeply with their audiences. As mentioned above, graphics production is still quite human-resource-heavy and that’s an area that will need to improve in the coming years (perhaps AI might have a role to play in this?) but the tools available are becoming increasingly powerful and that’s only a positive thing.”
REAL CASE STUDIES
Famous Studios just delivered the first automobile commercial in India using virtual production. It produced the entire commercial for the Mahindra concept car named Funster, which was unveiled at Auto Expo 2020 to a roaring applause. The car design, lighting and look was previsualised in VR and then an entire car reveal and drive sequence was choreographed using Real Time Technology.
The use of AR has changed the way new cars are demonstrated, offering the press a seat to enjoy the presentation, which is a high-end production with a variety of camera angles merged within an augmented set up. Egripment used this for the introduction of new cars (clients: Volkswagen, Porsche) to the press. “We used an Egripment T10 encoded telescopic crane that could view the car from all angles, while sending position data to the rendering engine. The viewer saw the result on a huge screen, where the different camera angles were merged with animated graphics and backgrounds as part of the show,” explained Villhaber.
CCTV’s Chinese Spring Festival Gala coverage was certified by Guinness World Records as the world’s most widely-viewed TV show, and it’s a real song and dance extravaganza that is broadcast on Chinese New Year’s Eve. “We’ve worked with CCTV for a few years on this show and there’s almost one billion pairs of eyes watching every year, so there is massive expectation and pressure. CCTV used Ross’ tools to help deliver the virtual worlds that serve as a backdrop to the various routines,” said Yu.
Imagine being present onsite an action scene from your favourite superhero movie, or being able to have a face-to-face chat with your favourite celebrity, perhaps? AR and VR can enable this. The future holds these technologies as sole communication platforms, providing enormous opportunity. “AR and VR may have specific use cases today but once the hardware is there in a convenient form factor at an affordable price, they touch every aspect of our modern society. It will enable communication like no other medium before it. It will unlock existence in worlds only depicted in science fiction. It will educate, save lives and have the potential to harm at the same time. How far will it go? As far as your imagination, and beyond…,” exclaimed Patel.
We will see photorealistic graphics which will not be distinguishable from real footage, all created live. “I wish I had a magic crystal ball, but I believe that we are heading to a world where we combine the virtual and real world as one. We already see this change in the game industry and similar technology has already been used in movies like Avatar and Star Wars. The possibilities are endless!” said Villhaber.
Soon, I reckon, there will be a blurred demarcation between the real and the virtual. We can only wait and watch!