With the latest software, the XR-1 Developer Edition headset from Varjo delivers chroma keying in real-time, offering a seamless solution for blending the real and virtual worlds.

Green-screening! That’s the keyword for the newest development from Varjo, a company that in February 2019 introduced to the market the world’s first human eye-resolution VR headset, the VR-1. Now, with its XR-1 Developer Edition device, introduced commercially in December 2019, the company takes things further, allowing professionals to mix and match real-world elements with virtual content with pixel-perfect accuracy.

Green screening is a term that goes hand in hand with chroma keying, which is an industry-standard technique where parts of the real world are marked with a coloured canvas or paint… usually using green or blue. In the post-process, these areas are then detected and replaced with synthetic graphics.  The technique which was mostly used in big Hollywood productions is now widely used by everyone, including YouTubers, and is a key part of live broadcasting where mixed realities are in constant use.

Chroma key and marker tracking

Varjo takes the whole concept of chroma key, adds marker tracking, which allows to instantly anchor any virtual objects to the real world using printable visual markers, and delivers chroma keying in real-time for mixed reality devices. Together, these two features allow enterprise customers to seamlessly integrate virtual and real worlds, interact with photorealistic virtual content as they would in real life, and achieve pixel perfect accuracy and occlusion inside mixed reality.

“Since its commercial launch in December 2019, Varjo’s XR-1 Developer Edition has quickly become the most demanded mixed reality product for professional users, transforming the way companies train, design and conduct research in immersive environments,” said Urho Konttori, Chief Product Officer and co-founder of Varjo. “When our customers asked us to create a seamless solution for blending the real and virtual worlds, we immediately jumped to the challenge. We’re excited to introduce real-time chroma keying and object tracking to our customers just three months after the first deliveries of the XR-1, enabling absolute immersion inside mixed reality.”

A headset for total immersion in flight training

Chroma keying is particularly beneficial for professional workflows where aligning virtual content accurately with the physical world is crucial. Users can now easily define parts of reality, identify them with color and replace them with virtual models or scenery without heavy development costs. With chroma key, virtual content also occludes perfectly with real-world objects or hands, allowing intuitive interactions. Using Varjo’s object tracking with visual markers, professionals can make virtual objects appear exactly where they want them in their surroundings.

One example is flight simulation. Last year, when I wrote about the launch of the Varjo VR-1, I noted that the headset also worked with a flight simulation tool, Prepar3D, from Lockheed Martin, a Microsoft FSX on steroids used for professional training. I added, then, that “knowing Prepar3D and having compared it in Virtual Reality with Eagle Dynamic’s Digital Combat Simulator, which is much better in terms of visual quality, I can only wonder how fantastic the virtual skies in DCS would look with the Varjo VR-1, not to mention how easier it would be to read cockpit gauges and displays.”

Well, the XR-1 Developer Edition headset with the new features is compatible with Digital Combat Simulator (or DCS for short) and the video published by Varjo to demonstrate the concept is impressive. In it a pilot sits in a replica of a  F/A 18 Hornet cockpit, in DCs, and is able to look outside while flying through an ultra-immersive visual scenery, while operating physical cockpit controllers for realistic training. What’s mind-boggling is that the pilot is seen holding a piece of paper – which is real – with a before flight check list, before placing it on the throttle support. That’s something that is not possible to do with current VR headsets.

Seamlessly merging virtual and real worlds

“With chroma key, Varjo took an industry-standard technique and turned it into a useful new feature for dynamic mixed reality simulations,” said Bob Vaughn, Product Manager at FlightSafety International, a worldwide leader in aviation training. “We look forward to further exploring the feature applied to a variety of simulation opportunities. We highly value our collaborative relationship with Varjo, and are excited to continue to push the boundaries of mixed reality.”

Mixed reality applications offer huge cost savings for companies that traditionally train their staff in expensive simulators or build physical prototypes. While the XR-1 Developer Edition is a professional mixed reality device for engineers, researchers and designers who are pioneering a new reality, it’s not difficult to see the technology behind it being used for other purposes. With photorealistic visual fidelity, ultra-low latency and integrated eye tracking, the XR-1 seamlessly merges virtual content with the real world for the first time ever, and this opens new options in terms of exploring the world around us, and narrating stories using these technologies.

35 x the resolution of HTC Vive Pro

Both chroma keying and marker tracking are available in early access to all users of the XR-1 Developer Edition headset. The XR-1 is the only device capable of streaming high-resolution video of the reality to the user without any observable latency, enabling tweaking of every single pixel that the user sees inside the headset in photorealistic fidelity.  With XR-1, says Varjo, “we’ve solved the ‘hard AR’ problem, which exists on optical see-through devices like HoloLens or Magic Leap that always portray virtual elements as hazy and holographic and can’t display shadows or opaque content.”

The Varjo XR-1 may be announced as strictly professional, and its $9,995.00 price makes it inaccessible to most people, but its specifications point the direction towards which the future consumer solutions will move. The visual quality of the mixed reality is, according to Varjo, ultra-high definition – 4.6k to be exact – and, in comparison to existing products on the market, XR-1 runs at around 35 x the resolution of HTC Vive Pro’s mixed reality, and each individual pixel is of a much higher quality.

A headset for Virtual Production?

Varjo promised, back in 2018, when the company announced its upcoming VR headset, that it would be the first in the world to enable cinematic-quality Mixed Reality. At the time the company released a real-time video filmed entirely through its early Mixed Reality prototype to demonstrate the seamless merging of real and virtual worlds in cinematic fidelity. With the XR-1 we’ve, it seems, achieved that stage of development.

Varjo provides an API for third-party tracking technology providers to integrate their solution with Varjo headsets. Currently Varjo supports SteamVR tracking and ART tracking. With the addition of real-time chroma keying and marker tracking, the XR-1 looks like a headset that could be used in Virtual Production. Movies as The Lion King, that showed how Virtual Production can be in the future, could use a higher-resolution, cinematic fidelity headset like the Varjo XR-1.