Here’s a sad story. In the middle of 2023, my editor here at ProVideoCoalition was invited to Apple for the launch of something secret, but due to a prior commitment, he was unable to attend. Alas, the invitation was not transferable, and I had to watch from afar as the Apple Vision Pro was launched, and then released in the US early in 2024. Here in Australia, it’s finally arrived this month, and so we’re well overdue for a detailed look at how this thing is going to impact the lives of video professionals. Wherever you live, you’ll have seen plenty about this device already, so I’ll skip over most of the basics and approach this thing with a video industry lens.
Magic
When you put this thing on, even if you’ve used VR or AR headsets before, you’ll experience something totally new. The interface, based around eye tracking and hand tracking, lets you look at things to virtually hover over them, then tap your thumb and finger together to select them. It’s accurate enough to work well, and it’ll be your first “wow” moment in a long series of happy surprises. But you have to try it yourself. No video, no description can come close to actually experiencing it for yourself.
Windows stay where you put them, in your environment, as big as you like, even if you walk around or through them. Everything has depth, and persistence, and casts shadows. All the cameras and screens melt away, and you forget, for a time, that you’re looking at the world through a camera processed by a computer. Yes, you could say that Spatial Computing is just Apple’s term for augmented reality, but that’s missing the point. We haven’t really had anything quite like this before, and it deserves a new name.
Your environment can be simply where you are, using the passthrough cameras, or you can transport yourself to one of Apple’s beautifully designed alternatives: Mount Hood, Yosemite, Bora Bora (if you’re a developer running beta software) or one of a few others. Most environments include subtle animation, and some third-party apps include their own options too. It’s a remarkable experience, and I want more options.
While the passthrough experience isn’t as good as I expected, I suspect this is simply a difficult problem to solve, because small wide-angle cameras aren’t perfect. There’s a paradox here. Sometimes, especially when watching an immersive video, the experience is so good that the image quality doesn’t need to be perfect. At other times, especially when looking at stills, an image can be so huge and clear that any limitations of the camera become obvious. For example, most 360° video just doesn’t have enough pixels to look good in the Apple Vision Pro, and it turns out that 180° video is a better fit for video content. Halving the number of pixels improves image quality, gives you a place for the crew to stand, and it’s still immersive.
Immersion is the key
More than any other factor, the most exciting thing about this device is that it can make you feel like you’re somewhere else. When a dinosaur tries to eat you, you’ll try to kick it away. When a butterfly lands on your finger, you feel wonder. When your 88-year old mother-in-law is having the best time she’s had in years, riding a camel or wandering across Moroccan rooftops, you feel as much joy as she does.
Tech hasn’t made me this happy in decades. It’s utterly transcendent.
You can watch a virtual concert in which a singer looks into your eyes, then walks right up to you, turns, and flicks her hair in your face — if you want. You can walk a tightrope above a Norwegian fjord, or watch parkour through Paris. An immersive 180° 3D movie is a totally different experience than watching a regular 2D movie on a screen. In some ways it’s like theater — you can look around wherever you want, as long as that’s not behind you — but it’s a magical theater that could be anywhere on earth. If you’re a director, that might make you excited, but cinematographers and editors have more of a technical challenge and less of a creative one in front of them.
Unfortunately, a camera needs so many pixels to capture reality in 180° that you’re right up against the bleeding edge of what’s possible. The new Blackmagic URSA Immersive camera should have enough pixels to do the job, but it’s not yet available, and if you want to be picky, there’s not quite enough resolution in the limited (but growing) selection of movies available today. Still, that’s a pixel peeper talking — most people don’t care. When you’re on a tightrope above a fjord, most people aren’t counting pixels, they’re simply terrified. Immersion brings a lot to the table.
As well as immersive 3D content, many Hollywood movies are available in 3D — not immersive, but using normal focal lengths and standard filmmaking techniques. (Cinematographers and editors still have something new to explore after all.) Here in Australia, the list is sadly a bit smaller than it is in the US (where’s my Ready Player One?) but there are still a decent number of options in Disney Plus. Watching Gravity or Avatar in 3D is a more exciting experience than any 3D movie I’ve seen in the cinema, and if you’re into 3D movies, you’re not sacrificing brightness or resolution to achieve it — this is as good as it gets.
Adding 3D isn’t going to be for everyone, as the extra depth can arguably make a film less filmic and more like a video game, but I think most viewers will prefer it. This is doubly true for watching Avatar: The Way of Water at 48fps (the frame rate is variable, using 24fps for slower moments and 48fps for faster action) but it’s definitely engrossing.
Screens of any size and shape
If you take away immersion and 3D, the standard 2D movie experience is still great, and you’ll definitely still want to watch traditional content. After all, the most obvious use of this thing is to watch videos, and if you can find a sweet spot, it’s an absolutely great experience that I’ve enjoyed for binge sessions lasting hours. It may take some time to discover the perfect fit of head strap, head position and light seal (or absence thereof) but when you do nail that combo, there’s no better screen to be found anywhere.
When watching 2D content, you can make a display as big as a huge movie screen, wherever you are. The aspect ratio will always match the content, so you now have total freedom to watch whatever you want without seeing black bars. TikTok, cinemascope movies and 6K 4:3 open gate videos are now all on an equal footing, because there is no screen. A creator can target whatever aspect they wish, and the viewer can make the screen whatever size they find comfortable. Everybody wins.
In fact, because you’re also in control of the environment around your screen, you could choose to watch a movie on a virtual screen hovering above a rippling lake that reflects the movie. The only thing that breaks the illusion is that captions aren’t part of the reflection. Yet. Because the illusion is so complete, watching great content in the Apple Vision Pro feels a lot closer to the cinema experience than watching a big TV ever has. And to be fair, it’s not really possible to erect a large cinema screen in the middle of Yosemite Valley in mid-winter.
(A big note to streaming platforms who aren’t on board with native Apple Vision Pro apps yet — please, hurry up. Anyone with an Apple Vision Pro will prioritise natively viewable productions first, because the experience is so much better.)
So how good is this image?
It’s really very very good, but there are some minor issues. First, if you’re a picky pixel peeper like me, you may notice a loss of fine detail in smaller areas. This is less of an issue when you make a display larger, but there’s a natural limit to how much you can see in your field of view. You may also find that you can’t see everything at once with the same degree of clarity. This is mostly an issue when sending your Mac’s display through to the Apple Vision Pro, because there tends to be a lot of fine detail on a Mac screen, but it can happen when watching native video content too.
Part of the tech in use here is called foveated rendering, where only the part of the screen you’re looking at is rendered in detail, but that’s not the whole reason for the problem. Lenses aren’t perfect, and in particular, internal reflections can cause glare across your field of vision when bright areas are present somewhere on the screen. Sometimes, it’s a little like having blurry glasses, but it’s not a constant problem. Most of the time, with most content, it’s simply the biggest and best screen I’ve ever seen. When I’ve shot 6K open gate content in the past, I’ve always planned to zoom into it and deliver a 4K widescreen timeline, but… now I could just deliver all those pixels in 4:3 instead.
Never enough pixels
I’ve never owned a 6K screen, so I’ve never been able to see all the detail in the highest quality formats I’ve shot: the 6K videos, the 100MP still images. But now that I can make everything huge and walk right up to it, the difference between HD and 4K has never been more obvious, and the benefits of shooting stills at huge resolutions has never been clearer. Now, it’s not just about cropping, it’s about clarity when an image is the size of your wall.
One interesting limitation is that although the displays are as good as humans have made, and a large part of why this device is so expensive, they’re still not quite a match for the human eye. In one of the many art apps I’ve downloaded, I’ve been able to examine famous paintings up close, and it’s quite interesting to notice that you see more detail the closer you get to the images. The fine craquelure in Girl with a Pearl Earring, for example, is only apparent when it’s filling your vision, and there aren’t enough pixels to show those lines clearly from a distance. None of this is a problem when watching movies, but if you resize any detailed window (like a Mac Virtual Display) down a bit, you will probably notice a lack of fine detail.
Can you work in this thing?
Yes, but pixel-for-pixel it’s not going to be as clear as a big regular monitor — there just aren’t enough pixels in the state-of-the-art displays inside this thing. As a point of reference, if you resize a virtual window to the size of a regular 27” 4K display, it won’t look as good as a regular 4K display in that same space. The bigger something is within your field of view, the better it looks, because it’s able to use more of the pixels in the displays, but then it’s like working on a large TV — you’ll have to turn your head to see the larger display. In the next six months, as the new macOS and visionOS fine tune the Mac Virtual Display and 8K ultrawide resolutions become available, I expect this to change.
Right not, Mac Virtual Display is currently limited to one display. I’ve edited in it, and while it worked well, small text isn’t as clear as I would have liked, and all NLEs use small text in their UIs. The default resolution is 2560×1440, but for better clarity, I prefer to reduce the resolution to a retina 1920×1080 (smoother, with less on the screen). This can be OK — you don’t always need a detailed interface, and while I prefer plenty of screen space most of the way through an edit, I’ve happily performed a grading pass in FCP while sharing a single display through the Apple Vision Pro. Color is accurate, and the viewer was clear while in the UI or in full-screen mode.
If you’d like to use more than one screen, you’ll have to look to third-party solutions. I’ve tried a few, some of which try to share single Mac app windows rather than entire screens, but the Immersed app is the early stand-out. After a little tweaking, I’ve been able to sustain two 1080p Retina-class screens alongside a slightly lower resolution main screen, and everything’s usable with minimal lag — good enough to edit in, though still not as crisp as real screens. I suspect Apple’s promised future ultrawide screen support will end up as a good solution for most people, but it’s months away and Immersed is working well today.
If you work in 3D, you’ll need extra support, in the form of a dedicated Vision Pro helper app, to see your content in true 3D. There are a few apps today which show promise, but I’m still waiting for to a helper app for Blender to show me what I’m working on in 3D.
Conclusions
Creatively, I want to inspire joy in my viewers, and this device gives me more opportunities to do that. Besides making me want to use the best possible cameras, to experiment with the oddest aspect ratios, to take ultra-high resolution stills and many more panoramas, I’m also excited by the possibility of creating new virtual environments. Using Apple’s free Reality Composer Pro, this blends a bit of code, a bit of old-school interactive multimedia, 3D modelling and potentially still and video content too. If you ever enjoyed the freedom of designing a DVD-ROM multimedia experience back in the 1990s, it’s back, in 3D, all around you.
One of the common complaints reviewers have had with this device has been a lack of content, but we are exactly the people to fix that problem. The audience is small right now, but this device is compelling enough that it’s not going away, and Apple is hardly the only way to deliver 3D content (immersive or non-immersive) — many other devices are on the market and coming soon at a much lower price point.
Without the best screens available, and without the fully interactive spatial experience, competing devices might not look as good or feel as magical as this does, but that’s OK. Not everyone buys the same brand of camera or phone, and we can make content to be delivered in many different ways. While 3D workflows are going to get a lot easier soon, there are still a few hoops to jump through; look for an article helping you through that minefield soon.
So — is this the device for you? If you’re in a country where you can try ones, definitely do, and if you can purchase one, definitely consider it, but remember that you’ll be creating something totally new for a relatively small audience. While the user base will grow, it’s not going to hit the mainstream for some time yet. Why am I so excited, then?
There are only a few really big tech changes in a single lifetime, and life is short. Programming on my first Apple II-compatible computer, using my first Mac, seeing video editing with QuickTime 1.0 in the early 1990s, DV editing in the late 1990s, affordable widescreen DSLR HD video, the iPhone launch in 2007, and now, the Apple Vision Pro. It’s a whole new world to explore. If you can, jump in.