Facing technical risk, Tim Cook is hedging his bets on immersive tech.

Tech news site The Information published an exclusive story last week, revealing that Apple plans to release two AR headsets, the first in 2022 and the second in 2023. (Mark Gurman at Bloomberg followed up with some more detail.) This is big news for AR/VR, but even close followers of the industry may be confused about the credibility and meaning of this story.

I’ve been working in AR/VR for over 5 years. I was part of the leadership team at the Nokia OZO program, and my startup Imeve develops a real-time immersive communication platform called AVATOUR. I know a bit about the strategic landscape and the technology, so I thought it might be useful if I were to try to put this news in context.

Before getting into it, let me make clear that my analysis is based entirely on public information; I’m just combining it with my technical understanding to create something like a narrative. That said, here we go…

The Information is credible

Let’s start with this: This new Information article is on much more solid ground than earlier reports on Apple’s AR/VR plans. The Information is an investigative news organization with deep networks in Silicon Valley, and has proven to be a trustworthy source of inside tech news. (If you’re in tech, it’s worth every penny, go subscribe.)

They reported an October meeting “fill[ed] the 1,000-seat Steve Jobs Theater at Apple Headquarters,” attributing the details to “several sources.” I feel confident that almost any meeting of 1,000 people at a Silicon Valley tech firm will get back to the Information’s reporters from several sources. Surely, Apple’s famously secretive management knows this as well. Why would they share such sensitive information with such a large number of employees, virtually guaranteeing it would leak?

Only because they had no choice. This is a big project. Launching an entire new ecosystem of products — including hardware, operating system, and both native and third-party apps — requires Apple’s fanatically compartmentalized teams to cooperate over a long period of time. And to do that successfully, the high level schedule, form factor, and capabilities for the hardware need to be mutually understood well in advance of ship date.

The fact that super-secretive Apple decided to hold a 1,000-person internal meeting on its headset plan is itself confirmation of the scale and depth of Apple’s commitment to this project.

This confirms and contextualizes earlier reports

Earlier, more thinly-sourced reports on Apple’s AR/VR plans can be summarized as follows:

• 2016: Apple starts hiring AR/VR experts, notably Doug Bowman from Virginia Tech.
• Oct/Nov 2017: Tim Cook goes on record being bullish on AR: “A big idea like the smartphone…it’s huge.”
• March 2019: Ming-chi Kuo (historically accurate, with sources in Apple’s manufacturing chain) claims 2020 release date
• April 2019: Jaunt co-founder & CTO Arthur van Hoff comes on board
• July 2019: Taiwanese tech site Digi-times (historically spotty) reports the AR project was canceled
• Sept 2019: MacRumors notes the pre-release iOS13 has a glasses mode
• Oct 2019: Kuo reiterates a 2020 launch date

Taken together, these stories seem confusing or even contradictory. But the new information, for me at least, helps to crystalize those earlier stories into a complete narrative. Going out on a limb, here’s what I think actually happened:

• Cook’s commitment to AR/VR is real. He recognizes the category as a major future market opportunity for Apple.
• Apple had originally planned a 2020 release date. All technical development and planning, including hiring, targeted this tentative date.
• At some point, it became clear that the quality of experience Apple requires would not be possible on this timeline. At this point, the goals were changed and the team was restructured, and some hardware pursuits were shut down. (This corresponds to the “project canceled” leak.)
• Teams were sent back to the drawing board and came up with new proposals. These proposals included two headsets with the 2022 and 2023 ship dates.
• After a period of evaluation, these new plans were deemed feasible enough that target ship dates could be shared with the larger group.
• Internal teams need hardware prototypes to develop against; the Information also reported external developers would be engaged by 2021. My guess is that Kuo’s reported 2020 headset is now only a small production run of a prototype headset for use by internal and eventually external developers.

So why did the schedule slip? Technical challenges, I think. What technical challenges? I’m glad you asked…

What makes a great AR headset?

We don’t have any great AR headsets yet. Despite billions of dollars of capital and thousands of very smart people, deployed over many years, we don’t even have any very good AR headsets. Why?

Well, it’s hard. Take it from Michael Abrash, the gaming/AR/VR legend and now Chief Scientist at Oculus, who wrote “ Why You Won’t See Hard AR Anytime Soon “ back in 2012. Go read that article, if you haven’t already, and come back. The distinction he raises between “pass-through” and “see-through” AR is crucial to the rest of this post, so if you don’t know what those terms mean, then seriously, go read it.

… I’ll wait.

OK.

Since Abrash’s article, and just as he anticipated, the field has made enormous strides on the software side in a very short amount of time. Inside-out tracking, SLAM, and even object occlusion are basically solved problems at this point. Also as Abrash predicted, the hardware side is proving harder to crack. But some things that Abrash thought were easy are proving not so easy, and at least one thing he thought was hard hasn’t turned out to be so hard. And these hardware constraints are driving Apple’s product decisions.

What Abrash got wrong (back then)

Abrash is the best futurist in the world on AR/VR — but nobody gets everything right when predicting the future. In his 2012 article, he overestimated one key technical challenge for pass-through AR, and didn’t mention a key technical challenge for see-through AR.

Among the key drawbacks for pass-through, Abrash cited latency in the video pipeline:

…because there is lag between head motion and the update of the image of the world on the screen (due to the time it takes for the image to be captured by the camera, transmitted for processing, processed, and displayed), it tends to induce simulator sickness.

If you’ve used the Oculus Quest “Guardian” system, you understand what he’s talking about. This mode allows you to see the outside world in passthrough mode while you’re wearing the headset, in order to make sure you don’t bump into anything outside your play space. It’s really well done, but the delay — which feels to me like it’s on the order of 100msec — makes it pretty disconcerting. It’s certainly not something you can imagine enjoying for a long period of time.

It turns out the latency problem for pass-through was pretty much solved by a small startup called Vrvana, from Montreal. Their CES demo in Jan 2017 reportedly ran with 14ms latency; in June 2017, CEO Bertrand Nepveu claimed in a YouTube video that they got it down to 3ms. THREE MILLISECONDS! That’s unbelievable. They accomplished this feat with $350k of funding according to CrunchBase, and filed a patent on the technique back in 2010. (Turns out Palmer Luckey wasn’t the only VR true believer keeping the flame during the dark times.)

Apple bought Vrvana — including its pending patent application — in November 2017 for $30m. So Apple, and possibly only Apple, has a solution for the latency problem.

Now for see-though AR, Abrash’s 2012 article didn’t even mention what has turned out to be perhaps the biggest technical hardware challenge for such devices: field of view.

A frustratingly narrow field of view (FOV) is the Achilles’ heel of current see-through AR headsets. FOV is hard to measure in a consistent manner, but the waveguide technology at the heart of the Hololens 2 and Magic Leap displays has technical constraints that so far has kept the FOV in a narrow rectangle of about 40–50° diagonal. This is simply not enough. It has the effect of forcing augmented content to be viewed only through a very narrow window, dramatically diminishing the utility of current pass-through headsets.

Increasing FOV in a headset that also meets weight and power and brightness requirements is hard. Really hard. Deep tech breakthroughs are thin on the ground and the waveguide space is heavily patented — and it’s not clear that the waveguide approach can even theoretically get much better.

My guess is that Apple does not have a solution for the FOV problem. Yet. Going further out on my increasingly slender limb, perhaps the program that Digi-times reported canceled was an attempt at some new deep-tech approach to a wide-FOV see-through AR display that didn’t pan out. Presumably Apple has other approaches in the works, but there’s significant technical risk here.

Incidentally, it seems Apple might have a solution for a different key problem that Abrash identified — the transparency of augmented content in the real world, what Abrash calls “Soft AR,” which could also be called simply “opacity.” A recent patent filing indicates a technique for “selectively darken[ing] portions of the real-world light from view…to allow improved contrast when displaying computer-generated content over the real-world objects.” Not incidentally, this might allow the lens to be fully darkened, converting an AR (transparent) display to a VR (opaque) display. Grain of salt here, as a lot of patents don’t turn into reality, but clearly Apple is working on the opacity problem for see-through HMDs as well.

So given all that background, here’s what seems to be going on.

Apple is building BOTH a pass-through AND a see-through headset.

Apple has two goals in tension. They want to ship an amazing product, a category-defining product, a slam dunk. At the same time, pushing out shipment of any AR headset much past 2022 would risk ceding this market entirely to Microsoft and Facebook. Only a see-through headset could deliver the product slam dunk they want — but only a pass-through headset has low enough risk to guarantee the business requirement. So they’re hedging their bets, and building both.

The 2022 headset, as described in the Information article, is clearly a pass-through headset. With the Vrvana technique and patent, the technical risk is virtually nil. And with pass-through, in addition, opacity isn’t a problem either. For sure, there are challenges around weight, battery life, form factor, etc., but these things are all more engineering optimizations than they are actual technical risks to the viability of the project itself.

Apple internally announced a 2022 headset ship date now because they’re confident they can hit that target — but only with pass-through. All the internal teams can move forward with their own resource allocations with a high confidence that they won’t be wasting effort on a stillborn platform, and Apple can plan at the strategic level for the battle they will wage with Microsoft and Facebook on this terrain.

The 2023 headset is a see-through headset, and it’s the real technical challenge. There are almost certainly core technology problems that haven’t yet been solved as of today, and considerable schedule risk around those problems. It’s a stretch goal, based on untried technology, and there’s a good chance it will slip. But Apple will have staked its claim to the headset form factor with the 2022 headset, giving the 2023 team some room to nail the product without compromising the business priorities.

One more thing.

Apple is targeting both AR and VR

Apple is not building AR headsets. It’s building AR+VR headsets.

Both the 2022 and 2023 models will share the design goals of wide FOV and the ability to have synthetic content occlude (not just be occluded by) real-world content. The combination of these capabilities will make it easy to use the same hardware to render both AR (that is, partially world-occluding) and VR (that is, fully world-occluding) experiences.

We all need to stop thinking of AR and VR as opposed or even really fundamentally different technologies. They’re really just different display modalities that leverage mostly the same set of core capabilities. A sufficiently advanced AR headset subsumes the VR use case within it. Both Apple headsets are are designed to meet this goal.

Apple’s strategy: both, and.

So there you have it: Apple is pursuing a “both, and” strategy. Both pass-through and see-through, both AR and VR. I have no doubt they’re gonna crush it. I can’t wait.

‍