With the rise in popularity of Mixed Reality goggle kits, access to goggle cameras has become one of the most requested features for developers. In a recent interview, Meta's CTO suggested the possibility of including this feature in the future.
You certainly own a smartphone. But did you realise that its cameras are available to developers? In other words - some apps can see exactly what your phone's cameras see. Of course, when you open an app that uses cameras, you will be asked for permission, so developers can't access it whenever they want. This is a good thing, as people are naturally wary of camera access due to privacy concerns. However, this is a problem that has largely been solved in smartphones.
Smartphones have their own version of augmented reality, but the latest Mixed Reality goggles (and future goggles) offer a much better experience than what is possible on phones - You can interact with virtual content using your hands or controllers and see it in full 6-DoF in your space. Despite this significant advantage over AR on smartphones, it is currently impossible for developers to exploit its full potential due to the lack of access to cameras on goggles.
W interview between Matthew Ball and Andrew Bosworth (Meta's CTO), Bosworth suggested that camera access could eventually appear in Quest goggles. He expressed excitement about the possibilities this feature would open up for developers, but noted that Meta has taken a conservative approach to camera access due to privacy concerns. Given Meta's history with privacy scandals, this caution is understandable. However, as the Mixed Reality industry grows and with Apple already allowing camera access for corporate developers on the visionOS 2 (which suggests it may eventually be available to ordinary developers), Meta may have to follow suit.
Below are several examples of Mixed/Augmented Reality applications using camera access. Some are just ideas, while others are completed projects that function through (sometimes inconvenient) workarounds without direct camera access. Including this access could make these applications much more user-friendly.
Winner of the Meta AR Hackathon - "Flaivor"
Flaivor is an AI-powered cooking assistant that, based on photos of the items in your fridge (or any other photo of available food), provides recipes that include the photographed ingredients. It also has a timer-setting function and an AI assistant to provide help and guidance.
Wait, the app requires the user to take a photo? That's a clever (though not ideal) way to get around the lack of camera access, which would make the experience much simpler. Imagine if all you had to do was look at your fridge and focus on specific ingredients that a single photo might miss - this would be possible with camera access.
Space Vacuuming!
This concept app is made purely for fun, but offers an interesting idea for making housework more interesting and less monotonous. It includes features such as scoring coins for each spot not vacuumed and using different floor colours to indicate which parts have not yet been cleaned, as shown in the video. The developer achieved this by mounting the Meta Touch Pro controller, briefly seen in the video.
With access to cameras, it would be possible to track the hoover without the need for controllers. As floor and space data is already available to developers, camera access is the only missing ingredient.
Mixed Reality app on Quest 3 - "Pencil"
This application is particularly interesting. In fact, I had the opportunity to try it out myself! I drew one of the available drawings, and... well, it wasn't a masterpiece. But hey, it looked much better when I used this app than when I did it without it. (And no, I'm not going to show that drawing.)
This app teaches drawing, just as shown in the video on Twitter. What it does is simple: it tracks a sheet of A4 paper with a controller on top and adds a layer to help the user align it with the real sheet of paper. With access to a camera, developers could track the sheet of paper itself, making the process much easier. Currently, the controller can move, causing the virtual sheet of paper to shift, potentially causing a mismatch between the real and virtual drawing. Camera access would eliminate this problem.
"World Skins" - Transform your world with AI and AR
What's intriguing about this example is how it uses AI and camera access to transform the entire world around you. Although only a concept at the moment, this is something extremely interesting. We could discuss the potential risks here - such as what would happen if someone wasn't careful, but more important is the sheer possibility of creating something like this. For a safer, but equally impressive experience, imagine applying the same concept to your own home: transforming it to resemble a jungle or spaceship, complete with a view of space through your windows.
What is the benefit of using this over Virtual Reality? Well, one benefit could be the ability to see your furniture or people moving around you. Another benefit is the potential to use it outdoors, for example in the garden - an area that remains safe from hazards such as cars, but could be greatly enhanced by using AI.
AI assistant
Large Language Models (LLMs) are rapidly improving. For example, the one shown in GPT-4o can 'understand' your environment and context, thereby providing highly useful assistance when needed.
If Mixed Reality Goggles and AR glasses were given access to cameras for developers, they could use their trained models to help. For example, when you encounter a problem with a tap and need the help of a plumber, you could potentially deal with the problem yourself. How? Simply by looking at the installation and asking AI to help. With access to a camera, the AI would have immediate access to what you see, greatly increasing its ability to understand the situation and provide effective support.
Examples such as these are endless, and many have not even been invented yet. They could become reality if developers had access to cameras in VR/AR goggles.
Fascinating AR Meta Reality Lab research project
This is a very cool demonstration from Meta Reality Labs. It allows the user to manipulate reality, not only adding virtual elements, but also changing existing ones. For example, you can move objects in the real world and the goggle changes the texture and colours of the wall behind them, creating a virtual wall that is meant to mimic the real wall behind the moved object.
This virtual wall is an approximation, so it is not identical to the real one, but there is a lot of room for improvement here. The developer behind the project mentioned that careful scanning and pre-planning of each object was necessary for the demonstration. However, with access to cameras and advanced AI models, we can imagine rendering all of this in real time, which would raise the usability of Mixed Reality to a whole new level.
Changing the world around you dynamically - Aldiffusion API
The prototype shown here is an AI-powered demonstration of the possibilities. There's not much to describe it - you touch something in your room and your hands (or body) changes to match what you've touched. Is it a gadget? Yes, definitely. However, ideas like these could evolve into something bigger and much more useful!
AR for packaging
What if, in the future, you walked into a shop with your AR glasses, and every product you saw had a creative animation? It could be overwhelming, and I can't imagine anyone would enjoy seeing hundreds of products on shelves that move. But if you could focus on a product, click your fingers and see an animation showing how to use it or show potential recipes using it (depending on whether it's a food product or something else), that would be amazing.
There are undoubtedly considerable benefits to be seen. Of course, it would require access to cameras to work.
'Try-On' augmented reality solution from Gucci
Another interesting example of AR use requiring access to the camera is the 'try-on' function. You can already do this on your phone, but it is not well supported. With a device capable of altering reality in a useful way already on your head, developers could extend features like this. Personally, I would definitely use this to see how the shoes look on my feet before buying them, without having to go to the shop!
The same could apply to looking in the mirror when goggles or glasses see your body. You could check if certain clothes fit you well. There are already apps that do something like this, but nothing beats seeing the image directly from your point of view on your glasses or goggles.
There are many more examples like the ones above, but I think the one I have given gives a good picture of what is possible. The Met and Apple are slowly moving towards giving developers access to the cameras in their goggles. One can predict that if this happens, it will be the moment when a mass of innovative, creative and useful applications for Mixed Reality will emerge.
