Smartphone companies have been promising to take mobile photography to the next level and some giant steps have been made in the last decade, but we’re now at a point where drastic changes may be about to happen… and this means that all the smartphones that are around and have just been bought will be obsolete if the partnership between French company Prophesee and Qualcomm delivers on its promise of putting an end to motion blur and other image quality artefacts caused by conventional sensors, especially in high dynamic scenes and low light conditions, bringing Photography and Video closer to our true experiences.
If this is not another Lytro promise – we will fix it ion post – , we’re about to watch as new era in terms of smartphone photography, as both companies agreed on a multi-year collaboration to enable native compatibility between Prophesee’s Event-Based Metavision Sensors & Software and premium Snapdragon mobile platforms.
With development kits expected to be available from Prophesee this year, the goal is to to help accelerate mobile industry adoption of Prophesee’s solutions, which are announced as the “Holy Grail” able to add a new sensing dimension to mobile photography. According to the company, its technology is able to change the paradigm in traditional image capture by focusing only on changes in a scene, pixel by pixel, continuously, at extreme speeds.
Prophesee claims that each pixel in the Metavision sensor embeds a logic core, enabling it to act as a neuron, adding that “they each activate themselves intelligently and asynchronously depending on the amount of photons they sense. A pixel activating itself is called an event. In essence, events are driven by the scene’s dynamics, not an arbitrary clock anymore, so the acquisition speed always matches the actual scene dynamics.”
The way it works, high-performance event-based deblurring is achieved by synchronizing a frame-based and Prophesee’s event-based sensor. The system then fills the gaps between and inside the frames with microsecond events to algorithmically extract pure motion information and repair motion blur. The two images of the tenis player explain it in a simple way: a normal camera will register the blur, while Prophesee will create a sharp result. Which part of that is photography or video and which is AI woking to fill the gaps, is what some purists may question by now. The way forward, though, seem to be clear: less blur!
According to the companies, the technical and business collaboration will provide mobile device developers a fast and efficient way to leverage the Prophesee sensor’s ability to dramatically improve camera performance, particularly in fast-moving dynamic scenes (e.g. sport scenes) and in low light, through its breakthrough event-based continuous and asynchronous pixel sensing approach. Prophesee is working on a development kit to support the integration of the Metavision sensor technology for use with devices that contain next generation Snapdragon platforms.
“Prophesee is a clear leader in applying neuromorphic techniques to address limitations of traditional cameras and improve the overall user experience. We believe this is game-changing technology for taking mobile photography to the next level and our collaboration on both the technical and business levels will help drive adoption by leading OEMs,” said Judd Heape, VP, Product Management at Qualcomm Technologies, Inc. “Their pioneering achievements with event cameras’ shutter-free capability offer a significant enhancement to the quality of photography available in the next generation of mobile devices powered by Snapdragon, even in the most demanding environments, unlocking a range of new possibilities for Snapdragon customers.”
“We are excited to be working with the provider of one of the world’s most popular mobile platforms to incorporate event-based vision into the Snapdragon ecosystem. Through this collaboration, product developers will be able to dramatically enhance the user experience with cameras that deliver image quality and operational excellence not available using just traditional frame-based methods,” said Luca Verre, CEO and co-founder of Prophesee.
So, by the end of 2023, Prophesee hopes to have the ways to bring the speed, efficiency, and quality of neuromorphic-enabled vision to mobile devices through premium Snapdragon mobile platforms. Inspired by the human eye, Prophesee Event-Based sensors are set to help smartphone photography take a giant step forward to cross the gap still keeping them behind conventional cameras.
If the system works as suggested, then sports and action photography, which have been no-go areas for smartphones in general, become a new playground. Only recently I heard a conversation between professional photographers about how smartphones could or not could be used for sports photography. With the new of Prophesee Event-Based sensors, any conclusion accepted as valid then have to be reconsidered.