World’s First Neuromorphic In-Cabin Sensing is an Automotive Game-Changer

Traditional imaging-based sensing poses privacy concerns that make deploying a personalized in-cabin experience harder than it needs to be. To overcome such concerns, sensor fusion needs to become a reality. In fact, a neuromorphic driver monitoring solution could solve those issues and help car manufacturers (OEMs) offer privacy by design.  

As industry veterans, we took it upon ourselves to imagine a neuromorphic in-cabin system that could offer the privacy drivers need, the safety they expect, and the efficiency OEMs look for. Our novel approach implied re-imagining in-cabin sensing algorithms, while handling short- and long-term periods of no motion.  

We leveraged Prophesee’s time-event sensor as it can capture information at an equivalent frame rate of 10,000 frames per second without needing active illumination. We then applied our image processing algorithms to the raw feed provided by the sensor, focusing on those areas with movement and that were of interest to the features we were looking for.  

Besides detecting and tracking the gaze of the driver, their head pose, and other factors critical to driver and occupant monitoring, the nature of the sensor gave us the opportunity to consider micro-expressions. This is vital when deploying a Driver Monitoring System (DMS) because the most subtle facial cues can be lifesavers on the road.  

Leveraging the detection of fine in-cabin motion, this new DMS solution can understand very well if the driver is drowsy or inattentive. At the same time, because the time-event sensor records only motion and changes in brightness, the privacy of the driving experience is not affected 

The solution is all-around beneficial to OEMs. It is cheaper than state-of-the-art imaging CMOS sensor-based solutions since it does not need active illumination; it involves simple light capturing and analog to digital conversion. In our tests, it has proven to be fast and low latency, with low energy consumption.  

These benefits make it ideal for driver monitoring systems and a cornerstone for next-gen custom in-cabin experiences. It also enables the automotive industry to innovate further, while respecting present and future regulations.  

By meeting drivers’ and passengers’ need for privacy, our unique neuromorphic in-cabin solution eliminates a key obstacle in OEMs roadmaps, while achieving a new standard of safety and privacy.  

Joseph Lemley leads Xperi’s Sensing Group from the office in Galway Ireland where he works on novel algorithms and artificial neural networks for upcoming sensor technologies. His research interests include artificial intelligence, deep learning, and computer vision.