A Multimodal Deep Learning-Based Road-Crossing Assistance System for Visually Impaired Pedestrians Using Zebra Crossing Detection and Traffic Sound Analysis

Abstract

Crossing the road safely continues to be among the top challenges for blind individuals in urban settings. Traditional assistive devices like white canes and guide dogs suffer from the awareness of dynamic traffic situation and also fail to provide enough environmental information at pedestrian crossings. This work presents an AI-based Multimodal Road-Crossing Assistance System, which enables visually impaired pedestrians on-road-crossing activities by integrating the detection of the zebra crossing with traffic sound analysis. The new system implements a computer vision model based on deep learning to visually identify zebra crossings in footage recorded from the wearer's wearable / smartphone device's camera. A CNN identifies the geometric pattern associated with pedestrian crossings and estimates the crossing path orientation. Simultaneously, an audio analysis module analyzes traffic sounds in the nearby surroundings with a recurrent neural network (RNN) model together with features extracted from spectrograms to detect distant cars, vehicle density and even light changes. A decision-level multimodal framework is proposed to fuse these two modalities to ensure trustworthy crossing guidance. It provides real-time auditory and vibration feedback for guiding the user in entering the correct side as well as warning about possible risks. We conducted an experimental evaluation of our approach on urban pedestrian crossing scenes in a dataset with traffic audio samples. The proposed model provided zebra crossing detection accuracy of 96.8%, traffic sound classification accuracy of 94.3% and a general crossing assistance reliability of 95.2%, thus surpassing multiple existing single-modal assistive systems. Our findings show that by combining visual and acoustic steps, the situational awareness of blind pedestrians can be significantly increased. Thus, the adaptive multimodal assisting framework presented here is a low-cost, lightweight and smart mobility solution that could provide much-needed safety support for pedestrians in a smart city context.