Image Exploration Procedure Classification with Spike-timing Neural Network for the Blind

Individuals who are blind use exploration procedures (EPs) to navigate and understand digital images. The ability to model and detect these EPs can help the assistive technologies' community build efficient and accessible interfaces for the blind and overall enhance human-machine interaction. In this paper, we propose a framework to classify various EPs using spike-timing neural networks (SNNs). While users interact with a digital image using a haptic device, rotation and translation-invariant features are computed directly from exploration trajectories acquired from the haptic control. These features are further encoded as model strings through trained SNNs. A classification scheme is then proposed to distinguish these model strings to identify the EPs. The framework adapted a modified Dynamic Time Wrapping (DTW) for spatial-temporal matching with Dempster-Shafer Theory (DST) for multimodal fusion. Experimental results (87.05% as EPs' detection accuracy) indicate the effectiveness of the proposed framework and its potential application in human-machine interfaces.