Automatic Accessible Multimodal Translation of Graphics Using A Refreshable Pin Array

摘要

Accessibility of multimedia content for all users, particularly blind and low-vision individuals (BLVIs), remains a significant challenge. While screen readers assist BLVIs by converting text to speech via Alt-Text and image descriptions, these methods are inherently text-based and struggle to convey spatial and graphical information effectively. To help this, we propose a framework that converts graphical components into tactile graphics rendered on a refreshable pin array. Our framework leverages on-device AI models to generate tactile representations without transmitting personal data. It thereby minimizes processing time and mitigates privacy concerns. The benchmark test showed that our on-device AI outperformed GPU servers (RTX 4090) operating in an intranet environment. To optimize the tactile output and evaluate the system's effectiveness on media accessibility, we conducted a series of user studies with three different use case scenarios. First, we derived the optimal threshold values for edge detection in tactile graphics, which resulted in 70 on a 0-255 scale. Then, we compared the proposed system to a vision language model (VLM; GPT-4o). The results indicated that our proposed framework is more effective regarding both information delivery and subjective satisfaction. The proposed framework can be directly applied to several visual media accessibility scenarios, with the benefits of using local AI, such as privacy protection, personalization, and cost-effectiveness.