Disability-Inclusive Communication Assistance based on Large Multimodal Models (LMM)

Overview

Although public services leveraging hyper-scale AI such as ChatGPT are gaining popularity, relatively little attention has been paid to developing services for persons with disabilities, particularly those with severe impairments.

As of 2022, approximately 250,000 individuals with brain lesions and 260,000 with developmental disabilities were reported in Korea. The 2023 Gyeonggi Province Survey on 24-Hour Care for persons with severe developmental disabilities indicated that nearly all respondents were classified as linguistic minorities, experiencing significant communication barriers with the public.

The project aimed to develop a service that supports mutual communication between persons with disabilities who are unable or have difficulty communicating and their parents, caregivers, or other third parties, by utilizing hyper-scale AI technology.

Project description

A hyper-scale multimodal Al Service was developed to recognize and infer communicative intent of persons with disabilities. Multimodal data—including facial expressions, voice, upper-body gestures, and lip movements—were tokenized and used to fine-tune an open-source model (Llama 3.0).

Reinforcement learning incorporating feedback from parents and caregivers in real-life settings further improved the model’s personalized recognition capabilities. The resulting service enables two-way communication between persons with disabilities and non-disabled individuals, enhancing daily care and support.

Key technological features

• Fine-tuning: Facial expressions, lip movements, upper-body gestures, and voice signals were converted into tokens and used to fine-tune an open-source model Llama 3.0 for the purpose of personalized intent recognition.
• Reinforcement Learning from Human Feedback (RLHF): The model’s ability to recognize individual communicative expressions in real-life environments was enhanced through reinforcement learning. Parents and caregivers provided feedback in the form of “correct” or “incorrect” responses to the system’s interpretations, which were incorporated into model updates to improve inference accuracy.

Expected outcomes

By providing communication tools and services that facilitate interaction between persons with disabilities and non-disabled individuals, the project has helped lay the foundation for a lifelong care society and improved labor-intensive care services. For persons with disabilities, the service offered opportunities for learning and enhanced both language acquisition and communication skills. For parents and caregivers, it provided much-needed respite and prevented mental and physical exhaustion.

Data for the project were collected through MOUs with Hanguk Woojin School, Angels’ Haven, Korea Parents' Association of Severe Multiple Cerebral Palsy, and Guro Brain Lesion Disability Vision Center. Following the service’s development (December, 2024), a satisfaction survey conducted among persons with disabilities and their caregivers who used the service recorded a high satisfaction score of 93.4 points.

Launch year: 2024

Attribution

Case study courtesy of the National Information Society Agency, Republic of Korea, Apolitical's Content Partner. Sourced from Use Cases of Public AI Service, Vol.1,  https://eng.nia.or.kr/site/nia_eng/ex/bbs/View.do?cbIdx=31975&bcIdx=28997&parentSeq=28997