# US 2026/0260579 A1 · System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0260579 A1
- Document ID: US20260260579A1
- Status: Published application
- Kind: A1
- Application number: 19/554,723
- Filed: 2026-03-03
- Published: 2026-09-03
- Earliest claimed: 2025-03-03
- Assignees: SoftBank Group Corp.
- Applicants: Softbank Group Corp.
- Inventors: Takuya NAGAI
- Web page: https://patentgrep.com/patent/US20260260579A1

## CPC

- G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)}
- G06V 10/82: using neural networks
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 21/10: Transforming into visible information
- G10L 25/63: for estimating an emotional state
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A sign language conversion system for facilitating communication between individuals using different sign languages or cultures, or between deaf individuals and hearing individuals. This system uses a high-resolution camera and motion sensor mounted on a terminal to capture sign language motions in real time and transmit them to a server. The server analyzes the motion data using a deep learning algorithm to recognize the sign language. The recognized sign language is converted by a conversion unit into sign language of a different language or into text, and is visually provided to the user on the terminal's display unit. This enables real-time communication between individuals using different sign languages or between deaf individuals and hearing individuals, thereby promoting social inclusion.

## Claims

### Claim 1

A system for real-time sign language conversion, comprising:
a terminal device including at least one imaging sensor configured to capture hand shape and motion of a user performing sign language, and
at least one motion sensor configured to acquire multidimensional motion data including hand position, movement speed, and direction;
a server device including one or more processors and a memory storing program instructions that, when executed by the one or more processors, cause the server device to receive the multidimensional motion data transmitted from the terminal device;
analyze the received motion data using a trained deep learning model to recognize an input sign language expression;
convert the recognized sign language expression into an output in a different sign language or a text representation; and
a display unit configured to output the converted sign language or text to at least one user in real time,
wherein the system enables bidirectional communication between users employing different sign languages or between a deaf user and a hearing user by performing sign language recognition and conversion with reduced latency compared to manual interpretation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the capture unit comprises the high-resolution camera and the high-precision motion sensor for capturing detailed hand shapes, positions, movement speed, and direction as multidimensional motion data in real time.

### Claim 3 (depends on 1)

The system of claim 1, wherein the analysis unit employs deep learning algorithms including a convolutional neural network (CNN) and a recurrent neural network (RNN) to identify hand shape features and motion patterns for sign language recognition.

### Claim 4 (depends on 1)

The system of claim 1, wherein the conversion unit is configured to translate the recognized sign language expression from a first sign language into a second sign language of a different language, by mapping the sign language motions of the first sign language to corresponding motions of the second sign language using natural language processing to preserve context and meaning.

### Claim 5 (depends on 1)

The system of claim 1, wherein the analysis unit utilizes context-aware processing to disambiguate similar sign language motions and select an appropriate sign based on contextual meaning, thereby improving recognition accuracy for signs that are otherwise ambiguous.

### Claim 6 (depends on 1)

The system of claim 1, further comprising a microphone and a speech recognition module configured to convert spoken words from a hearing individual into text, and wherein the conversion unit is further configured to convert the text into a corresponding sign language expression for display on the display unit via a customizable user interface, thereby facilitating smooth communication between the hearing individual and a hearing-impaired elderly user in a care facility environment.

### Claim 7

A computer-implemented method for real-time sign language conversion, the method comprising:
capturing, by a terminal device, sign language motions of a user using at least one camera and at least one motion sensor to generate multidimensional motion data including hand shape, position, movement speed, and direction;
transmitting the multidimensional motion data from the terminal device to a server device via a communication network;
analyzing, by one or more processors of the server device, the multidimensional motion data using a trained deep learning model to extract spatial and temporal features corresponding to an input sign language expression;
converting the recognized input sign language expression into an output in a different sign language or into a text representation while preserving semantic meaning; and
displaying the output sign language or text on a display device in synchronization with the captured sign language motions,
wherein all of the steps are executed continuously to enable real-time communication between users using different sign languages.

### Claim 8 (depends on 7)

The method of claim 7, wherein capturing the sign language motions includes using a high-resolution video camera and a high-precision motion sensor to collect detailed hand shape, position, and movement information as multidimensional data in real time.

### Claim 9 (depends on 7)

The method of claim 7, wherein analyzing the motion data comprises applying a deep learning recognition model that includes a convolutional neural network (CNN) to extract spatial features of the hand movements and a recurrent neural network (RNN) to analyze temporal sequences of the sign language motions.

### Claim 10 (depends on 7)

The method of claim 7, wherein converting the recognized sign language expression includes translating the expression from a first sign language into a second sign language of a different region or standard, such that an input in Japanese Sign Language is converted into an output in International Sign Language while preserving the original meaning.

### Claim 11 (depends on 7)

The method of claim 7, wherein the analyzing step further utilizes contextual information to improve accuracy by disambiguating similar signs, including accounting for variations in signer speed and ambient lighting conditions during recognition.

### Claim 12 (depends on 7)

The method of claim 7, further comprising capturing an audio input from a user via a microphone, converting the audio input into text using speech recognition, and converting the text into a corresponding sign language output on the display device, to enable communication from a hearing user to a deaf or hearing-impaired user in real time.

### Claim 13

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations comprising:
acquiring multidimensional sign language motion data captured by an imaging sensor and a motion sensor, the motion data including spatial and temporal hand movement information;
processing the motion data using a trained deep learning recognition model to generate a recognized sign language representation;
converting the recognized sign language representation into an output representation selected from a different sign language representation or a text representation using a language conversion model; and
generating display control data for presenting the output representation on a display device in real time,
wherein execution of the instructions reduces computational latency of sign language recognition and conversion relative to a rule-based sign language translation process.

### Claim 14 (depends on 13)

The computer-readable medium of claim 13, wherein the instructions for capturing the sign language motions cause the computing system to utilize a high-resolution camera and a motion sensor to capture detailed hand movement data as multidimensional information in real time.

### Claim 15 (depends on 13)

The computer-readable medium of claim 13, wherein the instructions for analyzing the motion data include instructions to apply a deep learning-based sign language recognition model comprising a convolutional neural network and a recurrent neural network, the model being configured to identify handshape features and motion patterns of the sign language from the motion data.

### Claim 16 (depends on 13)

The computer-readable medium of claim 13, wherein the instructions for converting the recognized sign language expression include instructions to translate the sign language expression from a first sign language into a second sign language different from the first, by mapping an input in Japanese Sign Language to an output in International Sign Language while preserving context and meaning using natural language processing.

### Claim 17 (depends on 13)

The computer-readable medium of claim 13, wherein the instructions further cause the computing system to present the converted output on a user interface that is customizable according to user preferences, allowing the user to adjust one or more display settings selected from font size and color for improved visibility.

### Claim 18 (depends on 13)

The computer-readable medium of claim 13, wherein the instructions further include instructions to transmit the motion data and the converted output via a secure, encrypted communication channel between a user-side device and a server, thereby protecting user privacy and ensuring low-latency, real-time performance of the sign language conversion system.
