Back to results
US 2026/0290309 A1Published application
System
AssigneeSoftBank Group Corp.
InventorHideki MORIWAKI
Life of this patent
Estimating the term…
Abstract
A system includes a processor that is configured to capture sign language using a camera and analyze captured video data to recognize content of the sign language, generate a prompt for instructing a generative AI model to generate text data so as to convert the recognized content of the sign language into the text data, and generate the text data based on the prompt by using the generative AI model, and convert the generated text data into audio data by using a speech synthesis technique and output the audio data through an audio output device.
Claims20
A system comprising:
- circuitry configured to:
- receive, via a communication interface coupled to a packet-switched network, video data captured by a camera of a terminal device, the video data comprising a sequence of image frames representing sign language gestures of a user;
- analyze the video data on a frame-by-frame basis using a gesture recognition model to extract hand shape features, hand position coordinates, and motion vectors from successive image frames, and recognize content of the sign language based on the extracted features;
- generate a prompt sentence for a generative neural network model based on the recognized sign language content, and transmit the prompt sentence to the generative neural network model to acquire text data representing the recognized content;
- obtain emotion-state information of the user by analyzing at least one of facial expression data and voice signal data received from the terminal device using an emotion recognition model;
- and
- apply speech synthesis processing to the text data to generate audio data, wherein a voice tone parameter of the speech synthesis processing is adjusted based on the emotion-state information, and transmit the audio data via the communication interface to the terminal device for output through an audio output device.
4 more elements
A system comprising:
- circuitry configured to:
- receive, via a communication interface coupled to a packet-switched network, video data captured by a camera of a terminal device, the video data comprising image frames representing sign language gestures of a user;
- apply a convolutional neural network to each image frame to extract hand keypoint coordinates, compute temporal motion vectors between successive frames, and apply a sequence model to the motion vectors to recognize content of the sign language;
- generate a prompt sentence for a generative neural network model based on the recognized sign language content, transmit the prompt sentence to the generative neural network model, and receive text data representing the recognized content;
- apply an emotion recognition model to at least one of facial landmark coordinates extracted from the video data and prosodic features extracted from voice signal data to determine an emotion label and an emotion intensity value;
- and
- apply speech synthesis processing to the text data with a voice tone parameter adjusted based on the emotion label and the emotion intensity value, and transmit resulting audio data to the terminal device for output.
4 more elements
A method comprising:
- receiving, via a communication interface coupled to a packet-switched network, video data captured by a camera of a terminal device, the video data comprising a sequence of image frames representing sign language gestures of a user;
- analyzing the video data on a frame-by-frame basis using a gesture recognition model to extract hand shape features, hand position coordinates, and motion vectors from successive image frames, and recognizing content of the sign language based on the extracted features;
- generating a prompt sentence for a generative neural network model based on the recognized sign language content, and transmitting the prompt sentence to the generative neural network model to acquire text data representing the recognized content;
- obtaining emotion-state information of the user by analyzing at least one of facial expression data and voice signal data received from the terminal device using an emotion recognition model;
- and
- applying speech synthesis processing to the text data to generate audio data, wherein a voice tone parameter of the speech synthesis processing is adjusted based on the emotion-state information, and transmitting the audio data via the communication interface to the terminal device for output through an audio output device.
3 more elements
Dependencies are inferred from the claim text.