# US 11,908,056 B2 · Sentiment-based interactive avatar system for sign language > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,056 B2 - Document ID: US11908056B2 - Status: Granted - Kind: B2 - Application number: 17/240,128 - Filed: 2021-04-26 - Published: 2022-10-27 - Granted: 2024-02-20 - Earliest claimed: 2021-04-26 - Assignees: Rovi Guides, Inc. - Applicants: Rovi Guides, Inc. - Inventors: Yusuf AbdElhakam AbdElkader Marey; Reda Harb - Web page: https://patentgrep.com/patent/US11908056B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06F 40/47: Machine-assisted translation, e.g. using translation memory - G06T 13/205: {driven by audio data} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)} - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 21/10: Transforming into visible information - G10L 25/63: for estimating an emotional state - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G10L 2021/065: {Aids for the handicapped in understanding} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Systems and methods for doing presenting an avatar that speaks sign language based on sentiment of a speaker is disclosed herein. A translation application running on a device receives a content item comprising a video and an audio, wherein the audio comprises a first plurality of spoken words in a first language. The video comprises a character speaking the first plurality of spoken words in the first language. The translation application translates the first plurality of spoken words of the first language into a first sign of a first sign language. The translation application determines an emotional state expressed by the character based on sentiment analysis. The translation application generates an avatar that speaks the first sign of the first sign language where the avatar exhibits the determined emotional state. The content item and the avatar are presented for display on the device. ## Claims ### Claim 1 A method comprising: receiving a content item comprising a video and an audio, wherein the audio comprises a first plurality of spoken words in a first language, the video comprises a character speaking the first plurality of spoken words in the first language; translating the first plurality of spoken words of the first language into a first sign of a first sign language; determining an emotional state expressed by the character based on sentiment analysis; generating an avatar that performs the first sign of the first sign language, the avatar exhibiting the determined emotional state; querying a sign language database for a word of the first plurality of spoken words to identify the first sign of the first sign language, wherein the sign language database comprises visual content showing the first sign; based on the visual content of the first sign, determining one or more skeleton models that apply to the first sign for configuring the avatar to perform the first sign; and performing a transformation of the avatar using the determined one or more skeleton models, wherein performing the transformation comprises modifying one or more joints of the avatar based on the visual content of the first sign; and causing the content item and the avatar for display on a first device. ### Claim 2 (depends on 1) The method of claim 1, wherein the transformation of the avatar is based on a movement of at least one of a hand, a finger, an arm, or a face of the avatar. ### Claim 3 (depends on 1) The method of claim 1, further comprising: converting the first plurality of spoken words of the first language into a first text using one or more speech recognition algorithms, the first text comprises one or more words corresponding to the first plurality of spoken words. ### Claim 4 (depends on 1) The method of claim 1, wherein the sentiment analysis is performed by: determining an emotion identifier contained in the first plurality of spoken words; determining a facial expression or a body expression of the character using one or more expression recognition algorithms; and determining a vocal tone of the character using one or more voice recognition algorithms. ### Claim 5 (depends on 1) The method of claim 1, further comprising: receiving user input specifying a visual characteristic of the avatar, wherein an appearance of the avatar is modified based on the specified visual characteristic. ### Claim 6 (depends on 1) The method of claim 1, further comprising: identifying a visual characteristic of the character in the video based on image analysis, wherein an appearance of the avatar is modified based on the identified visual characteristic of the character in the video. ### Claim 7 (depends on 1) The method of claim 1, further comprising: receiving a user request to transmit the avatar from the first device to a second device; transmitting a configuration file that includes a visual characteristic of the avatar to the second device different from the first device; and causing the avatar for display based on the transmitted configuration file on the second device. ### Claim 8 (depends on 1) The method of claim 1, further comprising: receiving a user input specifying a command for the avatar to perform from the first device; determining a second sign corresponding to the command; and generating the avatar performing the second sign of the first sign language. ### Claim 9 A method comprising: receiving user input comprising audio input and video input from a first device, the audio input comprising a first plurality of spoken words in a first language in proximity to the first device, the video input comprising an image of a user while speaking the first plurality of spoken words; translating the first plurality of spoken words of the first language into a first sign of a first sign language; determining an emotional state expressed by the user based on sentiment analysis; generating an avatar that performs the first sign of the first sign language, the avatar exhibiting the determined emotional state of the user; querying a sign language database for a word of the first plurality of spoken words to identify the first sign of the first sign language, wherein the sign language database comprises visual content showing the first sign; based on the visual content of the first sign, determining one or more skeleton models that apply to the first sign for configuring the avatar to perform the first sign; and performing a transformation of the avatar using the determined one or more skeleton models, wherein performing the transformation comprises modifying one or more joints of the avatar based on the visual content of the first sign; and causing the avatar for display on the first device. ### Claim 10 (depends on 9) The method of claim 9, wherein the transformation of the avatar is based on a movement of at least one of a hand, a finger, an arm, or a face of the avatar. ### Claim 11 (depends on 9) The method of claim 9, further comprising: converting the first plurality of spoken words of the first language into a first text using one or more speech recognition algorithms, the first text comprises one or more words corresponding to the first plurality of spoken words. ### Claim 12 (depends on 9) The method of claim 9, wherein the sentiment analysis is performed by: determining an emotion identifier contained in the first plurality of spoken words; determining a facial expression or a body expression of the user using one or more expression recognition algorithms; and determining a vocal tone of the user using one or more voice recognition algorithms. ### Claim 13 (depends on 9) The method of claim 9, further comprising: receiving user input specifying a visual characteristic of the avatar, wherein an appearance of the avatar is modified based on the specified visual characteristic. ### Claim 14 (depends on 9) The method of claim 9, further comprising: receiving a user request to transmit the avatar from the first device to a second device; transmitting a configuration file that includes a visual characteristic of the avatar to the second device different from the first device; and causing the avatar for display based on the transmitted configuration file on the second device. ### Claim 15 (depends on 9) The method of claim 9, wherein the audio input is received at the first device via a microphone of the first device and the video input is received at the first device via a camera of the first device. ### Claim 16 (depends on 9) The method of claim 9, wherein the avatar is automatically generated in real time.