# US 12,594,496 B2 · Translation of sign language in a virtual environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,594,496 B2 - Document ID: US12594496B2 - Status: Granted - Kind: B2 - Application number: 18/296,054 - Filed: 2023-04-05 - Published: 2024-10-10 - Granted: 2026-04-07 - Earliest claimed: 2023-04-05 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Celeste Bean; Todd Tokubo - Web page: https://patentgrep.com/patent/US12594496B2 ## CPC - A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle - A63F 13/85: Providing additional services to players - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/58: by computing conditions of game characters, e.g. stamina, strength, motivation or energy level - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 40/44: Statistical methods, e.g. probability models - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06T 11/00: Two-dimensional [2D] image generation - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G09B 21/009: {Teaching or communicating with deaf persons (G09B21/04 takes precedence)} - A63F 13/211: using inertial sensors, e.g. accelerometers or gyroscopes - A63F 2300/5553: user representation in the game field, e.g. avatar - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract A method is provided, including the following operations: receiving video of a first user; processing the video to identify signed communications in a first sign language made by the first user; translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the user performing the translated signed communications in the second sign language; presenting the avatar on a display for viewing by a second user. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving video of a first user of a video game; processing the video to identify gestures made by the first user while the first user plays the video game; receiving context data indicating actions occurring in the video game when the gestures are made by the first user; determining, using the context data, whether the gestures made by the first user include signed communications in a first sign language; in response to determining that the gestures made by the first user include signed communications in the first sign language, translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the first user performing the translated signed communications in the second sign language; and presenting the avatar on a display for viewing by a second user. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the translating includes translating the signed communication in the first sign language into text in a first written language. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the translating further includes translating the text in the first written language into text in a second written language, and translating the text in the second written language into the signed communications in the second sign language. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the avatar is rendered in a virtual environment, and wherein presenting the avatar includes presenting a view of the virtual environment on the display. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the virtual environment is defined for gameplay of the video game. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the method enables substantial real-time communication between the first user and the second user. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, comprising: determining, using the context data, that the identified gestures made by the first user further include non-signing gestures made by the first user; when the non-signing gestures are made by the first user, rendering the avatar performing the non-signing gestures in a substantially similar manner to the first user; and when the signed communications in the first sign language are made by the first user, rendering the avatar performing the translated signed communications in the second sign language. ### Claim 8 A non-transitory computer-readable medium having program instructions embodied thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: receiving video of a first user of a video game; processing the video to identify gestures made by the first user while the first user plays the video game; receiving context data indicating actions occurring in the video game when the gestures are made by the first user; determining, using the context data, whether the gestures made by the first user include signed communications in a first sign language; in response to determining that the gestures made by the first user include signed communications in the first sign language, translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the first user performing the translated signed communications in the second sign language; and presenting the avatar on a display for viewing by a second user. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the translating includes translating the signed communication in the first sign language into text in a first written language. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the translating further includes translating the text in the first written language into text in a second written language, and translating the text in the second written language into the signed communications in the second sign language. ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the avatar is rendered in a virtual environment, and wherein presenting the avatar includes presenting a view of the virtual environment on the display. ### Claim 12 (depends on 8) The non-transitory computer-readable medium of claim 8, the operations comprising: determining, using the context data, that the identified gestures made by the first user further include non-signing gestures made by the first user; when the non-signing gestures are made by the first user, rendering the avatar performing the non-signing gestures in a substantially similar manner to the first user; and when the signed communications in the first sign language are made by the first user, rendering the avatar performing the translated signed communications in the second sign language. ### Claim 13 A system comprising at least one computing device, the at least one computing device having logic configured to implement operations comprising: receiving video of a first user of a video game; processing the video to identify gestures made by the first user while the first user plays the video game; receiving context data indicating actions occurring in the video game when the gestures are made by the first user; determining, using the context data, whether the gestures made by the first user include signed communications in a first sign language; in response to determining that the gestures made by the first user include signed communications in the first sign language, translating the signed communications in the first sign language into signed communications in a second sign language; rendering an avatar of the first user performing the translated signed communications in the second sign language; and presenting the avatar on a display for viewing by a second user. ### Claim 14 (depends on 13) The system of claim 13, wherein the translating includes translating the signed communication in the first sign language into text in a first written language. ### Claim 15 (depends on 14) The system of claim 14, wherein the translating further includes translating the text in the first written language into text in a second written language, and translating the text in the second written language into the signed communications in the second sign language. ### Claim 16 (depends on 13) The system of claim 13, wherein the avatar is rendered in a virtual environment defined for gameplay of the video game. ### Claim 17 (depends on 13) The system of claim 13, the operations comprising: determining, using the context data, that the identified gestures made by the first user further include non-signing gestures made by the first user; when the non-signing gestures are made by the first user, th rendering the avatar performing the non-signing gestures in a substantially similar manner to the first user; and when the signed communications in the first sign language are made by the first user, rendering the avatar performing the translated signed communications in the second sign language. ### Claim 18 (depends on 1) The computer-implemented method of claim 1, wherein: the context data indicates a gaze direction of the first user in a virtual environment of the video game when the gestures are made by the first user; and determining whether the gestures made by the first user include signed communication in the first sign language is based at least in part on the gaze direction of the first user in the virtual environment of the video game when the gestures are made by the first user. ### Claim 19 (depends on 1) The computer-implemented method of claim 1, wherein: the context data indicates a proximity between the first user and the second user in a virtual environment of the video game when the gestures are made by the first user; and determining whether the gestures made by the first user include signed communication in the first sign language is based at least in part on the proximity between the first user and the second user in the virtual environment of the video game when the gestures are made by the first user. ### Claim 20 (depends on 1) The computer-implemented method of claim 1, wherein: the context data indicates an intensity level of activity of the first user in a virtual environment of the video game when the gestures are made by the first user; and determining whether the gestures made by the first user include signed communication in the first sign language is based at least in part on the intensity level of activity of the first user in the virtual environment of the video game when the gestures are made by the first user.