# US 11,308,672 B2 · Telepresence of users in interactive virtual spaces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,672 B2 - Document ID: US11308672B2 - Status: Granted - Kind: B2 - Application number: 15/929,990 - Filed: 2020-06-01 - Published: 2020-09-17 - Granted: 2022-04-19 - Earliest claimed: 2016-12-20 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment LLC - Inventors: Roxy Chaney - Web page: https://patentgrep.com/patent/US11308672B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay} - G09G 5/12: Synchronisation between the display unit and other units, e.g. other display units, video-disc players - G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence) - H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)} - H04N 9/3179: {Video signal processing therefor} - H04N 9/3194: {including sensor feedback} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G09G 2350/00: Solving problems of bandwidth in display systems - G09G 2354/00: Aspects of interface with display user - G09G 2370/02: Networking aspects - G09G 2370/22: Detection of presence or absence of input display information or of connection or disconnection of a corresponding information source - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A telepresence communication uses information captured by a first capture device about a first user and information captured by a second capture device about a second user to generate a first avatar corresponding to the first user and a second avatar corresponding to the second user. A scene can be rendered locally or by a remote server in which the first avatar and the second avatar are both rendered in a virtual space. The first avatar is rendered to move based on movements made by the first user as captured by the first capture device, and the second avatar is rendered to move based on movements made by the second user as captured by the second capture device. The avatars may be realistic, based on avatar templates, or some combination thereof. The rendered scene may include virtual interactive objects that the avatars can interact with. ## Claims ### Claim 1 A method for rendering three-dimensional virtual scenes, the method comprising: receiving a first visual dataset corresponding to a three-dimensional shape of a first user in a real-world environment, the first visual dataset captured by a first user device; rendering a three-dimensional virtual scene that includes a first three-dimensional avatar and one or more virtual objects, the first three-dimensional avatar rendered based on the first visual dataset; receiving sensor data including one or more measurements taken by one or more sensors in the real-world environment of the first user; identifying a trigger condition when the first three-dimensional avatar is detected interacting with at least one of the virtual objects, wherein identifying the trigger condition includes tracking boundaries of the first three-dimensional avatar and boundaries of the at least one virtual object; generating a visual overlay that displays the received measurements, the generated overlay appearing within the three-dimensional virtual scene that is updated based on the identified trigger condition associated with the detected interaction by the first three-dimensional avatar with the at least one virtual object; and transmitting the updated three-dimensional virtual scene to a second user device, the transmitted three-dimensional virtual scene including the first three-dimensional avatar interacting with the at least one virtual object and the visual overlay that displays the measurements. ### Claim 2 (depends on 1) The method of claim 1, further comprising: storing a plurality of trigger conditions in a trigger database in memory, each trigger condition corresponding to a different action, the plurality of stored trigger conditions including the trigger condition indentified within the three-dimensional virtual scene; and initiating the action corresponding to the identified trigger condition, the action initiated to update the three-dimensional scene. ### Claim 3 (depends on 2) The method of claim 2, further comprising storing one or more threshold values in memory, wherein detecting the trigger condition includes comparing the one or more measurements of the first user to the stored threshold values. ### Claim 4 (depends on 2) The method of claim 2, wherein the initiated action includes adjusting operation of a device connected to the first user device. ### Claim 5 (depends on 2) The method of claim 2, wherein the initiated action includes generating an alert associated with the detected trigger condition. ### Claim 6 (depends on 5) The method of claim 5, wherein the alert includes at least one of an audio noise, a visual indicator, and a message. ### Claim 7 (depends on 1) The method of claim 1, wherein the one or more sensors in the real-world environment of the first user include at least one of a heart rate monitor, a blood pressure monitor, a breathing monitor, and a haptic sensor, wherein the one or more measurements include at least one of a heart rate measurement, a blood pressure measurement, a breathing measurement, and a haptic measurement. ### Claim 8 (depends on 2) The method of claim 2, further comprising recognizing that the one or more measurements correspond to a gesture by the first user, wherein detecting the trigger condition is based on the recognized gesture. ### Claim 9 (depends on 1) The method of claim 1, wherein the visual overlay includes a virtual object that corresponds to the received measurements. ### Claim 10 (depends on 1) The method of claim 1, wherein the visual overlay includes a speech bubble that includes text corresponding to one or more spoken words associated with the three-dimensional virtual scene. ### Claim 11 (depends on 10) The method of claim 10, further comprising detecting that at least one of the spoken words is in a different language than a predetermined language associated with the three-dimensional virtual scene, and translating the at least one spoken word to generate the text within the speech bubble. ### Claim 12 (depends on 2) The method of claim 2, wherein the visual overlay includes a customized outline that corresponds to the first three-dimensional avatar, the detected trigger condition including an alignment of the first three-dimensional avatar to the customized outline, and wherein the initiated action is based on the alignment. ### Claim 13 (depends on 12) The method of claim 12, further comprising receiving a second visual dataset from the second user device, and generating a second three-dimensional avatar based on the second visual dataset, wherein the visual overlay further includes a different customized outline corresponding to the second three-dimensional avatar. ### Claim 14 (depends on 12) The method of claim 12, further comprising mapping a topography of the real-world environment based on one or more depth measurements in the first visual dataset. ### Claim 15 (depends on 1) The method of claim 1, further comprising generating a display of the three-dimensional virtual scene for the first user device, wherein the display generated for the first user device is different than a display of the three-dimensional virtual scene that is generated for the second user device. ### Claim 16 (depends on 2) The method of claim 2, wherein the initiated action includes generating haptic feedback via a haptic interface associated with the first user device. ### Claim 17 A system for rendering three-dimensional virtual scenes, the system comprising: a communication interface that communicates with a first user device over a communication network to: receive a first visual dataset corresponding to a three-dimensional shape of a first user in a real-world environment, the first visual dataset captured by the first user device; and receive sensor data including one or more measurements taken by one or more sensors in the real-world environment; and a processor coupled to memory storing instructions, wherein the processor executes the instructions stored in memory to: render a three-dimensional virtual scene that includes a first three-dimensional avatar and one or more virtual objects, the first three-dimensional avatar rendered based on the first visual dataset; identify a trigger condition when the first three-dimensional avatar is detected interacting with at least one of the virtual objects, wherein identifying the trigger condition includes tracking boundaries of the first three-dimensional avatar and boundaries of the at least one virtual object; and generate a visual overlay that displays the received measurements, the generated overlay appearing within the three-dimensional virtual scene that is updated based on the iidentified trigger condition associated with the detected interaction by the first three-dimensional avatar with the at least one virtual object, wherein communication interface transmits the updated three-dimensional virtual scene to a second user device, the transmitted three-dimensional virtual scene including the first three-dimensional avatar interacting with the at least one virtual object and the visual overlay that displays the measurements. ### Claim 18 (depends on 17) The system of claim 17, further comprising memory that stores a plurality of trigger conditions in a trigger database, each trigger condition corresponding to a different action, the plurality of stored trigger conditions including the trigger condition identified within the three-dimensional virtual scene; and wherein the processor executes further instructions to: initiate the action corresponding to the identified trigger condition, the action initiated to update the three-dimensional scene. ### Claim 19 A non-transitory computer-readable storage medium, having embodied thereon a program executable by a processor to perform a method for rendering three-dimensional virtual scenes, the method comprising: receiving a first visual dataset corresponding to a three-dimensional shape of a first user in a real-world environment, the first visual dataset captured by a first user device; rendering a three-dimensional virtual scene that includes a first three-dimensional avatar and one or more virtual objects, the first three-dimensional avatar rendered based on the first visual dataset; receiving sensor data including one or more measurements taken by one or more sensors in the real-world environment of the first user; identifying a trigger condition when the first three-dimensional avatar is detected interacting with at least one of the virtual objects, wherein identifying the trigger condition includes tracking boundaries of the first three-dimensional avatar and boundaries of the at least one virtual object; generating a visual overlay that displays the received measurements, the generated overlay appearing within the three-dimensional virtual scene that is updated based on the identified trigger condition associated with the detected interaction by the first three-dimensional avatar with the at least one virtual object; and transmitting the updated three-dimensional virtual scene to a second user device, the transmitted three-dimensional virtual scene including the first three-dimensional avatar interacting with the at least one virtual object and the visual overlay that displays the measurements.