# US 11,442,685 B2 · Remote interaction via bi-directional mixed-reality telepresence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,685 B2 - Document ID: US11442685B2 - Status: Granted - Kind: B2 - Application number: 16/903,303 - Filed: 2020-06-16 - Published: 2021-03-11 - Granted: 2022-09-13 - Earliest claimed: 2019-09-09 - Assignees: Autodesk, Inc. - Applicants: Autodesk, Inc. - Inventors: Fraser Anderson; Bala Thoravi Kumaravel; Tovi Grossman; George Fitzmaurice; Bjoern Hartmann - Web page: https://patentgrep.com/patent/US11442685B2 ## CPC - G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/147: using display panels - G09G 2354/00: Aspects of interface with display user - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract A technique for remote mixed-reality interaction between users includes determining a first position of a first object within a first three-dimensional (3D) space; generating first information associated with the first 3D space for the first object based on the first position; transmitting the first information to a computing device that renders first video content for display within a second 3D space based on the first information; and while transmitting the first information to the computing device, receiving second information that is associated with the second 3D space and with a second object and generated based on a second position determined for the second object within the second 3D space. ## Claims ### Claim 1 A computer-implemented method for remote mixed-reality interaction between users, the method comprising: determining a first position of a first object within a first three-dimensional (3D) space that is part of a first real-world environment; generating first information associated with the first 3D space, the first information including the first position; transmitting the first information to a computing device; while transmitting the first information to the computing device, receiving, from the computing device, second information associated with a second 3D space that is part of a second real-world environment, the second information including second position determined for a second object within the second 3D space; and displaying the first 3D space or the second 3D space, wherein displaying the first 3D space or the second 3D space includes rendering the first object based on the first position of the first object within the first 3D space and rendering the second object based on the second position of the second object within the second 3D space. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: rendering the second information into second video content; and causing the second video content to be displayed within the first 3D space. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising, prior to causing the second video content to be displayed within the first 3D space, temporally synchronizing the second video content with first video content based on the first information. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, further comprising, prior to causing the second video content to be displayed within the first 3D space, spatially synchronizing the second video content with first video content based on the first information. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the first information includes first 3D point-cloud information for the first 3D space. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising causing the first 3D space to be displayed within the second 3D space as augmented reality content. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the first object comprises a position-tracked object, and the first information includes 3D location information for the position-tracked object. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein determining the first position of the first object comprises receiving signals from a spatial tracker device coupled to the first object. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the first position of the first object within the first 3D space is determined while the second position of the second object within the second 3D space is determined. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the first information includes 3D video content. ### Claim 11 A system, comprising: a sensor configured to determine a first position of a first object disposed within a first three-dimensional (3D) space that is part of a first real-world environment; a memory that stores instructions; and a processor that is coupled to the memory and is configured to perform the steps of, upon executing the instructions: causing the sensor to determine the first position of the first object within the first 3D space; generating first information associated with the first 3D space, the first information including the first position; transmitting the first information to a computing device; and while transmitting the first information to the computing device, receiving, from the computing device, second information that is associated with a second 3D space that is part of a second real-world environment, the second information including a second position determined for a second object within the second 3D space; and displaying the first 3D space or the second 3D space, wherein displaying the first 3D space or the second 3D space includes rendering the first object based on the first position of the first object within the first 3D space and rendering the second object based on the second position of the second object within the second 3D space. ### Claim 12 (depends on 11) The system of claim 11, wherein the sensor comprises one of an IR depth sensor and a photodiode-based tracker configured to measure a timing of light pulses. ### Claim 13 (depends on 11) The system of claim 11, further comprising: a first 3D display device disposed within the first 3D space; and a second 3D display device disposed within the second 3D space, wherein the processor is further configured to perform the steps of, upon executing the instructions: rendering the second information into second video content; and causing the second video content to be displayed by the first 3D display device. ### Claim 14 (depends on 13) The system of claim 13, wherein the processor is further configured to cause the second video content to be displayed by the first 3D display device while the first 3D space is displayed by the second 3D display device. ### Claim 15 (depends on 13) The system of claim 13, wherein the first 3D display device comprises a first head-mounted augmented-reality (AR) display device, and the second 3D display device comprises a second head-mounted AR display device. ### Claim 16 (depends on 11) The system of claim 11, wherein the processor is further configured to determine the first position of the first object within the first 3D space by: determining a mixed reality mode associated with a second 3D display device disposed within the second 3D space and configured to display first video content based on the first information; based on the mixed reality mode, selecting a position measurement sampling rate; and performing one or more position measurements of the first object within the first 3D space at the position measurement sampling rate. ### Claim 17 (depends on 16) The system of claim 16, wherein the processor is further configured to, based on the mixed reality mode associated with the second 3D display device: select a 2D video sampling rate; and generate, at the 2D video sampling rate, 2D video of at least a portion of the first 3D space. ### Claim 18 (depends on 17) The system of claim 17, wherein the processor is further configured to transmit the 2D video of at least a portion of the first 3D space to the computing device in conjunction with the first information. ### Claim 19 (depends on 11) The system of claim 11, wherein the first information includes first 3D point-cloud information associated with a surface detected in the first 3D space. ### Claim 20 A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of: determining a first position of a first object within a first three-dimensional (3D) space that is part of a first real-world environment; generating first information associated with the first 3D space, the first information including the first position; transmitting the first information to a computing device; while transmitting the first information to the computing device, receiving, from the computing device, second information that is associated with a second 3D space that is part of a second real-world environment, the second information including a second position determined for a second object within the second 3D space; and displaying the first 3D space or the second 3D space, wherein displaying the first 3D space or the second 3D space includes rendering the first object based on the first position of the first object within the first 3D space and rendering the second object based on the second position of the second object within the second 3D space.