# US 11,113,863 B2 · Rendering content in a 3D environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,863 B2 - Document ID: US11113863B2 - Status: Granted - Kind: B2 - Application number: 16/807,779 - Filed: 2020-03-03 - Published: 2020-09-03 - Granted: 2021-09-07 - Earliest claimed: 2016-12-30 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Aayush Himanshu Upadhyay; Neel Mamidi Rao - Web page: https://patentgrep.com/patent/US11113863B2 ## CPC - G06T 15/005: {General purpose rendering architectures} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06T 19/003: {Navigation within 3D models or images} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems, methods, devices, and other techniques for rendering content in a 3D environment. In some implementations the system includes a memory subsystem, a communications interface, a rendering engine, an input handling apparatus, and a virtual object manager. The memory subsystem is configured to store first data that (i) defines a three-dimensional (3D) environment and (ii) identifies a virtual object in the 3D environment. The communications interface is configured to transmit requests over a network for third-party content to display with the virtual object in the 3D environment and to receive third-party content responsive to the requests. The rendering engine is configured to use the first data from the memory subsystem to render the 3D environment for presentation on a display device, including rendering the virtual object at a specified location of the 3D environment in a first mode in which the virtual object displays a first set of third-party content. ## Claims ### Claim 1 A computer-implemented method, comprising: displaying, on a display of a computing system, a three-dimensional (3D) environment; prior to user interaction with a virtual object in the 3D environment, rendering the virtual object in a first display mode in which the virtual object presents a first set of third-party content, wherein the first display mode has a first computational demand for rendering the virtual object in the first display mode; and in response to detecting user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, re-rendering the virtual object in the 3D environment in a second display mode in which the first set of third-party content is replaced with a second set of third-party content on the virtual object, wherein the second display mode has a second computational demand for rendering the virtual object that is higher than the first computational demand for rendering the virtual object in the first display mode, wherein the virtual object includes an instruction for re-rendering the virtual object in the second display mode from the first display mode, wherein the computing system receives the instruction prior to user interaction with the virtual object in the 3D environment. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: in response to detecting the user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, sending from the computing system a request for new third-party content; and receiving, by the computing system, the second set of third-party content as a response to the request for new third-party content. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the second set of third-party content is selected by an entity other than a user of the computing system. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein a type of the second set of third-party that is presented on the virtual object in the second display mode has a higher computational demand for rendering than a type of the first set of third-party content that is presented on the virtual object in the first display mode. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the type of the first set of third-party content is images and the type of the second set of third-party content is video. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein: the display is a head-mounted display; and the 3D environment is a virtual reality (VR) environment that is configured to be viewed using the head-mounted display. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising: in response to detecting the user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, triggering generation of a user notification outside of the 3D environment, the user notification being related to a topic or entity associated with the first set of third-party content. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the instruction identifies program code to execute for re-rendering the virtual object in the second display mode, and the method further comprises executing the program code according to the instruction in response to detecting user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode. ### Claim 9 One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more processors of a computing system, cause performance of operations comprising: displaying, on a display of the computing system, a three-dimensional (3D) environment; prior to user interaction with a virtual object in the 3D environment, rendering the virtual object in a first display mode in which the virtual object presents a first set of third-party content, wherein the first display mode has a first computational demand for rendering the virtual object in the first display mode; and in response to detecting user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, re-rendering the virtual object in the 3D environment in a second display mode in which the first set of third-party content is replaced with a second set of third-party content on the virtual object, wherein the second display mode has a second computational demand for rendering the virtual object that is higher than the first computational demand for rendering the virtual object in the first display mode, wherein the virtual object includes an instruction for re-rendering the virtual object in the second display mode from the first display mode, wherein the computing system receives the instruction prior to user interaction with the virtual object in the 3D environment. ### Claim 10 (depends on 9) The one or more non-transitory computer-readable media of claim 9, wherein the operations further comprise: in response to detecting the user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, sending from the computing system a request for new third-party content; and receiving, by the computing system, the second set of third-party content as a response to the request for new third-party content. ### Claim 11 (depends on 10) The one or more non-transitory computer-readable media of claim 10, wherein the second set of third-party content is selected by an entity other than a user of the computing system. ### Claim 12 (depends on 9) The one or more non-transitory computer-readable media of claim 9, wherein a type of the second set of third-party that is presented on the virtual object in the second display mode has a higher computational demand for rendering than a type of the first set of third-party content that is presented on the virtual object in the first display mode. ### Claim 13 (depends on 12) The one or more non-transitory computer-readable media of claim 12, wherein the type of the first set of third-party content is images and the type of the second set of third-party content is video. ### Claim 14 (depends on 9) The one or more non-transitory computer-readable media of claim 9, wherein: the display is a head-mounted display; and the 3D environment is a virtual reality (VR) environment that is configured to be viewed using the head-mounted display. ### Claim 15 (depends on 9) The one or more non-transitory computer-readable media of claim 9, wherein the operations further comprise: in response to detecting the user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, triggering generation of a user notification outside of the 3D environment, the user notification being related to a topic or entity associated with the first set of third-party content. ### Claim 16 A computing system, comprising: one or more processors; and one or more computer-readable media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: displaying, on a display of the computing system, a three-dimensional (3D) environment; prior to user interaction with a virtual object in the 3D environment, rendering the virtual object in a first display mode in which the virtual object presents a first set of third-party content, wherein the first display mode has a first computational demand for rendering the virtual object in the first display mode; and in response to detecting user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, re-rendering the virtual object in the 3D environment in a second display mode in which the first set of third-party content is replaced with a second set of third-party content on the virtual object, wherein the second display mode has a second computational demand for rendering the virtual object that is higher than the first computational demand for rendering the virtual object in the first display mode, wherein the virtual object includes an instruction for re-rendering the virtual object in the second display mode from the first display mode, wherein the computing system receives the instruction prior to user interaction with the virtual object in the 3D environment. ### Claim 17 (depends on 16) The computing system of claim 16, wherein the operations further comprise: in response to detecting the user interaction with the virtual object in the 3D environment while the virtual object is rendered in the first display mode, sending from the computing system a request for new third-party content; and receiving, by the computing system, the second set of third-party content as a response to the request for new third-party content. ### Claim 18 (depends on 17) The computing system of claim 17, wherein the second set of third-party content is selected by an entity other than a user of the computing system. ### Claim 19 (depends on 16) The computing system of claim 16, wherein a type of the second set of third-party that is presented on the virtual object in the second display mode has a higher computational demand for rendering than a type of the first set of third-party content that is presented on the virtual object in the first display mode. ### Claim 20 (depends on 19) The computing system of claim 19, wherein the type of the first set of third-party content is images and the type of the second set of third-party content is video. ### Claim 21 (depends on 16) The computing system of claim 16, wherein: the display is a head-mounted display; and the 3D environment is a virtual reality (VR) environment that is configured to be viewed using the head-mounted display.