# US 11,308,716 B1 · Tailoring a computer-generated reality experience based on a recognized object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,716 B1 - Document ID: US11308716B1 - Status: Granted - Kind: B1 - Application number: 16/583,151 - Filed: 2019-09-25 - Granted: 2022-04-19 - Earliest claimed: 2018-09-27 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Ian M. Richter - Web page: https://patentgrep.com/patent/US11308716B1 ## CPC - G06V 20/20: in augmented reality scenes - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2219/2004: Aligning objects, relative positioning of parts - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract In various implementations, a method is performed at a device including a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, the method includes detecting a representation of an object that is associated with computer-generated reality (CGR) content. In some implementations, the method includes obtaining a user environment map characterizing a user environment, wherein the user environment is limited by a volumetric region around the device. In some implementations, the method includes mapping a portion of the CGR content associated with the object to a portion of the user environment map. In some implementations, the method includes synthesizing a CGR environment in accordance with the mapping. ## Claims ### Claim 1 A method comprising: at a device including a display, a non-transitory memory and one or more processors coupled with the display and the non-transitory memory: detecting a representation of an object that is associated with computer-generated reality (CGR) content; obtaining an environment map characterizing an environment that includes a plurality of planar surfaces, wherein the environment is bounded by depth information characterizing a volumetric region around the device; selecting, based on a spatial criterion, a subset of the plurality of planar surfaces that are greater than a threshold size while filtering out planar surfaces that are smaller than the threshold size; and displaying, on the display, a CGR environment by overlaying a portion of the CGR content associated with the object onto the subset of the plurality of planar surfaces. ### Claim 2 (depends on 1) The method of claim 1, wherein detecting the representation of the object comprises: obtaining an image captured by an image sensor; and detecting the representation of the object in the image, wherein the object includes a real-world object. ### Claim 3 (depends on 2) The method of claim 2, wherein the image sensor includes an outward-facing camera coupled with the device, and wherein obtaining the image comprises capturing the image via the outward-facing camera. ### Claim 4 (depends on 1) The method of claim 1, wherein detecting the representation of the object comprises: displaying representations of a plurality of virtual objects; and detecting a selection of a representation of a first virtual object of the plurality of virtual objects. ### Claim 5 (depends on 1) The method of claim 1, wherein obtaining the environment map comprises: synthesizing a mesh map of a physical environment surrounding the device. ### Claim 6 (depends on 1) The method of claim 1, wherein obtaining the environment map comprises generating a map of an existing CGR environment associated with the device; and wherein synthesizing the CGR environment in accordance with the mapping comprises modifying the existing CGR environment in accordance with the mapping. ### Claim 7 (depends on 1) The method of claim 1, wherein displaying the CGR environment comprises: displaying an affordance to modify a visual presentation of the environment based on the CGR content associated with the object; detecting a selection of the affordance; and displaying the CGR environment in response to detecting the selection of the affordance. ### Claim 8 (depends on 1) The method of claim 1, wherein selecting the subset based on the spatial criterion comprises: selecting planar surfaces with a surface area that is greater than a threshold area; and filtering out planar surfaces with a surface area that is less than the threshold area. ### Claim 9 (depends on 1) The method of claim 1, wherein selecting the subset based on the spatial criterion comprises: selecting planar surfaces that intersect with a line of sight of a user of the device; and filtering out planar surfaces that do not intersect with the line of sight of the user of the device. ### Claim 10 (depends on 1) The method of claim 1, wherein selecting the subset based on the spatial criterion comprises: selecting planar surfaces that are located more than a threshold distance from a user of the device; and filtering out planar surfaces that are located less than the threshold distance from the user of the device. ### Claim 11 (depends on 1) The method of claim 1, wherein overlaying the portion of the CGR content onto the subset of the plurality of planar surfaces comprises: overlaying, onto a first planar surface in the subset, a first video, a first image or a first text string that is associated with the object; and overlaying, onto a second planar surface in the subset, a second video, a second image or a second text string that is associated with the object. ### Claim 12 (depends on 1) The method of claim 1, wherein overlaying the portion of the CGR content onto the subset of the plurality of planar surfaces comprises: overlaying, onto a first planar surface in the subset, a video that is associated with the object; overlaying, onto a second planar surface in the subset, an image that is associated with the object; and overlaying, onto a third planar surface in the subset, a text string that is associated with the object. ### Claim 13 A device comprising: one or more processors; a display; a non-transitory memory; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: detect a representation of an object that is associated with computer-generated reality (CGR) content; obtain an environment map characterizing an environment that includes a plurality of planar surfaces, wherein the environment is bounded by depth information characterizing a volumetric region around the device; select, based on a spatial criterion, a subset of the plurality of planar surfaces that are greater than a threshold size while filtering out planar surfaces that are smaller than the threshold size; and display, on the display, a CGR environment by overlaying a portion of the CGR content associated with the object onto the subset of the plurality of planar surfaces. ### Claim 14 (depends on 13) The device of claim 13, further comprising an image sensor, and wherein detecting the representation of the object comprises: obtaining an image captured by the image sensor; and detecting the representation of the object in the image, wherein the object includes a real-world object. ### Claim 15 (depends on 14) The device of claim 14, wherein the image sensor includes an outward-facing camera coupled with the device, and wherein obtaining the image comprises capturing the image via the outward-facing camera. ### Claim 16 (depends on 13) The device of claim 13, wherein detecting the representation of the object comprises: displaying representations of a plurality of virtual objects; and detecting a selection of a representation of a first virtual object of the plurality of virtual objects. ### Claim 17 (depends on 13) The device of claim 13, wherein obtaining the environment map comprises: synthesizing a mesh map of a physical environment surrounding the device. ### Claim 18 (depends on 13) The device of claim 13, wherein obtaining the environment map comprises generating a map of an existing CGR environment associated with the device; and wherein synthesizing the CGR environment in accordance with the mapping comprises modifying the existing CGR environment in accordance with the mapping. ### Claim 19 A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device that includes a display, cause the device to: detect a representation of an object that is associated with computer-generated reality (CGR) content; obtain an environment map characterizing an environment that includes a plurality of planar surfaces, wherein the environment is bounded by depth information characterizing a volumetric region around the device; select, based on a spatial criterion, a subset of the plurality of planar surfaces that are greater than a threshold size while filtering out planar surfaces that are smaller than the threshold size; and display, on the display, a CGR environment by overlaying a portion of the CGR content associated with the object onto the subset of the plurality of planar surfaces. ### Claim 20 (depends on 19) The non-transitory memory of claim 19, wherein selecting the subset based on the spatial criterion comprises: selecting planar surfaces with a surface area that is greater than a threshold area; and filtering out planar surfaces with a surface area that is less than the threshold area. ### Claim 21 (depends on 19) The non-transitory memory of claim 19, wherein selecting the subset based on the spatial criterion comprises: selecting planar surfaces that intersect with a line of sight of a user of the device; and filtering out planar surfaces that do not intersect with the line of sight of the user of the device. ### Claim 22 (depends on 19) The non-transitory memory of claim 19, wherein selecting the subset based on the spatial criterion comprises: selecting planar surfaces that are located more than a threshold distance from a user of the device; and filtering out planar surfaces that are located less than the threshold distance from the user of the device. ### Claim 23 (depends on 19) The non-transitory memory of claim 19, wherein overlaying the portion of the CGR content onto the subset of the plurality of planar surfaces comprises: overlaying, onto a first planar surface in the subset, a first video, a first image or a first text string that is associated with the object; and overlaying, onto a second planar surface in the subset, a second video, a second image or a second text string that is associated with the object. ### Claim 24 (depends on 19) The non-transitory memory of claim 19, wherein overlaying the portion of the CGR content onto the subset of the plurality of planar surfaces comprises: overlaying, onto a first planar surface in the subset, a video that is associated with the object; overlaying, onto a second planar surface in the subset, an image that is associated with the object; and overlaying, onto a third planar surface in the subset, a text string that is associated with the object.