# US 12,002,165 B1 · Light probe placement for displaying objects in 3D environments on electronic devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,165 B1 - Document ID: US12002165B1 - Status: Granted - Kind: B1 - Application number: 17/412,507 - Filed: 2021-08-26 - Granted: 2024-06-04 - Earliest claimed: 2020-09-23 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Daniel Kurz; Gowri Somanath; Tobias Holl - Web page: https://patentgrep.com/patent/US12002165B1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 15/506: {Illumination models} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2215/16: Using real world measurements to influence rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Various implementations disclosed herein include devices, systems, and methods that use light probes to facilitate the display of virtual objects in 3D environments. A light probe provides lighting information that describes light incident on a point in space in a 3D environment. For example, a light probe may describe such incident light using an environment map. Such lighting information can be used to provide realistic appearances for objects placed at or near light probe locations in the 3D environment. Implementations disclosed herein determine the light probe locations in real-time or other 3D environments that are generated based on a live physical environment. A digital representation of the live physical environment is used to determine where to position the light probes, how many light probes to use, and/or various light probe attributes. ## Claims ### Claim 1 A method comprising: at a processor: determining characteristics of a physical environment based on sensor data obtained in the physical environment, wherein the characteristics of the physical environment comprise a semantic understanding or an expected context; generating a digital representation representing the characteristics of the physical environment; based on characteristics of the digital representation, determining three-dimensional (3D) positions in a 3D environment for a set of virtual light probes, the set of virtual light probes defining light incident on the 3D positions in the 3D environment based on virtual cameras at the 3D positions; and providing a view of the 3D environment based on the 3D positions determined for the set of virtual light probes. ### Claim 2 (depends on 1) The method of claim 1, wherein the 3D positions are determined based on a rule that accounts for a geometry, semantics, or a lighting condition of the physical environment. ### Claim 3 (depends on 1) The method of claim 1, wherein the 3D positions are determined using a machine learning model. ### Claim 4 (depends on 1) The method of claim 1, wherein the 3D positions are determined based on an optimization that optimizes a number of virtual light probes or probe-to-surface distances. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the 3D positions comprises determining a number of virtual light probes to include in the set of virtual light probes based on the digital representation. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the 3D positions is based on an existing immutable light probe. ### Claim 7 (depends on 1) The method of claim 1 further comprising determining, based on the digital representation, 3D bounding box sizes or shapes, resolutions, bit depths, or lifespans of the set of virtual light probes. ### Claim 8 (depends on 1) The method of claim 1, wherein the digital representation is based on a camera image of the physical environment. ### Claim 9 (depends on 1) The method of claim 1, wherein the digital representation comprises: geometric primitives of the physical environment; a volumetric representation of the physical environment; semantics labels of objects of the physical environment; representations of light sources of the physical environment; or a two-dimensional (2D) floorplan of the physical environment. ### Claim 10 (depends on 1) The method of claim 1, wherein the 3D positions are determined based on expected usage of the 3D environment. ### Claim 11 (depends on 1) The method of claim 1, wherein providing the view of the 3D environment comprises: positioning a virtual object in the 3D environment; and lighting the virtual object based on a position of the virtual object relative to one or more of the set of virtual light probes. ### Claim 12 (depends on 1) The method of claim 1, wherein providing the view of the 3D environment comprises simulating parallax based on the 3D positions of the set of virtual light probes. ### Claim 13 (depends on 1) The method of claim 1, wherein the 3D positions of the set of virtual light probes are determined based on an application type associated with the 3D environment. ### Claim 14 (depends on 1) The method of claim 1, wherein the 3D positions of the set of virtual light probes are determined based on an interaction mode between the physical environment and the 3D environment. ### Claim 15 (depends on 1) The method of claim 1, wherein the 3D positions of the set of virtual light probes are determined based on pose, shape, or material of one or more virtual objects expected to utilize the set of virtual light probes. ### Claim 16 (depends on 1) The method of claim 1, further comprising, based on additional information about the physical environment, updating the 3D positions for the set of virtual light probes. ### Claim 17 (depends on 16) The method of claim 16, wherein the updating is triggered: at a regular cadence; based on user motion; based on user distance from one or more of the set of virtual light probes; based on a change in the physical environment; or based on a change of the digital representation. ### Claim 18 (depends on 16) The method of claim 16, wherein the 3D positions of less than all the set of virtual light probes are updated. ### Claim 19 (depends on 16) The method of claim 16, wherein the updating is based on previous 3D positions of the set of virtual light probes. ### Claim 20 (depends on 1) The method of claim 1, wherein the characteristics of the physical environment are geometric attributes. ### Claim 21 (depends on 1) The method of claim 1, wherein the characteristics of the physical environment comprise the semantic understanding. ### Claim 22 (depends on 1) The method of claim 1, wherein the characteristics of the physical environment comprise the expected context. ### Claim 23 A system comprising: a device with one or more processors; and a memory including instructions that, when executed by the one or more processors, cause the system to: determine characteristics of a physical environment based on sensor data obtained in the physical environment, wherein the characteristics of the physical environment comprise geometric attributes corresponding to fixed portions or empty spaces of the physical environment; generate a digital representation representing the characteristics of the physical environment; based on characteristics of the digital representation, determine three-dimensional (3D) positions in a 3D environment for a set of virtual light probes, the set of virtual light probes defining light incident on the 3D positions in the 3D environment based on virtual cameras at the 3D positions; and provide a view of the 3D environment based on the 3D positions determined for the set of virtual light probes. ### Claim 24 A non-transitory computer-readable medium storing program instructions executable via one or more processors to perform operations comprising: determining characteristics of a physical environment based on sensor data obtained in the physical environment, wherein the characteristics of the physical environment comprise a semantic understanding or an expected context; generating a digital representation representing the characteristics of the physical environment; based on characteristics of the digital representation, determining three-dimensional (3D) positions in a 3D environment for a set of virtual light probes, the set of virtual light probes defining light incident on the 3D positions in the 3D environment based on virtual cameras at the 3D positions; and providing a view of the 3D environment based on the 3D positions determined for the set of virtual light probes.