# US 12,182,952 B2 · Indicating a position of an occluded physical object > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,952 B2 - Document ID: US12182952B2 - Status: Granted - Kind: B2 - Application number: 18/386,073 - Filed: 2023-11-01 - Published: 2024-02-22 - Granted: 2024-12-31 - Earliest claimed: 2021-03-22 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Pavel Veselinov Dudrenov; Edwin Iskandar - Web page: https://patentgrep.com/patent/US12182952B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 1/1694: {the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 7/248: {involving reference images or patches} - G06T 7/74: {involving reference images or patches} - G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures - G06T 15/205: {Image-based rendering} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2210/21: Collision detection, intersection - G06T 2210/62: Semi-transparency - G06T 2215/12: Shadow map, environment map - G06T 2215/16: Using real world measurements to influence rendering - G06T 2219/2004: Aligning objects, relative positioning of parts - G06T 2219/2021: Shape modification - G06T 2219/2024: Style variation - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method is performed at an electronic device with one or more processors, a non-transitory memory, one or more environmental sensors, and a display. The method includes determining a first positional value associated with a physical agent based on environmental data from the one or more environmental sensors. The method includes determining that a portion of computer-generated content satisfies an occlusion criterion with respect to a corresponding portion of the physical agent, based on the first positional value. The method includes, in response to determining that the occlusion criterion is satisfied and determining that the physical agent satisfies a movement criterion or a pose criterion, generating a mesh associated with the physical agent based on the first positional value, and displaying the mesh on the display. ## Claims ### Claim 1 A method comprising: at an electronic device including one or more processors, a non-transitory memory, one or more environmental sensors, and a display: determining a first positional value associated with a physical agent based on environmental data from the one or more environmental sensors; determining that a portion of computer-generated content satisfies an occlusion criterion with respect to a corresponding portion of the physical agent, based on the first positional value; and in response to determining that the occlusion criterion is satisfied and determining that the physical agent satisfies a movement criterion or a pose criterion: generating a mesh associated with the physical agent based on the first positional value; and displaying the mesh on the display. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining a second positional value associated with the physical agent based on the environmental data, wherein satisfaction of the movement criterion is based on a difference between the first and second positional values. ### Claim 3 (depends on 1) The method of claim 1, wherein determining satisfaction of the pose criterion includes determining that the physical agent faces the electronic device. ### Claim 4 (depends on 3) The method of claim 3, wherein determining that the physical agent faces the electronic device includes identifying a face of the physical agent via semantic segmentation. ### Claim 5 (depends on 1) The method of claim 1, wherein the electronic device is associated with a user, and wherein the user is different from the physical agent. ### Claim 6 (depends on 5) The method of claim 5, wherein the electronic device corresponds to a head-mountable device (HMD) that is adapted to be worn by the user. ### Claim 7 (depends on 1) The method of claim 1, wherein the mesh at least partially overlaps with the portion of the computer-generated content. ### Claim 8 (depends on 1) The method of claim 1, wherein displaying the mesh corresponds to increasing a transparency characteristic associated with the portion of the computer-generated content. ### Claim 9 (depends on 1) The method of claim 1, wherein the mesh indicates a shadow that is associated with the physical agent. ### Claim 10 (depends on 1) The method of claim 1, further comprising determining a plurality of positional values associated with the physical agent based on the environmental data, wherein the plurality of positional values includes the first positional value, and wherein generating the mesh is based on the plurality of positional values. ### Claim 11 (depends on 10) The method of claim 10, wherein determining that the portion of computer-generated satisfies the occlusion criterion includes: determining that the portion of the computer-generated content at least partially overlaps with the corresponding portion of the physical agent on the display, based on at least a portion of the plurality of positional values; and determining that the portion of the computer-generated content is associated with a respective depth value that is less than a respective depth value associated with the physical agent, based on at least a portion of the plurality of positional values. ### Claim 12 (depends on 10) The method of claim 10, wherein the one or more environmental sensors include a depth sensor that outputs depth data, wherein the plurality of positional values includes a plurality of depth values based on the depth data, the depth value plurality of depth values characterizing the physical agent, and wherein the mesh corresponds to a volumetric mesh that is based on the plurality of depth values. ### Claim 13 (depends on 1) The method of claim 1, wherein determining that the portion of computer-generated satisfies the occlusion criterion includes determining that the portion of the computer-generated content is associated with an opacity characteristic that exceeds a threshold. ### Claim 14 (depends on 1) The method of claim 1, wherein the electronic device includes an eye tracker that outputs eye tracking data, wherein the eye tracking data indicates a gaze position of a user, and wherein generating the mesh is in further response to determining that the gaze position satisfies a proximity threshold with respect to the physical agent. ### Claim 15 (depends on 1) The method of claim 1, wherein the one or more environmental sensors includes at least two of an image sensor, an ambient light sensor, or a depth sensor. ### Claim 16 A system comprising: one or more environmental sensors; an object tracker to determine a first positional value associated with a physical agent based on environmental data from the one or more environmental sensors; a content identifier to determine that a portion of computer-generated content satisfies an occlusion criterion with respect to a corresponding portion of the physical agent, based on the first positional value; a mesh generator to generate a mesh associated with the physical agent based on the first positional value, in response to determining that the occlusion criterion is satisfied and determining that the physical agent satisfies a movement criterion or a pose criterion; and a display to display the mesh. ### Claim 17 A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or processors, one or more environmental sensors, and a display, cause the electronic device to: determine a first positional value associated with a physical agent based on environmental data from the one or more environmental sensors; determine that a portion of computer-generated content satisfies an occlusion criterion with respect to a corresponding portion of the physical agent, based on the first positional value; and in response to determining that the occlusion criterion is satisfied and determining that the physical agent satisfies a movement criterion or a pose criterion: generate a mesh associated with the physical agent based on the first positional value; and display the mesh on the display. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the instructions further cause the electronic device to determine a second positional value associated with the physical agent based on the environmental data, wherein satisfaction of the movement criterion is based on a difference between the first and second positional values. ### Claim 19 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein determining satisfaction of the pose criterion includes determining that the physical agent faces the electronic device. ### Claim 20 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein the electronic device is associated with a user, and wherein the user is different from the physical agent.