# US 12,217,329 B2 · Systems and methods for dynamic detection and presentation of obscured real-world objects in augmented or mixed reality virtual content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,329 B2 - Document ID: US12217329B2 - Status: Granted - Kind: B2 - Application number: 17/497,519 - Filed: 2021-10-08 - Published: 2023-04-13 - Granted: 2025-02-04 - Earliest claimed: 2021-10-08 - Assignees: JPMorgan Chase Bank, N.A. - Applicants: Jpmorgan Chase Bank, N.A. - Inventors: Marco Pistoia; Bill Moriarty; Shaohan Hu; Hargun Kalsi; Aniella Arantes - Web page: https://patentgrep.com/patent/US12217329B2 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G01S 17/89: for mapping or imaging - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for dynamic detection and presentation of obscured real-world objects in augmented or mixed reality virtual content may include an object of interest computer program executed by an electronic device: (1) receiving data for a virtual or augmented reality object to display on an image of a physical environment on a display of the electronic device; (2) receiving data for a physical object in the physical environment; (3) determining that the physical object is obscured by the virtual or augmented reality object; (4) generating a representation of the physical object; and (5) providing the representation of the physical object to a virtual reality or augmented reality computer program executed by the electronic device, wherein the virtual reality or augmented reality computer program is configured to display the physical object over the virtual or augmented reality object. ## Claims ### Claim 1 A method, comprising: receiving, by an object of interest computer program executed by an electronic device, a location for a virtual or augmented reality object to display on an augmented reality view of a physical environment by an augmented reality computer program executed by the electronic device, wherein the augmented reality view comprises a physical object that is fixed in the physical environment; generating, by the object of interest computer program, a graphical representation of the virtual or augmented reality object at the location for the virtual or augmented reality object; detecting, by the object of interest computer program, the physical object from image data or sensor data received from the electronic device; determining, by the object of interest computer program, a location for the physical object in the physical environment from the image data or the sensor data; determining, by the object of interest computer program by using the location for the virtual or augmented reality object and the location for the physical object, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object on the augmented reality view of the physical environment displayed on the electronic device; generating, by the object of interest computer program, a virtual representation of the physical object; and providing, by the object of interest computer program, the virtual representation of the physical object to the augmented reality computer program, wherein the augmented reality computer program is configured to display the virtual representation of the physical object over the graphical representation of the virtual or augmented reality object in the augmented reality view of the physical environment at the location for the physical object. ### Claim 2 (depends on 1) The method of claim 1, wherein the object of interest computer program receives the location for the virtual or augmented reality object from the augmented reality computer program. ### Claim 3 (depends on 1) The method of claim 1, further comprising: determining, by the object of interest computer program, that the physical object is an object of interest. ### Claim 4 (depends on 1) The method of claim 1, wherein the physical object comprises a dangerous, hazardous, or harmful object. ### Claim 5 (depends on 1) The method of claim 1, wherein the step of determining, by the object of interest computer program, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object comprises: determining that the location of the graphical representation of the virtual or augmented reality object is between the electronic device and the physical object. ### Claim 6 (depends on 5) The method of claim 5, wherein the object of interest computer program determines a location of the physical object using a ray cast query. ### Claim 7 (depends on 5) The method of claim 5, wherein the object of interest computer program determines a location of the physical object using LiDAR. ### Claim 8 (depends on 1) The method of claim 1, wherein the virtual representation of the physical object comprises an image of the physical object. ### Claim 9 A method, comprising: receiving, by an object of interest computer program executed by an electronic device, a location for a virtual or augmented reality object to display on an augmented reality view of a physical environment by an augmented reality computer program executed by the electronic device, wherein the augmented reality view comprises a physical object that is fixed in the physical environment; generating, by the object of interest computer program, a graphical representation of the virtual or augmented reality object at the location for the virtual or augmented reality object; detecting, by the object of interest computer program, the physical object from image data or sensor data received from the electronic device; determining, by the object of interest computer program, a location for the physical object in the physical environment from the image data or the sensor data; determining, by the object of interest computer program by using the location for the virtual or augmented reality object and the location for the physical object, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object on the augmented reality view of the physical environment displayed on the electronic device; detecting, by the object of interest computer program, a boundary of the physical object; and providing, by the object of interest computer program, the boundary of the physical object to the augmented reality computer program, wherein the augmented reality computer program is configured to remove an area within the boundary from the graphical representation of the virtual or augmented reality object, wherein the physical object is visible in the area of the graphical representation of the virtual or augmented reality object. ### Claim 10 (depends on 9) The method of claim 9, wherein the object of interest computer program receives the location for the virtual or augmented reality object from the augmented reality computer program. ### Claim 11 (depends on 9) The method of claim 9, further comprising: determining, by the object of interest computer program, that the physical object comprises a dangerous, hazardous, or harmful object. ### Claim 12 (depends on 11) The method of claim 11, wherein the step of determining, by the object of interest computer program, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object comprises: determining that the location of the graphical representation of the virtual or augmented reality object is between the electronic device and the physical object. ### Claim 13 (depends on 12) The method of claim 12, wherein the object of interest computer program determines a location of the physical object using a ray cast query. ### Claim 14 (depends on 12) The method of claim 12, wherein the object of interest computer program determines a location of the physical object using LiDAR. ### Claim 15 An electronic device, comprising: a display; an object of interest computer program; and a computer processor; wherein, when executed by the computer processor, the object of interest computer program causes the computer processor to: receive a location for a virtual or augmented reality object to display on an augmented reality view of a physical environment by an augmented reality computer program executed by the electronic device, wherein the augmented reality view comprises a physical object that is fixed in the physical environment; generate a graphical representation of the virtual or augmented reality object at the location for the virtual or augmented reality object; detecting the physical object from image data or sensor data received from the electronic device; determine a location for the physical object in the physical environment from the image data or the sensor data; determine that the physical object should not be obscured by identifying a type of the physical object in an object of interest database comprising a plurality of types of physical objects; determine by using the location for the virtual or augmented reality object and the location for the physical object that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object on the augmented reality view of the physical environment displayed on the electronic device; generate a virtual representation of the physical object; and provide the virtual representation of the physical object to the augmented reality computer program, wherein the augmented reality computer program displays the virtual representation of the physical object over the graphical representation of the virtual or augmented reality object in the augmented reality view of the physical environment at the location for the physical object. ### Claim 16 (depends on 15) The electronic device of claim 15, wherein the object of interest computer program receives the location for the virtual or augmented reality object from the augmented reality computer program. ### Claim 17 (depends on 15) The electronic device of claim 15, wherein the object of interest database comprises physical objects that are dangerous, hazardous, or harmful. ### Claim 18 (depends on 15) The electronic device of claim 15, wherein the object of interest computer program causes the computer processor to determine that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object by determining a location of the physical object and determining that the location of the virtual or augmented reality object is between the electronic device and the physical object. ### Claim 19 (depends on 18) The electronic device of claim 18, wherein location of the physical object is determined using a ray cast query. ### Claim 20 (depends on 18) The electronic device of claim 18, further comprising a LiDAR transceiver, and wherein the location of the physical object is determined using LiDAR from the LiDAR transceiver.