# US 11,442,591 B2 · System, method, computer readable medium, and viewer-interface for prioritized selection of mutually occluding objects in a virtual environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,442,591 B2
- Document ID: US11442591B2
- Status: Granted
- Kind: B2
- Application number: 15/948,421
- Filed: 2018-04-09
- Published: 2019-10-10
- Granted: 2022-09-13
- Earliest claimed: 2018-04-09
- Assignees: Lockheed Martin Corporation
- Applicants: Lockheed Martin Corporation
- Inventors: Jonathan E. Vathauer; Brandon P. Genesi
- Web page: https://patentgrep.com/patent/US11442591B2

## CPC

- G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method comprising determining, by the processor, a set of objects of the plurality of objects having the shared coordinates (XS, YS) and at a location along the depth direction (ZS); and prioritizing, by the processor, an object from the set of objects based on at least two of metadata of the set of objects, screen areas of the set of objects, transparency of the set of objects, and opaqueness of at least one object of the set of objects currently displayed to improve the selection of at least one of mutually occluded objects and mutually partially occluded objects in the virtual environment. The method includes associating the prioritized object with the viewer input device for detecting interactions with the prioritized object displayed on the display device by the viewer input device. The prioritized object is updated on the screen of the display device based on the interactions.

## Claims

### Claim 1

A computer-implemented method comprising:
detecting, by a processor having a graphical processing unit and coupled to a display device, a point location selected via a viewer input device within a screen of the display device displaying a virtual environment having a plurality of three dimensional (3D) objects, wherein some 3D objects of the plurality of 3D objects when displayed on the display device are mutually occluded 3D objects or partially occluded 3D objects;
determining, by the processor, coordinates (XS, YS) of the point location on the screen; and a depth direction (ZS) defining a path through the point location along a direction of an eye gaze of a viewer;
determining, by the processor, a set of 3D objects of the plurality of 3D objects sharing coordinates (XS, YS) on the screen and located along the depth direction (ZS), determining the set of 3D objects includes determining a depth value for each 3D object of the set of 3D objects;
prioritizing, by the processor, at least two 3D objects from the set of 3D objects based on screen areas of the set of 3D objects, transparency of the set of 3D objects, depth values of the set of 3D objects, and opaqueness of at least one 3D object of the set of 3D objects currently displayed to improve selection of at least one of mutually occluded 3D objects and mutually partially occluded 3D objects in the virtual environment, the screen areas of the set of 3D objects determined using actual dimensions of the set of 3D objects in the virtual environment;
selecting, by the processor, a first 3D object from the at least two 3D objects based on a number of previous actions performed on the first 3D object via the viewer input device being greater than a number of actions performed on a second 3D object of the at least two 3D objects when the first 3D object was included in one or more previous sets of 3D objects while sharing previous coordinates on the screen and located along previous depth directions; and
establishing, by the processor, the selected first 3D object with the viewer input device for detecting interactions with the selected first 3D object displayed on the display device by the viewer input device, the selected first 3D object being updated on the screen of the display device based on the interactions.

### Claim 2 (depends on 1)

The method of claim 1, wherein prioritizing the at least two 3D objects further comprises determining, by the processor, whether the at least two 3D objects of the set of 3D objects have screen areas equal to, within a tolerance of, a smallest screen area of the set of 3D objects.

### Claim 3 (depends on 2)

The method of claim 2, wherein selecting the first 3D object includes retrieving, by the processor, respective numbers of previous actions for each 3D object of the at least two 3D objects.

### Claim 4 (depends on 1)

The method of claim 1, wherein the set of 3D objects are transparent; and
wherein prioritizing, by the processor, the at least two 3D objects from the set of 3D objects includes setting, by the processor, the at least two 3D objects as any transparent 3D object with a smallest screen area of other transparent 3D objects in the set of 3D objects.

### Claim 5 (depends on 1)

The method of claim 1, wherein the detecting of the point location comprises detecting, by the processor, at least one of a touchpoint of a finger of the viewer on the screen of the display device, a touchpoint on the screen from a pen, a location of a cursor on the screen, a click location of the cursor displayed overlaid on the virtual environment and frame rendering associated with the virtual environment.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
prioritizing, by the processor, the at least two 3D objects from the set of 3D objects includes:
determining, by the processor, a first opaque object of the set of 3D objects;
developing, by the processor, a candidate 3D objects list of 3D objects of the set of 3D objects which are located along the depth direction (ZS) and have depth values smaller than or equal to a depth value of the first opaque object of the set of 3D objects;
determining, by the processor, the screen area of said 3D objects in the candidate 3D objects list; and
setting the at least two 3D objects as any 3D object with a smallest screen area of other 3D objects in the candidate 3D objects list.

### Claim 7

A tangible and non-transitory computer readable medium having program instructions which, when executed by a processor having a graphical processing unit, causes the processor with the graphical processing unit to:
detect a point location selected via a viewer input device within a screen of the a display device displaying a virtual environment having a plurality of three dimensional (3D) objects, wherein some 3D objects of the plurality of 3D objects when displayed on the display device are mutually occluded 3D objects or partially occluded 3D objects;
determine coordinates (XS, YS) of the point location on the screen and a depth direction (ZS) defining a path through the point location along a direction of an eye gaze of a viewer;
determine a set of 3D objects of the plurality of 3D objects sharing the coordinates (XS, YS) on the screen and located along the depth direction (ZS), determining the set of 3D objects includes determining a depth value for each 3D object of the set of 3D objects;
prioritize at least two 3D objects from the set of 3D objects based on screen areas of the set of 3D objects, transparency of the set of 3D objects, depth values of the set of 3D objects, and opaqueness of at least one 3D object of the set of 3D objects currently displayed to improve selection of at least one of mutually occluded 3D objects and mutually partially occluded 3D objects in the virtual environment, the screen areas of the set of 3D objects determined using actual dimensions of the set of 3D objects in the virtual environment;
select a first 3D object from the at least two 3D objects based on a number of previous actions performed on the first 3D object via the viewer input device being greater than a number of actions performed on a second 3D object of the at least two 3D objects when the first 3D object was included in one or more previous sets of 3D objects while sharing previous coordinates on the screen and located along previous depth directions; and
establish the selected first 3D object with the viewer input device for detecting interactions with the selected first 3D object displayed on the display device by the viewer input device, the selected first 3D object being updated on the screen of the display device based on the interactions.

### Claim 8 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the program instructions which prioritize the at least two 3D objects further comprise program instructions which, when executed, cause the processor with the graphical processing unit to determine whether the at least two 3D objects of the set of 3D objects have screen areas equal to, or within a tolerance of, a smallest screen area of the set of 3D objects.

### Claim 9 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein the program instructions to select the first 3D object further comprise program instructions which when executed cause the processor with the graphical processing unit to retrieve respective numbers of previous actions for each 3D object of the at least two 3D objects.

### Claim 10 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the set of 3D objects are transparent; and
wherein the program instructions to prioritize the at least two 3D objects from the set of 3D objects further comprise program instructions which, when executed, cause the processor with the graphical processing unit to set the at least two 3D objects as any transparent 3D object with a smallest screen area of all other transparent 3D objects in the set of 3D objects.

### Claim 11 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the program instructions to detect the point location further comprise program instructions which, when executed, cause the processor with the graphical processing unit to detect at least one of a touchpoint of a finger of the viewer on the screen of the display device, a touchpoint on the screen from a pen, a location of a cursor on the screen, a click location of the cursor displayed overlaid on the virtual environment and frame rendering associated with the virtual environment.

### Claim 12 (depends on 7)

The non-transitory computer readable medium of claim 7, wherein the program instructions to prioritize the at least two 3D objects from the set of 3D objects further comprise program instructions which, when executed, cause the processor with the graphical processing unit to:
determine a first opaque object of the set of 3D objects;
develop a candidate 3D objects list of 3D objects of the set of 3D objects which are located along the depth direction (ZS) and have depth values smaller than or equal to a depth value of the first opaque object of the set of 3D objects;
determine the screen area of said 3D objects in the candidate 3D objects list; and
set the at least two 3D objects as any 3D object with a smallest screen area of other 3D objects in the candidate 3D objects list.

### Claim 13

A system comprising:
a display device configured to display a virtual environment having a plurality of three dimensional (3D) objects, wherein some 3D objects of the plurality of 3D objects when displayed on the display device are mutually occluded 3D objects or partially occluded 3D objects; and
a processor having a graphical processing unit and configured to:
cause display of the plurality of 3D objects of the virtual environment on a screen of the display device;
detect a point location selected via a viewer input device within the screen of the display device displaying the virtual environment;
determine coordinates (XS, YS) of the point location on the screen and a depth direction (ZS) defining a path through the point location along a direction of an eye gaze of a viewer;
determine a set of 3D objects of the plurality of 3D objects sharing the coordinates (XS, YS) and located along the depth direction (ZS), determining the set of 3D objects includes determining a depth value for each 3D object of the set of 3D objects;
prioritize at least two 3D objects from the set of 3D objects based on screen areas of the set of 3D objects, transparency of the set of 3D objects, depth values of the set of 3D objects, and opaqueness of at least one 3D object of the set of 3D objects currently displayed to improve selection of at least one of mutually occluded 3D objects and mutually partially occluded 3D objects in the virtual environment, the screen areas of the set of 3D objects determined using actual dimensions of the set of 3D objects in the virtual environment;
select a first 3D object from the at least two 3D objects based on a number of previous actions performed on the first 3D object via the viewer input device being greater than a number of actions performed on a second 3D object of the at least 3D objects when the first 3D object was included in one or more previous sets of 3D objects while sharing previous coordinates on the screen and located along previous depth directions; and
establish the selected first 3D object with the viewer input device for detecting interactions with the selected first 3D object displayed on the display device by the viewer input device, the selected first 3D object being updated on the screen of the display device based on the interactions.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processor with the graphical processing unit is configured to determine whether the at least two 3D objects of the set of 3D objects have screen areas equal to, or within a tolerance of, a smallest screen area of the set of 3D objects.

### Claim 15 (depends on 14)

The system of claim 14, wherein the processor having the graphical processing unit is configured to retrieve respective numbers of interactions for each 3D object of the at least two 3D objects.

### Claim 16 (depends on 13)

The system of claim 13, wherein the set of 3D objects are transparent; and
wherein the processor with the graphical processing unit is configured to set the at least two 3D objects as any transparent 3D object with a smallest screen area of other transparent 3D objects in the set of 3D objects.

### Claim 17 (depends on 13)

The system of claim 13, wherein the processor with the graphical processing unit is configured to detect at least one of a touchpoint of a finger of the viewer on the screen of the display device, a touchpoint on the screen from a pen, a location of a cursor on the screen, a click location of the cursor displayed overlaid on the virtual environment and frame rendering associated with the virtual environment.

### Claim 18 (depends on 13)

The system of claim 13, wherein the processor with the graphical processing unit is configured to:
determine a first opaque 3D object of the set of 3D objects;
develop a candidate 3D objects list of 3D objects of the set of 3D objects which are located along the depth direction (ZS) and have depth values smaller than or equal to a depth value of the first opaque 3D object;
determine the screen area of said 3D objects in the candidate 3D objects list; and
set the at least two 3D objects as any 3D object with a smallest screen area of other 3D objects in the candidate 3D objects list.