# US 11,307,653 B1 · User input and interface design in augmented reality for use in surgical settings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,653 B1
- Document ID: US11307653B1
- Status: Granted
- Kind: B1
- Application number: 17/194,191
- Filed: 2021-03-05
- Granted: 2022-04-19
- Earliest claimed: 2021-03-05
- Assignees: Medivis, Inc.
- Applicants: MediVis, Inc.
- Inventors: Long Qian; Wenbo Lan; Christopher Morley
- Web page: https://patentgrep.com/patent/US11307653B1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2210/41: Medical
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to an Interaction Engine. According to various embodiments, the Interaction Engine generates within a unified three-dimensional (3D) coordinate space: (i) a virtual 3D model container; (ii) a virtual 3D medical model positioned according to a model pose within the virtual 3D model container; and (iii) a virtual 3D representation of at least a portion of at least one of a user's hands. The Interaction Engine renders an Augmented Reality (AR) display that includes concurrent display of the virtual 3D medical model container, the virtual 3D medical model and the virtual 3D representation of the user's hands. The Interaction Engine detects one or more physical gestures associated with the user and one or more types of virtual interacts associated with a detected physical gesture(s).

## Claims

### Claim 1

A computer-implemented method, comprising:
generating, within a unified three-dimensional (3D) coordinate space:
(i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and
(ii) at least one a virtual 3D hand representation;
rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
detecting a first physical gesture;
modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation;
modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model;
detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device;
determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose; and
identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and
modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical mode positioned according to the model pose and a virtual 3D hand representation further comprises:
rendering a virtual 3D model container; and
rendering the virtual 3D medical model positioned according to the model pose within a display of the rendered virtual 3D model container.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection comprises:
identifying an overlap between one or more coordinates within the virtual 3D model container and one or more portions of the virtual 3D clipping plane bounded within the first intersection;
identifying one or more portions of non-rendered 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising:
prior to modifying the AR display by rendering the virtual clipping plane at the second display position:
updating the virtual 3D medical model according to the identified one or more portions of 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
detecting a third physical gesture based on a subsequent second change of pose data of the AR headset device, the subsequent second change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a third display position of the virtual clipping plane based on the subsequent change of pose data of the AR headset device;
determining a second intersection between the virtual 3D clipping plane at the third display position and the rendered updated virtual 3D medical model at the current model pose.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
identifying one or more portions of non-rendered 3D medical model data for the updated virtual 3D medical model in the current model pose that map to the second intersection; and
modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the third display position concurrently with rendering of a second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising:
wherein rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection comprises: rendering the virtual 3D medical model within the virtual 3D container; and
wherein rendering the second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection comprises: rendering the second updated virtual 3D medical model within the virtual 3D container.

### Claim 8

A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
generating, within a unified three-dimensional (3D) coordinate space:
(i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and
(ii) at least one a virtual 3D hand representation;
rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
detecting a first physical gesture;
modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation;
modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model;
detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device;
determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose; and
identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and
modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.

### Claim 9 (depends on 8)

The system of claim 8, rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical mode positioned according to the model pose and a virtual 3D hand representation further comprises:
rendering a virtual 3D model container; and
rendering the virtual 3D medical model positioned according to the model pose within a display of the rendered virtual 3D model container.

### Claim 10 (depends on 9)

The system of claim 9, wherein identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection comprises:
identifying an overlap between one or more coordinates within the virtual 3D model container and one or more portions of the virtual 3D clipping plane bounded within the first intersection;
identifying one or more portions of non-rendered 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
prior to modifying the AR display by rendering the virtual clipping plane at the second display position:
updating the virtual 3D medical model according to the identified one or more portions of 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container.

### Claim 12 (depends on 11)

The system of claim 11, further comprising:
detecting a third physical gesture based on a subsequent second change of pose data of the AR headset device, the subsequent second change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a third display position of the virtual clipping plane based on the subsequent change of pose data of the AR headset device;
determining a second intersection between the virtual 3D clipping plane at the third display position and the rendered updated virtual 3D medical model at the current model pose.

### Claim 13 (depends on 12)

The system of claim 12, further comprising:
identifying one or more portions of non-rendered 3D medical model data for the updated virtual 3D medical model in the current model pose that map to the second intersection; and
modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the third display position concurrently with rendering of a second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection.

### Claim 14 (depends on 13)

The system of claim 13, further comprising:
wherein rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection comprises: rendering the virtual 3D medical model within the virtual 3D container; and
wherein rendering the second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection comprises: rendering the second updated virtual 3D medical model within the virtual 3D container.

### Claim 15

A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
generating, within a unified three-dimensional (3D) coordinate space:
(i) a virtual 3D medical model positioned according to a current model pose, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and
(ii) at least one a virtual 3D hand representation;
rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
detecting a first physical gesture;
modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
identifying selection of a clipping plane virtual interaction based on the type of movement of the virtual 3D hand representation;
modifying the AR display, via the AR headset device, by rendering a virtual clipping plane in the AR display concurrently displayed, at a first display position, with the virtual 3D medical model;
detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a second display position of the virtual clipping plane based on the first change of pose data of the AR headset device;
determining a first intersection between the virtual 3D clipping plane at the second display position and the rendered virtual 3D medical model at the current model pose; and
identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and
modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.

### Claim 16 (depends on 15)

The computer program product of claim 15, rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D medical mode positioned according to the model pose and a virtual 3D hand representation further comprises:
rendering a virtual 3D model container; and
rendering the virtual 3D medical model positioned according to the model pose within a display of the rendered virtual 3D model container.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection comprises:
identifying an overlap between one or more coordinates within the virtual 3D model container and one or more portions of the virtual 3D clipping plane bounded within the first intersection;
identifying one or more portions of non-rendered 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container.

### Claim 18 (depends on 17)

The computer program product of claim 17, further comprising:
prior to modifying the AR display by rendering the virtual clipping plane at the second display position:
updating the virtual 3D medical model according to the identified one or more portions of 3D medical model data that map to the identified one or more coordinates within the virtual 3D model container.

### Claim 19 (depends on 18)

The computer program product of claim 18, further comprising:
detecting a third physical gesture based on a subsequent second change of pose data of the AR headset device, the subsequent second change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a third display position of the virtual clipping plane based on the subsequent change of pose data of the AR headset device;
determining a second intersection between the virtual 3D clipping plane at the third display position and the rendered updated virtual 3D medical model at the current model pose;
identifying one or more portions of non-rendered 3D medical model data for the updated virtual 3D medical model in the current model pose that map to the second intersection; and
modifying the AR display, via the AR headset device, by rendering the virtual clipping plane at the third display position concurrently with rendering of a second updated virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the second intersection.

### Claim 20

A computer-implemented method, comprising:
generating, within a unified three-dimensional (3D) coordinate space:
(i) a virtual 3D container;
(ii) a virtual 3D medical model positioned according to a current model pose within the virtual 3D container, the current model pose representing a position and orientation of the virtual 3D medical model in the unified 3D coordinate space; and
(iii) at least one a virtual 3D hand representation;
rendering, via an Augmented Reality (AR) headset device, an AR display that includes display of the virtual 3D container and virtual 3D medical model positioned according to the current model pose and the virtual 3D hand representation;
detecting a first physical gesture;
modifying the AR display, via the AR headset device, by rendering display of a type of movement of the virtual 3D hand representation based on the detected first physical gesture;
identifying selection of a cropping virtual interaction based on the type of movement of the virtual 3D hand representation;
modifying the AR display, via the AR headset device, by rendering a selectable shape in the AR display concurrently displayed on a first surface of the virtual 3D container, the first surface displayed at a first display position;
detecting a second physical gesture based on a first change of pose data of the AR headset device, the first change of pose data representing a change of a position and orientation of the AR headset device in the unified 3D coordinate space;
determining a movement of the selectable shape on the first surface of the virtual 3D container based on the first change of pose data of the AR headset device;
determining a second display position of the first surface of the virtual 3D container resulting from the movement of the selectable shape;
determining a first intersection between the first surface of the virtual 3D container at the second display position and the rendered virtual 3D medical model at the current model pose; and
identifying one or more portions of non-rendered 3D medical model data for the virtual 3D medical model in the current model pose that map to the first intersection; and
modifying the AR display, via the AR headset device, by rendering the first surface of the virtual 3D container at the second display position concurrently with rendering the virtual 3D medical model portraying the identified one or more portions of 3D medical model data that map to the first intersection.