# US 2021/0233330 A1 · Virtual or Augmented Reality Aided 3D Visualization and Marking System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233330 A1
- Document ID: US20210233330A1
- Status: Published application
- Kind: A1
- Application number: 17/250,342
- Filed: 2019-07-08
- Published: 2021-07-29
- Earliest claimed: 2018-07-09
- Assignees: Ottawa Hospital Research Institute
- Applicants: Ottawa Hospital Research Institute
- Inventors: Justin SUTHERLAND; Daniel LA RUSSA
- Web page: https://patentgrep.com/patent/US20210233330A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- G06F 3/03545: {Pens or stylus}
- G06F 3/0418: {for error correction or compensation, e.g. based on parallax, calibration or alignment}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/63: for local operation
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- G06T 2210/41: Medical
- G06T 2219/004: Annotating, labelling
- G06T 2219/008: Cut plane or projection plane definition
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

There is provided a system and method for applying markings to a three-dimensional virtual image or virtual object, the system comprising a physical stylus, a surface, and a virtual or augmented reality display. A virtual space is displayed by the virtual or augmented reality display. The virtual space includes a three-dimensional target object; at least one plane, including a tracking plane, the tracking plane corresponding to the surface; and a virtual stylus in a virtual reality view, or the physical stylus in an augmented reality view. A position of the virtual stylus or the physical stylus relative to the tracking plane is correlated to an actual position of the physical stylus relative to the surface; and a cross-section of the target object is displayed on the tracking plane where the tracking plane intersects the target object.

## Claims

### Claim 1

A system for applying markings to a three-dimensional virtual image or virtual object using a surface, the system comprising:
an object capable of being used as a physical stylus; and
a virtual or augmented reality display;
wherein a virtual space is displayed by the virtual or augmented reality display, the virtual space comprising:
a three-dimensional target object;
at least one plane, including a tracking plane, the tracking plane corresponding to the surface; and
a virtual stylus in a virtual reality view, or the physical stylus in an augmented reality view; wherein:
a position of the virtual stylus or the physical stylus relative to the tracking plane is correlated to an actual position of the physical stylus relative to the surface; and
a cross-section of the target object is displayed on the tracking plane where the tracking plane intersects the target object.

### Claim 2 (depends on 1)

The system of claim 1, wherein the system is configured to determine the actual position of the physical stylus relative to the surface using a tracking system associated with the surface and the physical stylus.

### Claim 3 (depends on 2)

The system of claim 2, wherein the tracking system comprises one or more trackers provided on the surface and the physical stylus.

### Claim 4 (depends on 2)

The system of claim 2, wherein the tracking system comprises one or more calibrated positions in the virtual space.

### Claim 5 (depends on 1)

The system of claim 1, wherein the physical stylus is provided with a stylus tip sensor.

### Claim 6

(canceled)

### Claim 7 (depends on 1)

The system of claim 1, wherein the system further comprises a controller.

### Claim 8

(canceled)

### Claim 9 (depends on 7)

The system of claim 7, wherein the virtual space further comprises a virtual controller, and a movement of the virtual controller is associated with a movement of the controller.

### Claim 10 (depends on 9)

The system of claim 9, wherein the target object is moved by moving the virtual controller onto the target object, pressing a first controller button of the controller, and moving the virtual controller while holding the first controller button.

### Claim 11

(canceled)

### Claim 12

(canceled)

### Claim 13 (depends on 1)

The system of claim 1, wherein the system is configured to:
detect contact between the physical stylus and the surface;
delineate a region of interest using a circular or spherical fill region; and
save the region of interest to the target object upon detecting a break on the contact between the physical stylus and the surface.

### Claim 14 (depends on 1)

The system of claim 1, wherein the system is configured to:
detect contact between the physical stylus and the surface;
mark a region of interest along a path of the virtual stylus while detecting contact between the physical stylus and the surface; and
save the region of interest to the target object upon detecting a break in the contact between the physical stylus and the surface.

### Claim at position 15

-18. (canceled)

### Claim 19

A method of applying markings to a three-dimensional virtual image or virtual object, the method comprising:
displaying a virtual space using a virtual or augmented reality display;
providing in the virtual space: a three-dimensional target object, at least one plane including a tracking plane, the tracking plane corresponding to the surface, and a virtual stylus in a virtual reality view, or an object used as a physical stylus in an augmented reality view;
correlating a position of the virtual stylus or the physical stylus relative to the tracking plane to an actual position of the physical stylus relative to a surface; and
displaying a cross-section of the target object on the tracking plane where the tracking plane intersects the target object.

### Claim 20 (depends on 19)

The method of claim 19, further comprising determining the actual position of the physical stylus relative to the surface using a tracking system associated with the surface and the physical stylus.

### Claim 21 (depends on 20)

The method of claim 20, wherein the tracking system comprises one or more trackers provided on the surface and the physical stylus.

### Claim 22 (depends on 20)

The method of claim 20, wherein the tracking system comprises one or more calibrated positions in the virtual space.

### Claim 23 (depends on 19)

The method of claim 19, wherein the physical stylus is provided with a stylus tip sensor.

### Claim 24

(canceled)

### Claim 25 (depends on 19)

The method of claim 19, wherein the method utilizes a controller.

### Claim 26

(canceled)

### Claim 27 (depends on 25)

The method of claim 25, wherein the virtual space further comprises a virtual controller, and a movement of the virtual controller is associated with a movement of the controller.

### Claim 28 (depends on 27)

The method of claim 27, further comprising moving the target object by moving the virtual controller onto the target object, detecting pressing a first controller button of the controller, and moving the virtual controller while the first controller button is being held.

### Claim 29

(canceled)

### Claim 30

(canceled)

### Claim 31 (depends on 19)

The method of claim 19, further comprising:
detecting contact between the physical stylus and the surface;
delineating a region of interest using a circular or spherical fill region; and
saving the region of interest to the target object upon detecting a break on the contact between the physical stylus and the surface.

### Claim 32 (depends on 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30)

The method of any one of claims 19 to 30, further comprising:
detecting contact between the physical stylus and the surface;
marking a region of interest along a path of the virtual stylus while detecting contact between the physical stylus and the surface; and
saving the region of interest to the target object upon detecting a break in the contact between the physical stylus and the surface.

### Claim at position 30

-36. (canceled)

### Claim 37

A non-transitory computer readable medium comprising computer executable instructions for:
displaying a virtual space using a virtual or augmented reality display;
providing in the virtual space: a three-dimensional target object, at least one plane including a tracking plane, the tracking plane corresponding to the surface, and a virtual stylus in a virtual reality view, or an object used as a physical stylus in an augmented reality view;
correlating a position of the virtual stylus or the physical stylus relative to the tracking plane to an actual position of the physical stylus relative to a surface; and
displaying a cross-section of the target object on the tracking plane where the tracking plane intersects the target object.
