# US 10,872,460 B1 · Device and system for volume visualization and interaction in a virtual reality or augmented reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,460 B1
- Document ID: US10872460B1
- Status: Granted
- Kind: B1
- Application number: 16/839,803
- Filed: 2020-04-03
- Granted: 2020-12-22
- Earliest claimed: 2018-09-21
- Assignees: Immersivetouch, Inc.
- Applicants: Immersive Touch, Inc.
- Inventors: Jia Luo; Jonathan Linsner; P. Pat Banerjee; Carter James Egan; Andrew Durnford
- Web page: https://patentgrep.com/patent/US10872460B1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 15/08: Volume rendering
- G02B 27/017: {Head mounted}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06T 7/0012: {Biomedical image inspection}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2219/2016: Rotation, translation, scaling
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

The present technology relates to devices and systems for volume visualization and interaction in a virtual reality environment.

## Claims

### Claim 1

A workstation for volume visualization and interaction in a virtual reality or augmented reality environment, said workstation comprising:
a volume data import and preprocessing unit adapted to input three-dimensional (3D) volume data from an input data source and pre-process the 3D volume data;
a virtual or augmented reality interface adapted to receive user hand or head position and orientation data from one or more virtual reality or augmented reality devices;
a virtual or augmented reality interaction system coupled to the virtual or augmented reality interface, said virtual or augmented reality interaction system adapted to provide a plurality of interaction features for a user to interact with the 3D volume data and being adapted to output interaction data based on user interactions with the interaction features;
a graphics rendering unit adapted to receive the 3D volume data, user hand or head position and orientation data, and the interaction data, the graphics rendering unit adapted to render one or more selected 3D volumes in accordance with a view-ray-ordered rendering process; and
a virtual scenario management unit adapted to manage the virtual scenario data structure which maintains the original volume data, the preprocessed results, the outcome of graphics rendering and the consequences of user interaction.

### Claim 2 (depends on 1)

The workstation of claim 1, further comprising a haptics rendering unit adapted to interface with one or more haptic devices, the haptics rendering unit adapted to update the position and orientation of virtual tools from the one or more haptic devices to the virtual scenario and to output volume force feedback to the one or more haptic devices.

### Claim 3 (depends on 1)

The workstation of claim 1, wherein the 3D volume data comprises the medical imaging of a plurality of modalities, including Computer Tomography (CT), CT Angiography (CTA), Cone Beam CT (CBCT), Magnetic Resonance Imaging (MRI), and MR Angiography (MRA).

### Claim 4 (depends on 1)

The workstation of claim 1, wherein the plurality of interaction features comprises one or more of spatial tracking, grab manipulation, measurement, snapshots, visual configuration, cutaway and 2D overlay, drawing, volume editing, laser pointer indicator, multi-volume view, multi-user collaboration or undo-redo interaction features.

### Claim 5 (depends on 4)

The workstation of claim 4, wherein the spatial tracking feature is adapted to identify the object in the virtual scenario that the user interacts with and help with the execution of other interaction features.

### Claim 6 (depends on 4)

The workstation of claim 4, wherein the grab manipulation feature is adapted to move, rotate, scale and interact with the rendered 3D volume.

### Claim 7 (depends on 6)

The workstation of claim 6, wherein the grab manipulation feature further comprises a feature adapted to place one or more of the 3D models along with the rendered 3D volume in the virtual scenario, and being adapted to manipulate with the 3D models either along with the volume or as an independent object.

### Claim 8 (depends on 7)

The workstation of claim 7, wherein the 3D models comprise one or more of the models of medical implants or surgical instruments.

### Claim 9 (depends on 4)

The workstation of claim 4, wherein the measurement feature comprises quantifications of the distance between two points, the cumulative length of a polygonal chain, the angle between two lines, the angle between two planes, the circumference of a circle, the volume of a user defined space, or the volume within an isosurface.

### Claim 10 (depends on 4)

The workstation of claim 4, wherein the measurement feature further comprises a surface binding process adapted to optionally attach measurement points onto the surface close by.

### Claim 11 (depends on 4)

The workstation of claim 4, wherein the cutaway and 2D overlay feature is adapted to reveal the internal structure of the 3D volume rendering, and being adapted to overlay a 2D volume rendering with the 3D rendering.

### Claim 12 (depends on 4)

The workstation of claim 4, wherein the volume editing feature comprises a volume cutting mode to partition the surface representation of the volume data by user-specified regions and geometries, modify the original volume data to reflect the modification of shape, and generate additional volume data to represent the newly cut partitions.

### Claim 13 (depends on 12)

The workstation of claim 12, wherein the volume editing feature further comprises a connected component labeling process to detect the separation of the volume data and its surface representation.

### Claim 14 (depends on 4)

The workstation of claim 4, wherein the volume editing feature further comprises a volume sculpting mode to frequently modify small fractions of the volume data and its surface representation at the location specified by the user, to gradually remove materials from the represented anatomy, or add materials to it.

### Claim 15 (depends on 4)

The workstation of claim 4, wherein the snapshot feature is adapted to capture a static picture of the virtual scenario from a user specified perspective at a user defined time.

### Claim 16 (depends on 4)

The workstation of claim 4, wherein the undo-redo feature is adapted to reverse the user interactions and recover the virtual scenario, including the volume data and its surface representations, at the previous state; or reverse the undo operation and recover the virtual scenario at the state prior to the undo operation.

### Claim 17 (depends on 1)

The workstation of claim 1, wherein the preprocessing of the 3D volume data comprises noise reduction, segmentation, coregistration of multiple 3D volume data, re-combination of distinct segmented volumes, volume cropping, and topological smoothing.

### Claim 18 (depends on 1)

The workstation of claim 1, wherein the volume data import and preprocessing unit further comprises a volume-surface conversion process adapted to generate one or more of the surface models from the volume data, and being adapted to construct volume data based on a surface model.

### Claim 19 (depends on 1)

The workstation of claim 1, wherein the graphics rendering unit further comprises a plurality of graphics rendering mechanisms adapted to render one or more of volume data on two-dimensional (2D) planes or in three-dimensional (3D) spaces.

### Claim 20 (depends on 19)

The workstation of claim 19, wherein the plurality of graphics rendering mechanisms comprises Shaded Surface Display (SSD), Volume Rendering Technique (VRT), Multi-Planar Reformation (MPR), Maximum Intensity Projection (MIP), and Minimum Intensity Projection (MinIP).

### Claim 21 (depends on 19)

The workstation of claim 19, wherein one or more of the graphics rendering on 2D planes can be overlaid with the graphics rendering in 3D space.

### Claim 22 (depends on 1)

The workstation of claim 1, wherein the graphics rendering can be constrained within one or more of user-specified regions of interest, and can be culled out by one or more of user-specified area.

### Claim 23 (depends on 1)

The workstation of claim 1, wherein the graphics rendering unit further comprises an order-independent transparency rendering technique and architecture.

### Claim 24 (depends on 1)

The workstation of claim 1, wherein the graphics rendering unit is adapted to stereographically render the virtual scenario on a VR/AR display device, with the rendering perspective consistent with the physical position and orientation of the display device.

### Claim 25 (depends on 24)

The workstation of claim 24, wherein the graphics rendering on an AR display device can be superimposed with the patient or physical object associated with the volume data.

### Claim 26 (depends on 1)

The workstation of claim 1, wherein the graphics rendering unit is further adapted to render the virtual scenario on a traditional 2D display, with the rendering perspective steadily following the headset orientation or otherwise specified by the user.

### Claim 27 (depends on 1)

The workstation of claim 1, wherein the virtual scenario management unit comprises a case exporting and importing process adapted to save the virtual scenario of the active session, including the original, processed or edited volume, its surface representations, as well as the results of user interaction, as external files; and to load and recover the virtual scenario from a previously saved session.

### Claim 28 (depends on 1)

The workstation of claim 1, wherein the virtual scenario management unit further comprises a model exporting process adapted to save the surface representation of the original, processed or edited volume data as polygon meshes.

### Claim 29 (depends on 1)

The workstation of claim 1, wherein one or more of the volume data import and preprocessing unit, virtual or augmented reality interface, virtual or augmented reality interaction system, graphics rendering unit, or virtual scenario management unit are implemented by a graphic processing unit.