# US 10,755,379 B2 · Multi-point annotation using a haptic plane

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,755,379 B2
- Document ID: US10755379B2
- Status: Granted
- Kind: B2
- Application number: 16/101,417
- Filed: 2018-08-11
- Published: 2019-01-03
- Granted: 2020-08-25
- Earliest claimed: 2014-05-28
- Assignees: EchoPixel, Inc.
- Applicants: EchoPixel, Inc.
- Inventors: Sergio Aguirre-Valencia; William Johnsen; Anthony Chen
- Web page: https://patentgrep.com/patent/US10755379B2

## CPC

- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/60: for the operation of medical equipment or devices
- G06T 2207/30004: Biomedical image processing
- G06T 2210/41: Medical
- G06T 2219/004: Annotating, labelling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

During an analysis technique, a three-dimensional (3D) image of a portion of an individual is iteratively transformed to facilitate accurate determination of detailed multi-point annotation of an anatomical structure. In particular, for a given marker point, in response to receiving information specifying a two-dimensional (2D) plane having an angular position in the 3D image, the 3D image is translated and rotated from an initial position and orientation so that the 2D plane is presented in an orientation parallel to a reference 2D plane of a display. Then, after annotation information specifying the detailed annotation in the 2D plane of the given marker point is received, the 3D image is translated and rotated back to the initial position and orientation. These operations may be repeated for one or more other marker points.

## Claims

### Claim 1

A method for facilitating detailed multi-point annotation of an anatomical structure, the method comprising:
by a computer:
providing, on a display, a first three-dimensional (3D) image of a portion of an individual, wherein the first 3D image has an initial position and an initial orientation;
receiving information specifying a first two-dimensional (2D) plane in the first 3D image, wherein the first 2D plane has an arbitrary angular position relative to the initial orientation;
translating and rotating the first 3D image so that the first 2D plane is presented on a reference 2D plane of the display with an orientation parallel to the reference 2D plane, wherein a normal of the reference 2D plane is perpendicular to a plane of the display, and wherein, prior to the translating and the rotating, the angular position of the first 2D plane is different from an angular position of the reference 2D plane and is different from a predefined orientation of slices in the first 3D image;
receiving annotation information specifying detailed annotation in the first 2D plane of a first marker point;
after the annotation of the first marker point is complete, providing, on the display, a second 3D image of the portion of the individual, wherein the second 3D image is generated by translating image data along a normal direction to the first 2D plane by a predefined distance,
receiving annotation information specifying detailed annotation in a second 2D plane of a second marker point; and
after the annotation of the second marker point is complete, translating and rotating the second 3D image back to the initial position and the initial orientation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the method comprises:
iteratively performing a set of operations for a group of marker points, wherein, for a given marker point, the set of operations comprises:
providing, on the display, a given 3D image of the portion of the individual, wherein the given 3D image is generated by translating the image data along the normal direction to the first 2D plane by a given predefined distance;
receiving given annotation information specifying detailed annotation in a given 2D plane of the given marker point; and
after the annotation of the given marker point is complete, translating and rotating the given 3D image back to the initial position and the initial orientation.

### Claim 3 (depends on 1)

The method of claim 1, wherein the detailed annotation specifies at least one of: a size of the anatomical structure, an orientation of the anatomical structure, a direction of the anatomical structure or a location of the anatomical structure.

### Claim 4 (depends on 1)

The method of claim 1, wherein the annotation information specifying the detailed annotation is associated with an interaction tool.

### Claim 5 (depends on 1)

The method of claim 1, wherein the annotation information specifying the detailed annotation corresponds to haptic interaction with the display.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first 3D image, the second 3D image or the given 3D image comprises a stereoscopic image.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first 2D plane, the second 2D plane or the given 2D plane are positioned at a zero parallax position so that 3D information in the first 2D plane, the second 2D plane or the given 2D plane is perceived as 2D information.

### Claim 8 (depends on 1)

The method of claim 1, wherein at least a pair of marker points describe one of: a linear distance, or a 3D vector.

### Claim 9 (depends on 1)

The method of claim 1, wherein at least three marker points describe one of: a plane, or an angle between two intersecting lines.

### Claim 10 (depends on 1)

The method of claim 1, wherein at least some marker points describe one of: a poly-line, an open contour, a closed contour, or a closed surface.

### Claim 11

A non-transitory computer-readable storage medium for use in conjunction with a computer, the computer-readable storage medium storing program instructions, wherein, when executed by the computer, the program instructions cause the computer to facilitate detailed multi-point annotation of an anatomical structure by performing one or more operations comprising:
providing, on a display, a first three-dimensional (3D) image of a portion of an individual, wherein the first 3D image has an initial position and an initial orientation;
receiving information specifying a first two-dimensional (2D) plane in the first 3D image, wherein the first 2D plane has an arbitrary angular position relative to the initial orientation;
translating and rotating the first 3D image so that the first 2D plane is presented on a reference 2D plane of the display with an orientation parallel to the reference 2D plane, wherein a normal of the reference 2D plane is perpendicular to a plane of the display, and wherein, prior to the translating and the rotating, the angular position of the first 2D plane is different from an angular position of the reference 2D plane and is different from a predefined orientation of slices in the first 3D image;
receiving annotation information specifying detailed annotation in the first 2D plane of a first marker point;
after the annotation of the first marker point is complete, providing, on the display, a second 3D image of the portion of the individual, wherein the second 3D image is generated by translating image data along a normal direction to the first 2D plane by a predefined distance;
receiving annotation information specifying detailed annotation in a second 2D plane of a second marker point; and
after the annotation of the second marker point is complete, translating and rotating the second 3D image back to the initial position and the initial orientation.

### Claim 12 (depends on 11)

The computer-readable storage medium of claim 11, wherein the one or more operations comprise:
iteratively performing a set of operations for a group of marker points, wherein, for a given marker point, the set of operations comprises:
providing, on the display, a given 3D image of the portion of the individual, wherein the given 31) image is generated by translating the image data along the normal direction to the first 2D plane by a given predefined distance;
receiving given annotation information specifying detailed annotation in a given 2D plane of the given marker point; and
after the annotation of the given marker point is complete, translating and rotating the given 3D image back to the initial position and the initial orientation.

### Claim 13 (depends on 11)

The computer-readable storage medium of claim 11, wherein the detailed annotation specifies at least one of: a size of the anatomical structure, an orientation of the anatomical structure, a direction of the anatomical structure or a location of the anatomical structure.

### Claim 14 (depends on 11)

The computer-readable storage medium of claim 11, wherein the information specifying the detailed annotation is associated with an interaction tool.

### Claim 15 (depends on 11)

The computer-readable storage medium of claim 11, wherein the information specifying the detailed annotation corresponds to haptic interaction with the display.

### Claim 16 (depends on 11)

The computer-readable storage medium of claim 11, wherein the first 2D plane, the second 2D plane or the given 2D plane are positioned at a zero parallax position so that 3D information in the first 2D plane, the second 2D plane or the given 2D plane is perceived as 2D information.

### Claim 17 (depends on 11)

The computer-readable storage medium of claim 11, wherein at least a pair of marker points describe one of: a linear distance, or a 3D vector.

### Claim 18 (depends on 11)

The computer-readable storage medium of claim 11, wherein at least three marker points describe one of: a plane, or an angle between two intersecting lines.

### Claim 19 (depends on 11)

The computer-readable storage medium of claim 11, wherein at least some marker points describe one of: a poly-line, an open contour, a closed contour, or a closed surface.

### Claim 20

A computer, comprising:
a processor;
memory storing program instructions, wherein, when executed by the processor, the program instructions cause the computer to facilitate detailed multi-point annotation of an anatomical structure by performing one or more operations comprising:
providing, on a display, a first three-dimensional (3D) image of a portion of an individual, wherein the first 3D image has an initial position and an initial orientation;
receiving information specifying a first two-dimensional (2D) plane in the first 3D image, wherein the first 2D plane has an arbitrary angular position relative to the initial orientation;
translating and rotating the first 3D image so that the first 2D plane is presented on a reference 2D plane of the display with an orientation parallel to the reference 2D plane, wherein a normal of the reference 2D plane is perpendicular to a plane of the display, and wherein, prior to the translating and the rotating, the angular position of the first 2D plane is different from an angular position of the reference 2D plane and is different from a predefined orientation of slices in the first 3D image;
receiving annotation information specifying detailed annotation in the first 2D plane of a first marker point;
after the annotation of the first marker point is complete, providing, on the display, a second 3D image of the portion of the individual, wherein the second 3D image is generated by translating image data along a normal direction to the first 2D plane by a predefined distance;
receiving annotation information specifying detailed annotation in a second 2D plane of a second marker point; and
after the annotation of the second marker point is complete, translating and rotating the second 3D image back to the initial position and the initial orientation.
