# US 2021/0233206 A1 · Line-based image registration and cross-image annotation devices, systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233206 A1
- Document ID: US20210233206A1
- Status: Published application
- Kind: A1
- Application number: 17/192,967
- Filed: 2021-03-05
- Published: 2021-07-29
- Earliest claimed: 2013-10-01
- Assignees: Ventana Medical Systems, Inc.
- Applicants: Ventana Medical Systems, Inc.
- Inventors: Srinivas Chukka; Anindya Sarkar; Quan Yuan
- Web page: https://patentgrep.com/patent/US20210233206A1

## CPC

- G06T 3/14: Transformations for image registration, e.g. adjusting or mapping for alignment of images
- G06T 7/33: using feature-based methods
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G06T 2207/10024: Color image
- G06T 2207/10056: Microscopic image
- G06T 2207/10064: Fluorescence image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The disclosure relates to devices, systems and methods for image registration and annotation. The devices include computer software products for aligning whole slide digital images on a common grid and transferring annotations from one aligned image to another aligned image on the basis of matching tissue structure. The systems include computer-implemented systems such as work stations and networked computers for accomplishing the tissue-structure based image registration and cross-image annotation. The methods include processes for aligning digital images corresponding to adjacent tissue sections on a common grid based on tissue structure, and transferring annotations from one of the adjacent tissue images to another of the adjacent tissue images. The basis for alignment may be a line-based registration process, wherein sets of lines are computed on the boundary regions computed for the two images, where the boundary regions are obtained using information from two domains—soft-weighted foreground images and gradient magnitude images. The binary mask image, based on whose boundary the line features are computed, may be generated by combining two binary masks—a first binary mask is obtained on thresholding a soft-weighted (continuous valued) foreground image, which is computed based on the stain content in an image, while a second binary mask is obtained after thresholding a gradient magnitude domain image, where the gradient is computed from the grayscale image obtained from the color image.

## Claims

### Claim 1

A method comprising:
generating a first set of line segments corresponding to at least part of a first tissue section depicted in a first image;
generating a second set of line segments corresponding to at least part of a second tissue section depicted in a second image, wherein the first tissue section includes one or more mismatching regions relative to corresponding regions of the second tissue section;
identifying a first subset of the first set of line segments, wherein the first subset includes line segments that are determined to be capable of being matched to corresponding line segments of a second subset of the second set of line segments;
determining that a count of line segments corresponding to the first subset exceeds a predetermined threshold; and
in response to determining that the count of line segments corresponding to the first subset exceed the predetermined threshold, generating, based on the first subset of line segments, one or more transformation parameters for performing a first alignment operation to align the first image to the second image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first alignment operation includes applying the one or more transformation parameters to align the first image to a common grid generated from the second image.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
performing, based on the one or more transformation parameters, the first alignment operation to align the first image to the second image; and
in response to performing the first alignment operation, transferring an annotation at a particular location of the first image to a corresponding location of the second image.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the one or more transformation parameters includes:
for each line segment of the first subset:
determining a rotation sub-parameter and a shift sub-parameter between the line segment of the first subset and a corresponding line segment of the second subset; and
generating a line-matching cost of the line segment based on the rotation sub-parameter and the shift sub-parameter;
selecting a line segment of the first subset having a minimum line-matching cost; and
generating, based on the selected line segment, a transformation condition for generating the one or more transformation parameters.

### Claim 5 (depends on 1)

The method of claim 1, wherein a line segment of the first subset is represented by a line-center value, a line-angle value, and a gradient-strength value, wherein the gradient-strength value is generated based on a sum of gradient magnitude across the line segment.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more mismatching regions include an inserted sub-region, a deleted sub-region, and/or a flipped sub-region.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
performing, based on the one or more transformation parameters, the first alignment operation to align the first image to the second image, thereby generating an aligned first image;
identifying a first window region of the aligned first image and a second window region of the second image;
determining a normalized correlation score between the first window region and the second window region; and
performing a second alignment operation by applying the normalized correlation score to the aligned first image.

### Claim 8

A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising:
generating a first set of line segments corresponding to at least part of a first tissue section depicted in a first image;
generating a second set of line segments corresponding to at least part of a second tissue section depicted in a second image, wherein the first tissue section includes one or more mismatching regions relative to corresponding regions of the second tissue section;
identifying a first subset of the first set of line segments, wherein the first subset includes line segments that are determined to be capable of being matched to corresponding line segments of a second subset of the second set of line segments;
determining that a count of line segments corresponding to the first subset exceeds a predetermined threshold; and
in response to determining that the count of line segments corresponding to the first subset exceed the predetermined threshold, generating, based on the first subset of line segments, one or more transformation parameters for performing a first alignment operation to align the first image to the second image.

### Claim 9 (depends on 8)

The system of claim 8, wherein the first alignment operation includes applying the one or more transformation parameters to align the first image to a common grid generated from the second image.

### Claim 10 (depends on 8)

The system of claim 8, wherein the instructions further cause the one or more data processors to perform operations comprising:
performing, based on the one or more transformation parameters, the first alignment operation to align the first image to the second image; and
in response to performing the first alignment operation, transferring an annotation at a particular location of the first image to a corresponding location of the second image.

### Claim 11 (depends on 8)

The system of claim 8, wherein generating the one or more transformation parameters includes:
for each line segment of the first subset:
determining a rotation sub-parameter and a shift sub-parameter between the line segment of the first subset and a corresponding line segment of the second subset; and
generating a line-matching cost of the line segment based on the rotation sub-parameter and the shift sub-parameter;
selecting a line segment of the first subset having a minimum line-matching cost; and
generating, based on the selected line segment, a transformation condition for generating the one or more transformation parameters.

### Claim 12 (depends on 8)

The system of claim 8, wherein a line segment of the first subset is represented by a line-center value, a line-angle value, and a gradient-strength value, wherein the gradient-strength value is generated based on a sum of gradient magnitude across the line segment.

### Claim 13 (depends on 8)

The system of claim 8, wherein the one or more mismatching regions include an inserted sub-region, a deleted sub-region, and/or a flipped sub-region.

### Claim 14 (depends on 8)

The system of claim 8, wherein the instructions further cause the one or more data processors to perform operations comprising:
performing, based on the one or more transformation parameters, the first alignment operation to align the first image to the second image, thereby generating an aligned first image;
identifying a first window region of the aligned first image and a second window region of the second image;
determining a normalized correlation score between the first window region and the second window region; and
performing a second alignment operation by applying the normalized correlation score to the aligned first image.

### Claim 15

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform operations comprising:
generating a first set of line segments corresponding to at least part of a first tissue section depicted in a first image;
generating a second set of line segments corresponding to at least part of a second tissue section depicted in a second image, wherein the first tissue section includes one or more mismatching regions relative to corresponding regions of the second tissue section;
identifying a first subset of the first set of line segments, wherein the first subset includes line segments that are determined to be capable of being matched to corresponding line segments of a second subset of the second set of line segments;
determining that a count of line segments corresponding to the first subset exceeds a predetermined threshold; and
in response to determining that the count of line segments corresponding to the first subset exceed the predetermined threshold, generating, based on the first subset of line segments, one or more transformation parameters for performing a first alignment operation to align the first image to the second image.

### Claim 16 (depends on 15)

The computer-program product of claim 15, wherein the first alignment operation includes applying the one or more transformation parameters to align the first image to a common grid generated from the second image.

### Claim 17 (depends on 15)

The computer-program product of claim 15, further comprising:
performing, based on the one or more transformation parameters, the first alignment operation to align the first image to the second image; and
in response to performing the first alignment operation, transferring an annotation at a particular location of the first image to a corresponding location of the second image.

### Claim 18 (depends on 15)

The computer-program product of claim 15, wherein generating the one or more transformation parameters includes:
for each line segment of the first subset:
determining a rotation sub-parameter and a shift sub-parameter between the line segment of the first subset and a corresponding line segment of the second subset; and
generating a line-matching cost of the line segment based on the rotation sub-parameter and the shift sub-parameter;
selecting a line segment of the first subset having a minimum line-matching cost; and
generating, based on the selected line segment, a transformation condition for generating the one or more transformation parameters.

### Claim 19 (depends on 15)

The computer-program product of claim 15, wherein a line segment of the first subset is represented by a line-center value, a line-angle value, and a gradient-strength value, wherein the gradient-strength value is generated based on a sum of gradient magnitude across the line segment.

### Claim 20 (depends on 15)

The computer-program product of claim 15, further comprising:
performing, based on the one or more transformation parameters, the first alignment operation to align the first image to the second image, thereby generating an aligned first image;
identifying a first window region of the aligned first image and a second window region of the second image;
determining a normalized correlation score between the first window region and the second window region; and
performing a second alignment operation by applying the normalized correlation score to the aligned first image.
