# US 12,579,668 B2 · Viewer systems and related methods with multi-piece registration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,579,668 B2
- Document ID: US12579668B2
- Status: Granted
- Kind: B2
- Application number: 18/632,791
- Filed: 2024-04-11
- Published: 2024-12-19
- Granted: 2026-03-17
- Earliest claimed: 2023-06-14
- Assignees: Sectra AB
- Applicants: Sectra AB
- Inventors: Jesper Molin
- Web page: https://patentgrep.com/patent/US12579668B2

## CPC

- G06F 16/54: Browsing; Visualisation therefor
- G06T 7/33: using feature-based methods
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 16/583: using metadata automatically derived from the content
- G06T 7/0014: {using an image reference approach}
- G06T 7/38: Registration of image sequences
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G06F 2203/04803: Split screen, i.e. subdividing the display area or the window area into separate subareas
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10056: Microscopic image
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30168: Image quality inspection
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Viewer systems and related methods with user interactive user interfaces that provide a viewport with first and second digital images that allow a user to navigate an active one of the digital images which drives a viewing window over the other digital image to provide a view that moves in response to navigation of the active slide to allow a user to compare and select one of several candidate registrations to register the first and second slides for pathology or other medical review. The user can interact with the user interface to switch which image is the active images (e.g., a WSI or slide image), select an object in the digital image and direct the system to provide other digital images.

## Claims

### Claim 1

A method of registering medical images for viewing, comprising:
electronically providing a database of candidate registrations of a plurality of different medical images for respective patients, wherein a pair of the different medical images of a single patient of the plurality of different medical images has multiple candidate registrations in the database of candidate registrations, and wherein multiple image regions of a first medical image of the pair of the different medical images are each related to an image region in a second medical image of the pair of different medical images through at least one candidate registration in the database of candidate registrations;
electronically providing a viewer comprising a first window with a first viewing window user interface and a second window with a second viewing window user interface;
displaying a viewport of the first medical image in the first window;
displaying a viewport of the second medical image in the second window;
accepting user input of a user to select one of the first window or the second window to be an active window for user navigation and which makes a non-selected other of the first window or the second window a non-active, follower window;
obtaining candidate registrations from the database of candidate registrations that are related to the first medical image or the second medical image in the active window; and
electronically automatically selecting a registration from the obtained candidate registrations based on proximity of the image region of the obtained candidate registrations to the viewport of the active window,
wherein each image region is an area that is defined by a respective keypoint cluster, and
wherein the electronically automatically selecting a registration from the obtained candidate registrations is based on a candidate registration quality measure.

### Claim 2 (depends on 1)

The method of claim 1, wherein each candidate registration in the database of candidate registrations has a respective candidate registration quality measure.

### Claim 3 (depends on 1)

The method of claim 1, wherein the candidate registrations in the database of candidate registrations are computed through keypoint pairing of pairs of first and second keypoint clusters, the first keypoint cluster in the first medical image and the second keypoint cluster in the second medical image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the candidate registration quality measure is calculated based, at least in part, by a number of keypoints in a respective keypoint cluster, with keypoint clusters having a greatest number correlated to a greater registration quality measure.

### Claim 5 (depends on 1)

The method of claim 1, wherein the selecting comprises determining viewport proximity of the first or second viewport that is the active window to a midpoint of keypoint clusters connected to the candidate registrations.

### Claim 6 (depends on 1)

The method of claim 1, further comprising automatically electronically moving the viewport of the follower window according to the selected registration whereby the viewport of the first medical image and the viewport of the second medical display respective, corresponding image portions of the first and second medical images.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
changing the viewport of the active window as directed by the user in a user interface of the active viewing window to navigate about image regions in the active window; then,
in response to change of the viewport of the active window, automatically electronically moving the viewport of the follower window, synchronized to follow movement of the viewport of the active window.

### Claim 8 (depends on 1)

The method of claim 1, further comprising accepting user input to change the viewport of the active window and automatically electronically repeating the selecting registration step and automatically moving the viewport of the follower window according to the change in the viewport of the active window.

### Claim 9 (depends on 8)

The method of claim 8, further comprising changing one or more selected registrations from the obtained candidate registrations whereby different objects in the image region of the active window becomes aligned with a common (the same) region in the image region of the follower window.

### Claim 10 (depends on 1)

The method of claim 1, wherein the medical images are digital pathology slide images of a tissue block of the single patient.

### Claim 11 (depends on 1)

The method of claim 1, wherein keypoints of spaced apart objects in different image regions are used to identify similar objects in the different medical images to identify candidate registrations for the obtaining step.

### Claim 12 (depends on 1)

The method of claim 1, further comprising electronically determining if there are candidate registrations that have an image region overlapping with the viewport in the active window and accepting user input via a single user interface action, whereby each single user interface action executes selection of another candidate registration in the order of decreasing candidate registration quality measure.

### Claim 13 (depends on 12)

The method of claim 12, wherein a sequence of selecting candidate registrations in response to respective single user interface actions is ordered according to spatial location of the image region of the non-active window with the image region of the candidate registrations.

### Claim 14 (depends on 1)

The method of claim 1, wherein the selected registration is carried out, at least in part, by selecting a plurality of different candidate registrations from the obtained candidate registrations corresponding to a current position of the active image viewport, wherein each of the selected plurality of candidate registrations corresponds to a different relevant image region in the active image.

### Claim 15 (depends on 14)

The method of claim 14, wherein each of the selected plurality of candidate registrations has a weighting used for the candidate registration quality measure, and wherein the weighting is based on a distance from a center of mass of an active image keypoint cluster in a respective selected candidate registration to centerpoint of the active image viewport.

### Claim 16

A viewer system for evaluating Whole Slide Images (WSI), comprising:
a display; and
a circuit in communication with the display, the circuit comprising at least one processor that:
provides or in communication with a database of candidate registrations of a plurality of different medical images for respective patients, wherein a pair of the different medical images of a single patient of the plurality of different medical images has multiple candidate registrations in the database of candidate registrations, wherein each region of multiple regions of a first medical image of the pair of the different medical images is related to a region in a second medical image of the pair of different medical images through at least one candidate registration in the database of candidate registrations;
provides a viewer comprising a first window and a second window;
displays a viewport of the first medical image in the first window;
displays a viewport of the second medical image in the second window;
accepts user input of a user to select one of the first window or the second window to be an active window for user navigation and which makes a non-selected other of the first window or the second window a non-active, follower window;
obtains candidate registrations from the database of candidate registrations that are related to the first medical image or the second medical image in the active window; and
selects a registration from the obtained candidate registrations based on proximity of the image region of the obtained candidate registration to the viewport of the active window,
wherein each image region is an area that is defined by a respective keypoint cluster, and
wherein the obtained candidate registrations are based on candidate registration quality measures.

### Claim 17 (depends on 16)

The viewer system of claim 16, wherein an image region is associated with an area spanned by keypoint clusters.

### Claim 18 (depends on 16)

The viewer system of claim 16, and wherein each candidate registration in the database of candidate registrations has a respective candidate registration quality measure.

### Claim 19 (depends on 18)

The viewer system of claim 18, wherein the at least one processor is configured to:
determine if there are candidate registrations that have an image region overlapping with the viewport in the active window; and
accept user input via a single user interface action, where each action executes selection of another candidate registration in the order of decreasing candidate registration quality measure.

### Claim 20 (depends on 16)

The viewer system of claim 16, wherein the at least one processor is configured to automatically move the viewport of the follower window according to the selected registration whereby the viewport of the first medical image and the viewport of the second medical display respective, corresponding image portions of the first and second medical images.

### Claim 21 (depends on 16)

The viewer system of claim 16, wherein the at least one processor is further configured to:
change the viewport of the active window as directed by the user in a user interface of the active viewing window to navigate about image regions in the active window; then,
in response to change of the viewport of the active window, automatically move the viewport of the follower window, synchronized to follow movement of the viewport of the active window.

### Claim 22 (depends on 16)

The viewer system of claim 16, wherein the circuit is configured to accept user input to change the viewport of the active window whereby the at least one processor then automatically selects another candidate registration as a current selected registration and moves the viewport of the follower window according to the change in the viewport.

### Claim 23 (depends on 16)

The viewer system of claim 16, wherein the at least one processor is configured to provide a sequence of selecting candidate registrations in response to user action that is ordered according to spatial location of the image region of the non-active window with the image region of the candidate registrations.

### Claim 24 (depends on 16)

The viewer system of claim 16, wherein the selected registration is carried out, at least in part, by selecting a plurality of candidate registrations corresponding to a position of the active image viewport, each selected plurality of candidate registrations corresponds to a different relevant image region in the active image.