# US 2021/0342646 A1 · Systems, methods, and apparatuses for training a deep model to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342646 A1
- Document ID: US20210342646A1
- Status: Published application
- Kind: A1
- Application number: 17/240,271
- Filed: 2021-04-26
- Published: 2021-11-04
- Earliest claimed: 2020-04-30
- Assignees: Arizona Board of Regents on behalf of Arizona State University
- Applicants: Arizona Board of Regents on Behalf of Arizona State University
- Inventors: Ruibin Feng; Zongwei Zhou; Jianming Liang
- Web page: https://patentgrep.com/patent/US20210342646A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 15/08: Volume rendering
- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- G06V 10/82: using neural networks
- G16H 30/40: for processing medical images, e.g. editing
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30016: Brain
- G06T 2207/30056: Liver; Hepatic
- G06T 2207/30064: Lung nodule
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V 2201/031: of internal organs
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Described herein are means for training a deep model to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework, in which the trained deep models are then utilized for the processing of medical imaging. For instance, an exemplary system is specifically configured for performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input, performing a resize operation of the cropped 3D cubes, performing an image reconstruction operation of the resized and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function. Other related embodiments are disclosed.

## Claims

### Claim 1

A system comprising:
a memory to store instructions;
a processor to execute the instructions stored in the memory;
wherein the system is specially configured to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework, by performing the following operations:
performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input;
performing a resize operation of the cropped 3D cubes;
performing an image reconstruction operation of the resited and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and
generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function.

### Claim 2 (depends on 1)

The system of claim 1, wherein randomly cropping the 3D cube comprises cropping the 3D cube utilizing random scales and random aspect ratios.

### Claim 3 (depends on 2)

The system of claim 2, wherein the random scales and random aspect ratios utilized for the random cropping erase low-level cues across different parts but maintain informative structures and textures amongst the randomly cropped 3D cubes.

### Claim 4 (depends on 1)

The system of claim 1, wherein resizing the cropped 3D cubes comprises resizing the cropped. 3D cubes to produce transformed part for later reconstruction.

### Claim 5 (depends on 1)

The system of claim 1, wherein the reconstruction is to predict the resized whole image from a local patch by training an encoder-decoder pair to minimize the loss function between the transformed part produced via the random cropping and resizing and the original whole image.

### Claim 6 (depends on 1)

The system of claim 1, wherein the encoder learns contrastive representations that embed the part-whole semantics.

### Claim 7 (depends on 1)

The system of claim 1, wherein all skip connections connecting the encoder and decoder are removed from a U-Net architecture.

### Claim 8 (depends on 7)

The system of claim 7, wherein the skip connections remain absent during training so as to force the bottleneck representations encoding high-level information.

### Claim 9 (depends on 1)

The system of claim 1, wherein a part size via which to resize the cropped 3D cube is configurable to avoid training an auto encoder without learning semantics when the part size is too large and to avoid an unsolvable task when the part size is too small so as to lack sufficient information.

### Claim 10

Non-transitory computer-readable storage media having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, the instructions cause the system to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework, by performing operations including:
performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input;
performing a resize operation of the cropped 3D cubes;
performing an image reconstruction operation of the resized and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and
generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function.

### Claim 11 (depends on 10)

The non-transitory computer readable storage media of claim 10:
wherein randomly cropping the 3D cube comprises cropping the 3D cube utilizing random scales and random aspect ratios: and
wherein the random scales and random aspect ratios utilized for the random cropping erase low-level cues across different parts but maintain informative structures and textures amongst the randomly cropped 3D cubes.

### Claim 12 (depends on 10)

The non-transitory computer readable storage media of claim 10:
wherein resizing the cropped 3D cubes comprises resizing the cropped 3D cubes to produce transformed part for later reconstruction.

### Claim 13 (depends on 10)

The non-transitory computer readable storage media of claim 10:
wherein the reconstruction is to predict the resited whole image from a local patch by training an encoder-decoder pair to minimize the loss function between the transformed part produced via the random cropping and resizing and the original whole image.

### Claim 14 (depends on 10)

The non-transitory computer readable storage media of claim 10:
wherein the encoder learns contrastive representations that embed the part-whole semantics.

### Claim 15 (depends on 10)

The non-transitory computer readable storage media of claim 10:
wherein all skip connections connecting the encoder and decoder are removed from a U-Net architecture; and
wherein the skip connections remain absent during training so as to force the bottleneck representations encoding high-level information.

### Claim 16 (depends on 10)

The non-transitory computer readable storage media of claim 10:
wherein a part size via which to resize the cropped 3D cube is configurable to avoid training an auto encoder without learning semantics when the part size is too large and to avoid an unsolvable task when the part size is too small so as to lack sufficient information.

### Claim 17

A method performed by a system having at least a processor and a memory therein to execute instructions for learning contrastive representations embedded within part-whole semantics via a self-supervised learning framework, wherein the method comprises:
performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input;
performing a resize operation of the cropped 3D cubes;
performing an image reconstruction operation of the resited and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and
generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function.

### Claim 18 (depends on 17)

The method of claim 17:
wherein randomly cropping the 3D cube comprises cropping the 3D cube utilizing random scales and random aspect ratios; and
wherein the random scales and random aspect ratios utilized for the random cropping erase low-level cues across different parts but maintain informative structures and textures amongst the randomly cropped 3D cubes.

### Claim 19 (depends on 17)

The method of claim 17:
wherein the reconstruction is to predict the resited whole image from a local patch by training an encoder-decoder pair to minimize the loss function between the transformed part produced via the random cropping and resizing and the original whole image.

### Claim 20 (depends on 17)

The method of claim 17:
wherein all skip connections connecting the encoder and decoder are removed from a U-Net architecture; and
wherein the skip connections remain absent during training so as to force the bottleneck representations encoding high-level information.
