# US 2021/0240978 A1 · Systems and methods for processing images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240978 A1
- Document ID: US20210240978A1
- Status: Published application
- Kind: A1
- Application number: 16/778,324
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Element Ai Inc.
- Applicants: Element Ai Inc.
- Inventors: Patrick STEEVES; Ying ZHANG
- Web page: https://patentgrep.com/patent/US20210240978A1

## CPC

- G06V 10/243: {by compensating for image skew or non-uniform image deformations}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/337: {involving reference images or patches}
- G06T 7/73: using feature-based methods
- G06V 10/82: using neural networks
- G06V 30/19173: {Classification techniques}
- G06V 30/40: Document-oriented image-based pattern recognition
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/418: Document matching, e.g. of document images
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30176: Document
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for identifying landmarks of a document from a digital representation of the document. The method comprises accessing the digital representation of the document and operating a Machine Learning Algorithm (MLA), the MLA having been trained based on a set of training digital representations of documents associated with labels. The operating the MLA comprises down-sampling the digital representation of the document, detecting landmarks, generating fractional pixel coordinates for the detected landmarks. The method further determines the pixel coordinates of the landmarks by upscaling the fractional pixel coordinates from the second resolution to the first resolution and outputs the pixel coordinates of the landmarks.

## Claims

### Claim 1

A computer-implemented method of identifying landmarks of a document from a digital representation of the document, the method comprising:
accessing the digital representation of the document, the digital representation being associated with a first resolution;
operating a Machine Learning Algorithm (MLA), the MLA having been trained:
based on a set of training digital representations of documents associated with labels, the labels identifying landmarks of the documents represented by the training digital representations;
to learn a first function allowing detection of landmarks of documents represented by digital representations;
to learn a second function allowing generation of fractional pixel coordinates for the landmarks detected by the first function;
the operating the MLA comprising:
down-sampling the digital representation of the document, the down-sampled digital representation of the document being associated with a second resolution, the second resolution being lower than the first resolution;
detecting landmarks from the down sampled digital representation of the document;
generating fractional pixel coordinates for the detected landmarks in accordance with the second resolution, the fractional pixel coordinates allowing reconstructing pixel coordinates in accordance with the first resolution;
determining the pixel coordinates of the landmarks by upscaling the fractional pixel coordinates from the second resolution to the first resolution; and
outputting the pixel coordinates of the landmarks.

### Claim 2 (depends on 1)

The method of claim 1, wherein the MLA comprises a Convolutional Neural Network (CNN) comprising multiple layers, the multiple layers comprising a first layer implementing the learning of the first function and a second layer implementing the learning of the second function.

### Claim 3 (depends on 1)

The method of claim 1, wherein the labels identifying landmarks comprise coordinates.

### Claim 4 (depends on 1)

The method of claim 1, wherein the fractional pixel coordinates comprise floating values.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first function implements a classification task, the classification task predicting whether a sub-portion of the digital representation of the document comprises a landmark.

### Claim 6 (depends on 5)

The method of claim 5, wherein the second function implements a regression task, the regression task generating fractional pixel coordinates from the sub-portion of the digital representation identified as comprising a landmark.

### Claim 7 (depends on 1)

The method of claim 1, wherein the landmarks comprise one of corners or edges.

### Claim 8

A computer-implemented method of aligning a first image with a second image, the method comprising:
accessing the first image;
accessing the second image comprising known landmarks;
determining pixel coordinates of landmarks of the first image;
determining a transformation based on the determined pixel coordinates of the landmarks of the first image and known landmarks of the second image, the transformation allowing mapping of the first image onto the second image;
calculating a covariance of pixel values of the first image aligned and superimposed to the second image; and
determining, based on the covariance of the pixel values, whether the first image is to be associated with the second image.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first image is associated with a first resolution and wherein determining the pixel coordinates of landmarks of the first image comprises:
operating a Machine Learning Algorithm (MLA), the MLA having been trained:
based on a set of training images associated with labels, the labels identifying landmarks of the training images;
to learn a first function allowing detection of landmarks of images;
to learn a second function allowing generation of fractional pixel coordinates for the landmarks detected by the first function;
the operating the MLA comprising:
down-sampling the first image, the down-sampled first image being associated with a second resolution, the second resolution being lower than the first resolution;
detecting landmarks from the down-sampled first image;
generating fractional pixel coordinates for the detected landmarks in accordance with the second resolution, the fractional pixel coordinates allowing reconstructing pixel coordinates in accordance with the first resolution;
determining the pixel coordinates of the landmarks by upscaling the fractional pixel coordinates from the second resolution to the first resolution; and
outputting the pixel coordinates of the landmarks.

### Claim 10

A system for identifying landmarks of a document from a digital representation of the document, the system comprising:
at least one processor, and
memory storing a plurality of executable instructions which, when executed by the at least one processor, cause the system to:
access the digital representation of the document, the digital representation being associated with a first resolution;
operate a Machine Learning Algorithm (MLA), the MLA having been trained:
based on a set of training digital representations of documents associated with labels, the labels identifying landmarks of the documents represented by the training digital representations;
to learn a first function allowing detection of landmarks of documents represented by digital representations;
to learn a second function allowing generation of fractional pixel coordinates for the landmarks detected by the first function;
the operating the MLA comprising:
down-sampling the digital representation of the document, the down-sampled digital representation of the document being associated with a second resolution, the second resolution being lower than the first resolution;
detecting landmarks from the down sampled digital representation of the document;
generating fractional pixel coordinates for the detected landmarks in accordance with the second resolution, the fractional pixel coordinates allowing reconstructing pixel coordinates in accordance with the first resolution;
determine the pixel coordinates of the landmarks by upscaling the fractional pixel coordinates from the second resolution to the first resolution; and
output the pixel coordinates of the landmarks.

### Claim 11 (depends on 10)

The system of claim 10, wherein the MLA comprises a Convolutional Neural Network (CNN) comprising multiple layers, the multiple layers comprising a first layer implementing the learning of the first function and a second layer implementing the learning of the second function.

### Claim 12 (depends on 10)

The system of claim 10, wherein the labels identifying landmarks comprise coordinates.

### Claim 13 (depends on 10)

The system of claim 10, wherein the fractional pixel coordinates comprise floating values.

### Claim 14 (depends on 10)

The system of claim 10, wherein the first function implements a classification task, the classification task predicting whether a sub-portion of the digital representation of the document comprises a landmark.

### Claim 15 (depends on 14)

The system of claim 14, wherein the second function implements a regression task, the regression task generating fractional pixel coordinates from the sub-portion of the digital representation identified as comprising a landmark.

### Claim 16 (depends on 10)

The system of claim 10, wherein the landmarks comprise one of corners or edges.