# US 2022/0036108 A1 · Automatically detecting and isolating objects in images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036108 A1
- Document ID: US20220036108A1
- Status: Published application
- Kind: A1
- Application number: 17/279,727
- Filed: 2019-09-24
- Published: 2022-02-03
- Earliest claimed: 2018-09-25
- Assignees: Element Ai Inc.
- Applicants: Element AI Inc.
- Inventors: Ying Zhang
- Web page: https://patentgrep.com/patent/US20220036108A1

## CPC

- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06F 18/2431: Multiple classes
- G06N 3/02: Neural networks
- G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for automatically detecting and isolating objects in images. An image containing at least one object of interest is segmented by a segmentation module, based on the class of object each pixel of the image depicts. A bounding module then determines coordinates of a predetermined shape that covers at least a portion of the at least one object of interest. An application module then applies a bounding box having those coordinates and having the predetermined shape to the original image. In some embodiments, the coordinates are determined based on a mask layer that is based on the object classes in the image. In other embodiments, the coordinates are determined based on the mask layer and on an edge mask layer. Some embodiments comprise at least one neural network. In some embodiments, the objects of interest are text objects.

## Claims

### Claim 1

A method for isolating at least one object of interest in an image, the method comprising:
(a) receiving said image;
(b) passing said image through a segmentation module to thereby produce a segmented image, wherein contents of said segmented image are classified into at least two object classes, and wherein said at least one object of interest is a member of one of said at least two object classes;
(c) identifying a location of said at least one object of interest within said segmented image;
(d) determining coordinates of a predetermined shape surrounding at least a portion of said at least one object of interest at said location; and
(e) applying a bounding box having said predetermined shape and having said coordinates to said image, to thereby isolate said at least one object of interest from said image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein said at least one object of interest is text.

### Claim 3 (depends on 1)

The method according to claim 1, wherein said shape is a parallelogram.

### Claim 4 (depends on 3)

The method according to claim 3, wherein said shape is a rectangle.

### Claim 5 (depends on 1)

The method according to claim 1, wherein said segmentation module comprises a neural network.

### Claim 6 (depends on 1)

The method according to claim 1, wherein step (b) further comprises the step of generating a mask layer, said mask layer being based on said object classes, and wherein said coordinates are determined in step (d) based on said mask layer.

### Claim 7 (depends on 6)

The method according to claim 6, wherein:
said image is an image array of image pixels;
each individual image pixel in said image array is classified into one of said at least two object classes, said classifying being based on a kind of object depicted by said individual image pixel;
said mask layer is a mask pixel array of mask pixels, such that each mask pixel corresponds to at least one image pixel;
an object value is assigned to said each mask pixel;
said object value for a specific mask pixel is a first value when at least one corresponding image pixel is a member of a predetermined first class; and
said object value for said specific mask pixel is a second value when said at least one corresponding image pixel is a member of a predetermined second class.

### Claim 8 (depends on 6)

The method according to claim 6, wherein step (d) further comprises using a neural network to determine said coordinates.

### Claim 9 (depends on 6)

The method according to claim 6, wherein said coordinates correspond to a largest said predetermined shape that surrounds at least a portion of said object of interest, such that, when said largest said predetermined shape is applied to said mask layer, contents of said largest said predetermined shape meet at least one criterion, wherein said at least one criterion is related to said object classes.

### Claim 10 (depends on 9)

The method according to claim 9, wherein each individual image pixel in said image array is classified into one of said at least two object classes, said classifying being based on a kind of object depicted by said individual image pixel, and wherein step (d) further comprises:
(d.1) applying a test bounding box of random size to said mask layer, wherein said text bounding box has said predetermined shape and wherein said test bounding box surrounds said at least a portion of said object of interest;
(d.2) determining object values for all mask pixels within said test bounding box, wherein:
said object value for a specific mask pixel is a first value when at least one corresponding image pixel is a member of a predetermined one of said at least two classes; and
said object value for said specific mask pixel is a second value when said at least one corresponding image pixel is a member of a predetermined other one of said at least two classes;
(d.3) increasing an area surrounded by said test bounding box to result in a larger bounding box when all of said mask pixels within said test bounding box have a predetermined object value, said predetermined object value being one of said first value and said second value;
(d.4) decreasing an area surrounded by said test bounding box to result in a larger bounding box when at least one of said mask pixels within said test bounding box has a different object value, said different object value being another one of said first value and said second value;
(d.5) repeating steps (d.2)-(d.4) until an exit condition is reached, said exit condition being one of:
at least one predetermined criterion is met; and
a maximum number of iterations is reached; and
(d.6) determining said coordinates based on said test bounding box.

### Claim 11 (depends on 7)

The method according to claim 7, wherein step (b) further comprises the step of generating an edge mask layer, said edge mask layer being based on said object classes, and wherein said coordinates are determined in step (d) based on said mask layer and on said edge mask layer.

### Claim 12 (depends on 11)

The method according to claim 11, wherein said edge mask layer is an edge mask array of edge mask pixels, such that each edge mask pixel corresponds to at least one image pixel, and wherein:
an edge value is assigned to each edge mask pixel;
said edge value for a particular edge mask pixel is derived from an edge probability; and
said edge probability for said particular edge mask pixel is a probability that a corresponding particular image pixel in said image array is on an edge of said at least one object of interest.

### Claim 13 (depends on 12)

The method according to claim 12, wherein step (d) further comprises the steps of:
(d.1) processing said edge mask layer to thereby produce a binary edge mask, said binary edge mask comprising binary pixels, wherein each binary pixel corresponds to an edge mask pixel, and wherein:
a binary pixel value of a specific binary pixel is said first value when, for a corresponding specific edge mask pixel, said edge probability is equal to or above an edge threshold;
said binary pixel value of said specific binary pixel is said second value when, for said corresponding specific edge mask pixel, said edge probability is below an edge threshold;
(d.2) for each specific mask pixel of said mask layer, subtracting a corresponding binary pixel's binary pixel value from an object value of said specific mask pixel, to thereby identify edges of said at least one object of interest;
(d.3) determining said coordinates based on said edges.

### Claim 14 (depends on 11)

The method according to claim 11, wherein step (d) further comprises the steps of:
(d.1) passing said mask layer and said edge mask layer through a feature-extraction module to thereby generate at least one region containing at least a portion of said at least one object of interest; and
(d.2) determining said coordinates based on said at least one region.

### Claim 15 (depends on 13)

The method according to claim 13, wherein said feature-extraction module is a neural network.

### Claim 16 (depends on 15)

The method according to claim 15, wherein said neural network learns a maximum number of iterations to perform based on a predetermined number.

### Claim 17 (depends on 1)

The method according to claim 1, further comprising the step of determining an angle of said at least one object of interest relative to at least one axis of said image, based on said predetermined shape and on said coordinates.

### Claim 18

A system for isolating at least one object of interest in an image, the system comprising:
a segmentation module for receiving said image and for segmenting said image to thereby produce a segmented image, wherein contents of said segmented image are classified into at least two object classes, wherein said at least one object of interest is a member of one of said at least two object classes;
a bounding module for identifying a location of said at least one object of interest based on said segmented image and for determining coordinates of a predetermined shape surrounding at least a portion of said at least one object of interest at said location; and
an application module for applying a bounding box having said predetermined shape and having said coordinates to said image, to thereby isolate said at least one object of interest from said image.

### Claim 19 (depends on 18)

The system according to claim 18, wherein said segmentation module further comprises a mask generation module for generating a mask layer, said mask layer being based on said object classes, and wherein said bounding module determines said coordinates based on said image and on said mask layer.

### Claim 20 (depends on 18)

The system according to claim 18, wherein said segmentation module further comprises an edge mask generation module for generating an edge mask layer, said edge mask layer being based on said object classes, and wherein said bounding module determines said coordinates based on said mask layer and on said edge mask layer.

### Claim 21 (depends on 18)

The system according to claim 18, wherein said segmentation module comprises a neural network.

### Claim 22 (depends on 18)

The system according to claim 18, wherein said bounding module comprises at least one neural network.

### Claim 23 (depends on 18)

The system according to claim 18, wherein said bounding module is further for determining an angle of said at least one object of interest relative to at least one axis of said image, based on said predetermined shape and on said coordinates.

### Claim 24

Non-transitory computer-readable media having encoded thereon computer-readable and computer-executable instructions that, when executed, implement a method for isolating at least one object of interest in an image, the method comprising:
(a) receiving said image;
(b) passing said image through a segmentation module to thereby produce a segmented image, wherein contents of said segmented image are classified into at least two object classes, and wherein said at least one object of interest is a member of one of said at least two object classes;
(c) identifying a location of said at least one object of interest within said segmented image;
(d) determining coordinates of a predetermined shape surround at least a portion of said object of interest at said location; and
(e) applying a bounding box having said predetermined shape and having said coordinates to said image, to thereby isolate said at least one object of interest from said image.
