# US 2021/0279461 A1 · Performing electronic document segmentation using deep neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279461 A1
- Document ID: US20210279461A1
- Status: Published application
- Kind: A1
- Application number: 17/327,382
- Filed: 2021-05-21
- Published: 2021-09-09
- Earliest claimed: 2019-08-13
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Mausoom Sarkar; Arneh Jain
- Web page: https://patentgrep.com/patent/US20210279461A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/12: Edge-based segmentation
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques for document segmentation. In an example, a document processing application segments an electronic document image into strips. A first strip overlaps a second strip. The application generates a first mask indicating one or more elements and element types in the first strip by applying a predictive model network to image content in the first strip and a prior mask generated from image content of the first strip. The application generates a second mask indicating one or more elements and element types in the second strip by applying the predictive model network to image content in the second strip and the first mask. The application computes, from a combined mask derived from the first mask and the second mask, an output electronic document that identifies elements in the electronic document and the respective element types.

## Claims

### Claim 1

A computer-implemented method comprising:
accessing, by a processing device, a document image that is a captured image of a document;
segmenting the document image into a plurality of overlapping strips;
generating a plurality of masks indicating one or more elements and element types in the plurality of overlapping strips by applying a predictive neural network to:
a given strip of the plurality of overlapping strips, and
a portion of a previously-generated mask of an adjacent strip of the plurality of overlapping strips, the portion of the previously-generated mask of the adjacent strip corresponding to a portion of the given strip that overlaps with the adjacent strip;
generating, from the plurality of masks, a combined mask that indicates elements and corresponding element types present in the document; and
creating, from the combined mask, a segmented electronic version of the document that identifies the elements and corresponding element types.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating a mask for a first strip by applying the predictive neural network to the first strip and a previously generated blank mask.

### Claim 3 (depends on 2)

The method of claim 2, wherein segmenting the document image into a plurality of overlapping strips comprises:
creating the first strip by extracting a first portion of the electronic document image, wherein the first portion extends from an edge of the electronic document image by a width and comprises an intermediate point; and
creating a second strip by extracting a second portion of the electronic document image, wherein the second portion extends from the intermediate point by the width.

### Claim 4 (depends on 1)

The method of claim 1, wherein the predictive neural network comprises an encoder, a reconstruction decoder, a recurrent neural network, and a plurality of segmentation decoders.

### Claim 5 (depends on 4)

The method of claim 4, further comprising, for the given strip and corresponding portion of the previously-generated mask of the adjacent strip:
generating, utilizing the encoder and the recurrent neural network, one or more feature maps; and
generating, utilizing the plurality of segmentation decoders, one or more masks indicating the elements and the corresponding element types present in the given strip.

### Claim 6 (depends on 5)

The method of claim 5, wherein generating, utilizing the plurality of segmentation decoders, the one or more masks indicating the elements and the corresponding element types present in the given strip comprises generating a respective mask for each element type utilizing the plurality of segmentation decoders.

### Claim 7 (depends on 6)

The method of claim 6, further comprising combining the respective mask for each element type for each strip to generate a plurality of combined masks, each combined mask indicating a respective element type present in the document.

### Claim 8 (depends on 7)

The method as recited in claim 7, further comprising generating, utilizing the reconstruction decoder, an electronic version of the document.

### Claim 9 (depends on 8)

The method as recited in claim 8, wherein creating, from the combined mask, the segmented electronic version of the document comprising combining the electronic version of the document and the plurality of combined masks.

### Claim 10 (depends on 1)

The method of claim 1, wherein an element type is one of a border, a field, a choice field, a background, a text box, a widget, or an image.

### Claim 11

A non-transitory computer-readable medium having program code stored thereon that, when executed by a processing device, causes the processing device to perform operations comprising:
accessing a document image that is a captured image of a document;
segmenting the document image into a plurality of overlapping strips;
generating, utilizing a predictive neural network, a plurality of strip masks indicating one or more elements and element types in the plurality of overlapping strips;
generating, from the plurality of strip masks, a plurality of combined masks that each indicate elements of a given element type present in the document;
generating, utilizing the predictive neural network, an electronic version of the document; and
creating, from the plurality of combined masks and the electronic version of the document, a segmented electronic version of the document that identifies the elements and corresponding element types.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein generating the plurality of strip masks indicating one or more elements and element types in the plurality of overlapping strips comprises applying the predictive neural network to:
a given strip of the plurality of overlapping strips, and
a portion of a previously-generated mask of an adjacent strip of the plurality of overlapping strips, the portion of the previously-generated mask of the adjacent strip corresponding to a portion of the adjacent strip that overlaps with the given strip.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the predictive neural network comprises an encoder, a reconstruction decoder, a recurrent neural network, and a plurality of segmentation decoders.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, further comprising generating one or more feature maps for the plurality of strips utilizing the encoder and the recurrent neural network.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein generating, utilizing the predictive neural network, the electronic version of the document comprises generating the electronic version of the document from the one or more feature maps utilizing the reconstruction decoder.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein generating, from the plurality of strip masks, the plurality of combined masks that each indicate elements of a given element type present in the document comprises utilizing each segmentation decoder of the plurality of segmentation decoders to generate a combined mask for a respective element type.

### Claim 17

A system comprising:
a computer memory device comprising an electronic document image of a document and a predictive neural network; and
at least one processor configured to cause the system to:
segment the document image into a plurality of overlapping strips;
generate, utilizing the predictive neural network, a first plurality of strip masks indicating elements of a first element type in the plurality of overlapping strips;
generate, from the first plurality of strip masks, a first combined mask indicating elements of the first element type present in the document;
generate, utilizing the predictive neural network, a second plurality of strip masks indicating elements of a second element type in the plurality of overlapping strips;
generate, from the second plurality of strip masks, a second combined mask indicating elements of the first element type present in the document;
generate, utilizing the predictive neural network, an electronic version of the document; and
create, from the first and second combined masks and the electronic version of the document, a segmented electronic version of the document that identifies the elements of the first and second element type in the electronic version of the document.

### Claim 18 (depends on 17)

The system of claim 17, wherein the at least one processor is configured to cause the system to generate the first and second plurality of strip masks by applying the predictive neural network to: a given strip of the plurality of overlapping strips, and a portion of a previously-generated mask of an adjacent strip of the plurality of overlapping strips, the portion of the previously-generated mask of the adjacent strip corresponding to a portion of the adjacent strip that overlaps with the given strip.

### Claim 19 (depends on 17)

The system of claim 17, wherein:
the predictive neural network comprises an encoder, a reconstruction decoder, a recurrent neural network, and a plurality of segmentation decoders;
the at least one processor is configured to cause the system to:
generate one or more feature maps for the plurality of overlapping strips utilizing the encoder and the recurrent neural network; and
generate, utilizing the predictive neural network, the electronic version of the document by processing the one or more feature maps utilizing the reconstruction decoder.

### Claim 20 (depends on 19)

The system of claim 19, wherein the at least one processor is configured to cause the system to generate, from the first plurality of strip masks, the first combined mask by utilizing a first segmentation decoder of the plurality of segmentation decoders to generate the first combined mask from the one or more feature maps.
