# US 2023/0267345 A1 · Form structure extraction by predicting associations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267345 A1
- Document ID: US20230267345A1
- Status: Published application
- Kind: A1
- Application number: 18/135,948
- Filed: 2023-04-18
- Published: 2023-08-24
- Earliest claimed: 2020-06-17
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Milan Aggarwal; Mausoom Sarkar; Balaji Krishnamurthy
- Web page: https://patentgrep.com/patent/US20230267345A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06V 10/82: using neural networks
- G06V 30/19173: {Classification techniques}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques described herein extract form structures from a static form to facilitate making that static form reflowable. A method described herein includes accessing low-level form elements extracted from a static form. The method includes determining, using a first set of prediction models, second-level form elements based on the low-level form elements. Each second-level form element includes a respective one or more low-level form elements. The method further includes determining, using a second set of prediction models, high-level form elements based on the second-level form elements and the low-level form elements. Each high-level form element includes a respective one or more second-level form elements or low-level form elements. The method further includes generating a reflowable form based on the static form by, for each high-level form element, linking together the respective one or more second-level form elements or low-level form elements.

## Claims

### Claim 1

A method comprising:
accessing textruns and widgets extracted from a form;
applying a first set of prediction models to the textruns to determine textblocks from the textruns, each textblock of the textblocks comprising a respective group of textruns;
applying a second set of prediction models to the textblocks and the widgets to determine form groups, each form group of the form groups comprising a combination of textblocks or widgets; and
generating a reflowable form based on the form and comprising the form groups.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
predicting a first association between a first textrun and a second textrun;
predicting a second association between the second textrun and the first textrun; and
grouping the first textrun and the second textrun into a shared textblock based on the first association and the second association.

### Claim 3 (depends on 1)

The method of claim 1, further comprising training the first set of prediction models to predict groupings of textruns into textblocks.

### Claim 4 (depends on 3)

The method of claim 3, further comprising training the second set of prediction models to predict groupings of widgets and the textblocks into high-level form elements.

### Claim 5 (depends on 1)

The method of claim 1, wherein each prediction model in the first set of prediction models shares a respective common architecture with a corresponding prediction model in the second set of prediction models.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the first set of prediction models comprises a first image encoder, a first text encoder, a first context encoder, a first fusion model, and a first sequential association model;
the second set of prediction models comprises a second image encoder, a second text encoder, a second context encoder, a second fusion model, and a second sequential association model; and
the second set of prediction models are trained separately from the first set of prediction models.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second set of prediction models comprises a sequential association model configured to classify each of groupings of widgets and the textblocks into one of three or more classes of high-level form elements.

### Claim 8

A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
accessing textruns and widgets extracted from a form;
applying a first set of prediction models to the textruns to determine textblocks from the textruns, each textblock of the textblocks comprising a respective group of textruns;
applying a second set of prediction models to the textblocks and the widgets to determine form groups, each form group of the form groups comprising a combination of textblocks or widgets; and
generating a reflowable form based on the form and comprising the form groups.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the executable instructions further cause the processing device to perform operations comprising:
predicting a first association between a first textrun and a second textrun;
predicting a second association between the second textrun and the first textrun; and
grouping the first textrun and the second textrun into a shared textblock based on the first association and the second association.

### Claim 10 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the executable instructions further cause the processing device to perform operations comprising:
training the first set of prediction models to predict groupings of textruns into textblocks.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the executable instructions further cause the processing device to perform operations comprising:
training the second set of prediction models to predict groupings of widgets and the textblocks into high-level form elements.

### Claim 12 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein each prediction model in the first set of prediction models shares a respective common architecture with a corresponding prediction model in the second set of prediction models.

### Claim 13 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein:
the first set of prediction models comprises a first image encoder, a first text encoder, a first context encoder, a first fusion model, and a first sequential association model;
the second set of prediction models comprises a second image encoder, a second text encoder, a second context encoder, a second fusion model, and a second sequential association model; and
the second set of prediction models are trained separately from the first set of prediction models.

### Claim 14 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the second set of prediction models comprises a sequential association model configured to classify each of groupings of widgets and the textblocks into one of three or more classes of high-level form elements.

### Claim 15

A system comprising:
one or more processors; and
a memory component storing executable instructions, which when executed by the one or more processors, cause the one or more processors to perform operations comprising:
accessing textruns and widgets extracted from a form;
applying a first set of prediction models to the textruns to determine textblocks from the textruns, each textblock of the textblocks comprising a respective group of textruns;
applying a second set of prediction models to the textblocks and the widgets to determine form groups, each form group of the form groups comprising a combination of textblocks or widgets; and
generating a reflowable form based on the form and comprising the form groups.

### Claim 16 (depends on 15)

The system of claim 15, wherein the executable instructions further cause the one or more processors to perform operations comprising:
predicting a first association between a first textrun and a second textrun;
predicting a second association between the second textrun and the first textrun; and
grouping the first textrun and the second textrun into a shared textblock based on the first association and the second association.

### Claim 17 (depends on 15)

The system of claim 15, wherein the executable instructions further cause the one or more processors to perform operations comprising:
training the first set of prediction models to predict groupings of textruns into textblocks.

### Claim 18 (depends on 17)

The system of claim 17, wherein the executable instructions further cause the one or more processors to perform operations comprising:
training the second set of prediction models to predict groupings of widgets and the textblocks into high-level form elements.

### Claim 19 (depends on 15)

The system of claim 15, wherein each prediction model in the first set of prediction models shares a respective common architecture with a corresponding prediction model in the second set of prediction models.

### Claim 20 (depends on 15)

The system of claim 15, wherein:
the first set of prediction models comprises a first image encoder, a first text encoder, a first context encoder, a first fusion model, and a first sequential association model;
the second set of prediction models comprises a second image encoder, a second text encoder, a second context encoder, a second fusion model, and a second sequential association model; and
the second set of prediction models are trained separately from the first set of prediction models.
