# US 2024/0371190 A1 · Entry detection and recognition for custom forms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0371190 A1
- Document ID: US20240371190A1
- Status: Published application
- Kind: A1
- Application number: 18/761,241
- Filed: 2024-07-01
- Published: 2024-11-07
- Earliest claimed: 2021-09-13
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Yijuan LU; Lynsey LIU; Andrei A. GAIVORONSKI; Yu CHENG; Dinei Afonso Ferreira FLORENCIO; Cha ZHANG; John Richard CORRING
- Web page: https://patentgrep.com/patent/US20240371190A1

## CPC

- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 40/174: Form filling; Merging
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 20/00: Machine learning
- G06V 30/1448: {based on markings or identifiers characterising the document or the area}
- G06V 30/147: {Determination of region of interest}
- G06V 30/19067: {Matching configurations of points or features, e.g. constellation matching}
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06F 40/279: Recognition of textual entities
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The disclosure herein describes providing signature data of an input document. Text data of the input document is obtained (e.g., OCR data generated from image data) and a first set of signature fields are identified using signature key-value pairs of the text data. A first subset of signed signature fields and a first subset of unsigned signature fields are determined based on mapping to a set of predicted values. A second set of signature fields are determined using a region prediction model applied to image data of the input document. Region images associated with the first subset of unsigned signature fields and with second set of signature fields are obtained and a second set of signed signature fields and a second set of unsigned signature fields are determined using a signature recognition model. Signature output data is provided including signed signature fields and/or unsigned signature fields.

## Claims

### Claim 1

A system comprising:
a processor; and
a memory comprising computer program code, the memory and the computer program code configured to, with the processor, cause the processor to:
identify a first set of entry fields of an input document using entry key-value pairs of text data of the input document;
determine a first subset of filled entry fields and a first subset of unfilled entry fields from the first set of entry fields based on mapping to a set of predicted values;
determine a second set of entry fields of the input document using a region prediction model, the region prediction model trained to identify an entry region in the input document, wherein the region prediction model is trained using a set of anchors identified in training document data and a set of points sampled from the training document data;
obtain region images associated with each entry field of the first subset of unfilled entry fields and with each entry field of the second set of entry fields;
determine a second subset of filled entry fields and a second subset of unfilled entry fields from the obtained region images using an entry recognition model; and
provide entry output data of the input document, wherein the entry output data includes at least one of the following: the first subset of filled entry fields, the second subset of filled entry fields, and the second subset of unfilled entry fields.

### Claim 2 (depends on 1)

The system of claim 1, wherein the training document data comprises training document image data and training document text data, and wherein the set of points are sampled from the training document image data.

### Claim 3 (depends on 1)

The system of claim 1, wherein determining the second set of entry fields of the input document using the region prediction model comprises:
identifying, by the region prediction model, anchors in the input documents; and
determining, by the region prediction model, points in the input document associated with the second set of entry fields using locations of the identified anchors in the input document.

### Claim 4 (depends on 1)

The system of claim 1, wherein the set of anchors comprises form labels, heading text, table corners, or patterns.

### Claim 5 (depends on 1)

The system of claim 1, wherein the set of points are labeled as being in a signature region or not in the signature region.

### Claim 6 (depends on 1)

The system of claim 1, wherein the set of points are labeled to indicate a distance and/or direction from a signature region.

### Claim 7 (depends on 1)

The system of claim 1, wherein the set of points are labeled according to proximity to a signature region.

### Claim 8 (depends on 1)

The system of claim 1, wherein the region prediction model is applied to Optical Character Recognition (OCR) text data or layout data of the input document.

### Claim 9

A computerized method for providing signature data of an input document, the computerized method comprising:
identifying a first set of entry fields of an input document using entry key-value pairs of text data of the input document;
determining a first subset of filled entry fields and a first subset of unfilled entry fields from the first set of entry fields based on mapping to a set of predicted values;
determining a second set of entry fields of the input document using a region prediction model, the region prediction model trained to identify an entry region in the input document, wherein the region prediction model is trained using a set of anchors identified in training document data and a set of points sampled from the training document data;
obtaining region images associated with each entry field of the first subset of unfilled entry fields and with each entry field of the second set of entry fields;
determining a second subset of filled entry fields and a second subset of unfilled entry fields from the obtained region images using an entry recognition model; and
providing entry output data of the input document, wherein the entry output data includes at least one of the following: the first subset of filled entry fields, the second subset of filled entry fields, and the second subset of unfilled entry fields.

### Claim 10 (depends on 9)

The computerized method of claim 9, wherein the training document data comprises training document image data and training document text data, and wherein the set of points are sampled from the training document image data.

### Claim 11 (depends on 9)

The computerized method of claim 9, wherein determining the second set of entry fields of the input document using the region prediction model comprises:
identifying, by the region prediction model, anchors in the input documents; and
determining, by the region prediction model, points in the input document associated with the second set of entry fields using locations of the identified anchors in the input document.

### Claim 12 (depends on 9)

The computerized method of claim 9, wherein the set of anchors comprises form labels, heading text, table corners, or patterns.

### Claim 13 (depends on 9)

The computerized method of claim 9, wherein the set of points are labeled as being in a signature region or not in the signature region.

### Claim 14 (depends on 9)

The computerized method of claim 9, wherein the set of points are labeled to indicate a distance and/or direction from a signature region.

### Claim 15 (depends on 9)

The computerized method of claim 9, wherein the set of points are labeled according to proximity to a signature region.

### Claim 16 (depends on 9)

The computerized method of claim 9, wherein the region prediction model is applied to Optical Character Recognition (OCR) text data or layout data of the input document.

### Claim 17

One or more computer storage media having computer-executable instructions for providing entry field data of an input document that, upon execution by a computer, cause the processor to at least:
identify a first set of entry fields of an input document using entry key-value pairs of text data of the input document;
determine a first subset of filled entry fields and a first subset of unfilled entry fields from the first set of entry fields based on mapping to a set of predicted values;
determine a second set of entry fields of the input document using a region prediction model, the region prediction model trained to identify an entry region in the input document, wherein the region prediction model is trained using a set of anchors identified in training document data and a set of points sampled from the training document data;
obtain region images associated with each entry field of the first subset of unfilled entry fields and with each entry field of the second set of entry fields;
determine a second subset of filled entry fields and a second subset of unfilled entry fields from the obtained region images using an entry recognition model; and
provide entry output data of the input document, wherein the entry output data includes at least one of the following: the first subset of filled entry fields, the second subset of filled entry fields, and the second subset of unfilled entry fields.

### Claim 18 (depends on 17)

The one or more computer storage media of claim 17, wherein determining the second set of entry fields of the input document using the region prediction model comprises:
identifying, by the region prediction model, anchors in the input documents; and
determining, by the region prediction model, points in the input document associated with the second set of entry fields using locations of the identified anchors in the input document.

### Claim 19 (depends on 17)

The one or more computer storage media of claim 17, wherein the set of points are labeled as one of the following: being in a signature region or not in the signature region, indicating a distance and/or direction from the signature region, or indicating proximity to the signature region.

### Claim 20 (depends on 17)

The one or more computer storage media of claim 17, wherein the region prediction model is applied to Optical Character Recognition (OCR) text data or layout data of the input document.
