# US 12,183,104 B2 · Handwriting recognition pipelines for genealogical records

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,104 B2
- Document ID: US12183104B2
- Status: Granted
- Kind: B2
- Application number: 17/867,390
- Filed: 2022-07-18
- Published: 2023-01-12
- Granted: 2024-12-31
- Earliest claimed: 2021-07-09
- Assignees: Ancestry.com Operations Inc.
- Applicants: Ancestry.com Operations Inc.
- Inventors: Masaki Stanley Fujimoto; Kalyan Chakravarthi Murahari; Siteng Chen
- Web page: https://patentgrep.com/patent/US12183104B2

## CPC

- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06Q 10/10: Office automation; Time management
- G06Q 50/18: Legal services
- G06Q 50/26: Government or public services (business processes related to the transportation industry G06Q50/40)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06V 30/1801: {Detecting partial patterns, e.g. edges or contours, or configurations, e.g. loops, corners, strokes or intersections (extracting features by contour coding G06V30/182)}
- G06V 30/18057: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 30/226: of cursive writing
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/42: based on the type of document
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Disclosed herein relates to example embodiments for recognizing handwritten information in a genealogical record. A computing server may receive a genealogical record. The genealogical record may take the form of an image of a physical form having a structured layout, fields, and handwritten information. The computing server may divide the genealogical record into a plurality of areas based on the structured layout. The computing server may identify, for a particular area, a type of field that is included within the particular area. The computing server may select a handwriting recognition model for identifying the handwritten information in the particular area. The handwriting recognition model may be selected based on the type of the field. The computing server may input an image of the particular area to the handwriting recognition model to generate text of the handwritten information. The computing server may store the text of the handwritten information.

## Claims

### Claim 1

A computer-implemented method, comprising:
receiving a genealogical record, the genealogical record being an image of a physical form having a structured layout and a plurality of fields, the physical form having handwritten information;
dividing the genealogical record into a plurality of areas based on the structured layout;
identifying, for a particular area, a type of a field that is included within the particular area;
selecting, from a set of candidate handwriting recognition models, a handwriting recognition model for identifying the handwritten information in the particular area, the handwriting recognition model selected based on the type of the field included within the particular area;
inputting an image of the particular area to the handwriting recognition model to generate text of the handwritten information; and
storing the text of the handwritten information as data associated with the genealogical record in a genealogical database.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising generating a confidence score for the text predicted by the handwriting recognition model.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the confidence score is generated by an audit model, and wherein the audit model is trained together with the handwriting recognition model.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the audit model is configured to receive a plurality of image features from different layers of the handwriting recognition model and generate the confidence score based on the plurality of image features.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein dividing the genealogical record into a plurality of areas comprises:
identifying a first set of keypoints in the genealogical record;
comparing the first set of keypoints to a second set of keypoints in a template record having template area coordinates;
generating a transform based on the first and second set of keypoints; and
generating divided area coordinates by applying the transform to the template area coordinates in the template record.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein identifying the first set of keypoints in the genealogical record comprises comparing segments of the image of the genealogical record to segments of the template record, and the segments comprise corners but not a center.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the type of the field in the particular area includes a name field, and the handwriting recognition model comprises convolution layers, a transformer encoder and a transformer decoder.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the type of the field in the particular area includes a checkbox field, and the handwriting recognition model comprises convolution layers that is configured to identify a checkbox and a binary classifier that is configured to determine whether the checkbox is checked.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the type of the field in the particular area includes a machine labeled field, and the handwriting recognition model comprises a field prior that provides information related to machine-printed labels in the machine labeled field.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the handwriting recognition model comprises:
a convolutional neural network backbone that is configured to identify image features of the image;
a sequence engine that includes a bi-directional long short-term memory network that is configured to use the image features to generate a sequence prediction; and
a decoder that is configured to generate the sequence prediction to a text prediction.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein generating the text of the handwritten information by the handwriting recognition model comprises:
dividing a handwritten area into multiple spatial regions;
identifying a character in each spatial region;
generating a string of characters based on the character identified in each spatial region; and
generating a text prediction based on the string of characters.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the handwritten information comprises a first collection of handwriting written in a first direction and a second collection of handwriting written in a second direction, and the handwriting recognition model is configured to rotate the second collection of handwriting written in the second direction.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the genealogical record is part of a historical collection that has a recurring layout, and the structured layout is part of the recurring layout.

### Claim 14

A system comprising:
one or more processors; and
memory coupled to the one or more processors, the memory configured to store computer code comprising instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform steps comprising:
receiving a genealogical record, the genealogical record being an image of a physical form having a structured layout and a plurality of fields, the physical form having handwritten information;
dividing the genealogical record into a plurality of areas based on the structured layout;
identifying, for a particular area, a type of a field that is included within the particular area;
selecting, from a set of candidate handwriting recognition models, a handwriting recognition model for identifying the handwritten information in the particular area, the handwriting recognition model selected based on the type of the field included within the particular area;
inputting an image of the particular area to the handwriting recognition model to generate text of the handwritten information; and
storing the text of the handwritten information as data associated with the genealogical record in a genealogical database.

### Claim 15 (depends on 14)

The system of claim 14, wherein the steps further comprise generating a confidence score for the text predicted by the handwriting recognition model.

### Claim 16 (depends on 14)

The system of claim 14, wherein dividing the genealogical record into a plurality of areas comprises:
identifying a first set of keypoints in the genealogical record;
comparing the first set of keypoints to a second set of keypoints in a template record having template area coordinates;
generating a transform based on the first and second set of keypoints; and
generating divided area coordinates by applying the transform to the template area coordinates in the template record.

### Claim 17 (depends on 14)

The system of claim 14, wherein the type of the field in the particular area includes a name field, and the handwriting recognition model comprises convolution layers, a transformer encoder and a transformer decoder.

### Claim 18 (depends on 14)

The system of claim 14, wherein the type of the field in the particular area includes a checkbox field, and the handwriting recognition model comprises convolution layers that is configured to identify a checkbox and a binary classifier that is configured to determine whether the checkbox is checked.

### Claim 19 (depends on 14)

The system of claim 14, wherein the handwriting recognition model comprises:
a convolutional neural network backbone that is configured to identify image features of the image;
a sequence engine that includes a bi-directional long short-term memory network that is configured to use the image features to generate a sequence prediction; and
a decoder that is configured to generate the sequence prediction to a text prediction.

### Claim 20

A non-transitory computer-readable medium configured to store computer code comprising instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform steps comprising:
receiving a genealogical record, the genealogical record being an image of a physical form having a structured layout and a plurality of fields, the physical form having handwritten information;
dividing the genealogical record into a plurality of areas based on the structured layout;
identifying, for a particular area, a type of a field that is included within the particular area;
selecting, from a set of candidate handwriting recognition models, a handwriting recognition model for identifying the handwritten information in the particular area, the handwriting recognition model selected based on the type of the field included within the particular area;
inputting an image of the particular area to the handwriting recognition model to generate text of the handwritten information; and
storing the text of the handwritten information as data associated with the genealogical record in a genealogical database.