# US 2023/0410541 A1 · Segmentation of page stream documents for bidirectional encoder representational transformers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410541 A1
- Document ID: US20230410541A1
- Status: Published application
- Kind: A1
- Application number: 17/843,991
- Filed: 2022-06-18
- Published: 2023-12-21
- Earliest claimed: 2022-06-18
- Assignees: KYOCERA Document Solutions Inc.
- Applicants: KYOCERA Document Solutions Inc.
- Inventors: Yury Ageev
- Web page: https://patentgrep.com/patent/US20230410541A1

## CPC

- G06V 30/10: Character recognition
- G06F 16/35: Clustering; Classification
- G06F 16/36: Creation of semantic tools, e.g. ontology or thesauri
- G06F 16/906: Clustering; Classification
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods relate generally to performing a machine learning task on training documents to generate an output. In an example method, a pretrained Sentence Bidirectional Encoder Representational Transformers (“S-BERT”) model is obtained. The training documents are scanned by a plurality of scanners. Content of the training documents is recognized with character recognition. The content is templated responsive to the character recognition. The content is processed with the pretrained S-BERT model for training thereof. A trained S-BERT model is generated from the processing of the content as the output. The trained S-BERT model is configured to automatically categorize and assemble non-training documents into original configurations thereof.

## Claims

### Claim 1

A method for performing a machine learning task on training documents to generate an output, comprising:
obtaining a pretrained Sentence Bidirectional Encoder Representational Transformers (“S-BERT”) model;
scanning the training documents by a plurality of scanners;
recognizing content of the training documents with character recognition;
templating the content responsive to the character recognition;
processing the content with the pretrained S-BERT model for training thereof; and
generating a trained S-BERT model from the processing of the content as the output;
wherein the trained S-BERT model is configured to automatically categorize and assemble non-training documents into original configurations thereof.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising streaming the content for:
extracting features from and segmenting and the content; and
categorizing the content.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising:
buffering the content after the extracting, the segmenting and the categorizing in a buffer memory; and
reading the content from the buffer memory for the processing.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the trained S-BERT model is configured for linking multiple files to reassemble the training documents.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the trained S-BERT model is configured with a merge function to order multiple scanned pages corresponding to the training documents distributed across the plurality of scanners into original sets thereof.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the character recognition is optical character recognition.

### Claim 7 (depends on 5)

The method according to claim 5, wherein the character recognition is optical word recognition.

### Claim 8 (depends on 4)

The method according to claim 4, wherein the content is first content, the method further comprising:
reprocessing each of the training documents with the character recognition to provide second content; and
streaming the second content for:
extracting features from and segmenting and the second content; and
categorizing the second content.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising:
processing the second content with the trained S-BERT model for tuning thereof; and
generating a fine-tuned S-BERT model from the processing of the second content as the output.

### Claim 10 (depends on 9)

The method according to claim 9, wherein:
the training documents are first training documents; and
the reprocessing is of second training documents.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the fine-tuned S-BERT model is configured to automatically categorize and assemble the non-training documents into the original configurations thereof with greater accuracy than the trained S-BERT model.

### Claim 12 (depends on 9)

The method according to claim 9, further comprising buffering the second content in the buffer memory;

### Claim 13 (depends on 13)

The method according to claim 13, wherein the S-BERT model is implemented as a stack of encoders, a pooling layer and an embedding layer.

### Claim 14 (depends on 13)

The method according to claim 13, wherein each of the encoders of the stack comprises a self-attention head, a first and a second adder and normalizer, and a first and a second feed forward neural network.

### Claim 15

A system for performing a machine learning task on training documents to generate an output, the system comprising:
a plurality of scanners configured to scan the training documents;
each of the plurality of scanners configured with character recognition to recognize content of the training documents;
each of the plurality of scanners configured to template the content responsive to the character recognition;
a pretrained Sentence Bidirectional Encoder Representational Transformers (“S-BERT”) model configured to process the content to train the S-BERT model; and
a trained S-BERT model generated from the processing of the content as the output;
wherein the trained S-BERT model is configured to automatically categorize and assemble non-training documents into original configurations thereof.

### Claim 16 (depends on 14)

The system according to claim 14, wherein each of the plurality of scanners is configured to:
extract features from and segmenting and the content; and
categorize the content.

### Claim 17 (depends on 16)

The system according to claim 16, further comprising:
a buffer memory coupled to buffer streams of the content after extracting, segmenting and categorizing thereof; and
wherein the S-BERT model is configured to read the content from the buffer memory for processing thereof.

### Claim 18 (depends on 17)

The system according to claim 17, wherein the trained S-BERT model is configured to link multiple files to reassemble the training documents.

### Claim 19 (depends on 17)

The system according to claim 17, wherein the trained S-BERT model is configured with a merge function to order multiple scanned pages corresponding to the training documents distributed across the plurality of scanners into original sets thereof.

### Claim 20 (depends on 19)

The system according to claim 19, wherein the content is first content, the system further comprising:
each of the plurality of scanners configured with the character recognition to recognize second content of the training documents from reprocessing thereof;
each of the plurality of scanners configured to stream the second content to:
extract features from and segment and the second content; and
categorize the second content; and
the trained S-BERT model configured to process the second content for tuning thereof to generate a fine-tuned S-BERT model.
