# US 11,908,219 B2 · Method and device for processing information, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,219 B2
- Document ID: US11908219B2
- Status: Granted
- Kind: B2
- Application number: 17/244,291
- Filed: 2021-04-29
- Published: 2021-09-02
- Granted: 2024-02-20
- Earliest claimed: 2020-09-29
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd.
- Inventors: Zihan Ni; Yipeng Sun; Kun Yao; Junyu Han; Errui Ding; Jingtuo Liu; Haifeng Wang
- Web page: https://patentgrep.com/patent/US11908219B2

## CPC

- G06F 16/5846: {using extracted text}
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06F 16/355: Creation or modification of classes or clusters
- G06F 40/30: Semantic analysis
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 10/82: using neural networks
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The disclosure provides a method and a device for processing information, an electronic device, and a storage medium, belonging to a field of artificial intelligence including computer vision, deep learning, and natural language processing. In the method, the computing device recognizes multiple text items in the image. The computing device classifies multiple text items into a first set of name text items and a second set of content text items based on semantics of the text items. The computing device performs a matching operation between the first set and the second set based on a layout of the text items in the image, and determines matched name-content text items. The matched name-content text items include a name text item in the first set and a content text item matching the name text item and in the second set. The computing device outputs the matched name-content text items.

## Claims

### Claim 1

A method for processing information, including:
recognizing text items in an image through an optical character recognition (OCR) technology or through a text recognition model based on a convolutional neural network;
classifying the text items into a first set of name text items and a second set of content text items based on semantics of the text items;
performing a matching operation between the first set and the second set based on a layout of the text items in the image, and determining matched name-content text items, wherein the matched name-content text items comprise a name text item in the first set and a content text item matching the name text item and in the second set; and
outputting the matched name-content text items;
wherein performing the matching operation between the first set and the second set comprises:
determining a candidate set comprising content text items to be matched with a name text item to be matched in the first set;
determining a candidate content text item nearest the name text item to be matched from the candidate set; and
determining that the candidate content text item satisfying a matching condition matches the name text item to be matched;
wherein the matching condition comprises at least one of:
a positional relationship between the candidate content text item and the name text item to be matched conforming to an information arrangement of the image;
a distance between the candidate content text item and the name text item to be matched being less than a distance threshold; or
a difference between a reference angle and an intersection angle between a reference direction and a line connecting center points of the candidate content text item and the name text item to be matched being less than an angle threshold.

### Claim 2 (depends on 1)

The method of claim 1, wherein recognizing the text items in the image comprises:
detecting text item regions in the image based on a text item detection model; and
recognizing the text items from the text item regions based on a text recognition model.

### Claim 3 (depends on 2)

The method of claim 2, wherein the text item detection model is constructed by performing incremental learning of an initial text item detection model based on specific data, the initial text item detection model is trained using general data, and the specific data includes annotated name text items and annotated content text items.

### Claim 4 (depends on 1)

The method of claim 1, wherein classifying the text items into the first set and the second set comprises:
determining whether each text item is a name text item or a content text item based on a semantic classification model.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
removing the candidate content text item unsatisfying the matching condition from the candidate set.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining that no content text item matches the name text item to be matched based on the number of removed content text items from the candidate set being equal to a threshold.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining a pair of reference text items corresponding to a minimum distance, wherein the pair of reference text items comprises a reference name text item in the first set and a reference content text item in the second set; and
determining the information arrangement of the image based on a positional relationship between the reference name text item and the reference content text item.

### Claim 8 (depends on 7)

The method of claim 7, wherein determining the information arrangement comprises:
determining an intersection angle between a reference direction and a line connecting center points of the reference name text item and the reference content text item;
determining the information arrangement as a left-right pattern based on the intersection angle being within a first angle range;
determining the information arrangement as an up-down pattern based on the intersection angle being within a second angle range; and
determining the information arrangement as a right-left pattern based on the intersection angle being within a third angle range.

### Claim 9 (depends on 1)

The method of claim 1, further comprising at least one of:
determining distances of pairs of matched name-content text items and determining the distance threshold based on an average value of the distances; or
determining intersection angles between the reference direction and lines connecting the center points of pairs of matched name-content text items, and determining the reference angle based on an average value of the intersection angles.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining a first rectangular region corresponding to the name text item to be matched in the image;
determining a second rectangular region corresponding to the candidate content text item in the image; and
determining the distance between the candidate content text item and the name text item to be matched based on a distance between corner points of the first rectangular region and the second rectangular region.

### Claim 11

An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor;
wherein the memory is configured to store instructions executable by the at least one processor, when the instructions are executed by the at least one processor, the at least one processor is configured to:
recognize text items in an image through an optical character recognition (OCR) technology or through a text recognition model based on a convolutional neural network;
classify the text items into a first set of name text items and a second set of content text items based on semantics of the text items;
perform a matching operation between the first set and the second set based on a layout of the text items in the image, and determine matched name-content text items, wherein the matched name-content text items comprise a name text item in the first set and a content text item matching the name text item and in the second set; and
output the matched name-content text items;
wherein the at least one processor is further configured to:
determine a candidate set comprising content text items to be matched with a name text item to be matched in the first set;
determine a candidate content text item nearest the name text item to be matched from the candidate set; and
determine that the candidate content text item satisfying a matching condition matches the name text item to be matched;
wherein the matching condition comprises at least one of:
a positional relationship between the candidate content text item and the name text item to be matched conforming to an information arrangement of the image;
a distance between the candidate content text item and the name text item to be matched being less than a distance threshold; or
a difference between a reference angle and an intersection angle between a reference direction and a line connecting center points of the candidate content text item and the name text item to be matched being less than an angle threshold.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to:
detect text item regions in the image based on a text item detection model; and
recognize the text items from the text item regions based on a text recognition model.

### Claim 13 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to:
determine whether each text item is a name text item or a content text item based on a semantic classification model.

### Claim 14 (depends on 13)

The electronic device of claim 13, wherein the at least one processor is further configured to:
determine a first rectangular region corresponding to the name text item to be matched in the image;
determine a second rectangular region corresponding to the candidate content text item in the image; and
determine the distance between the candidate content text item and the name text item to be matched based on a distance between corner points of the first rectangular region and the second rectangular region.

### Claim 15 (depends on 11)

The electronic device of claim 11, wherein the at least one processor is further configured to:
determine a pair of reference text items corresponding to a minimum distance, wherein the pair of reference text items comprises a reference name text item in the first set and a reference content text item in the second set; and
determine the information arrangement of the image based on a positional relationship between the reference name text item and the reference content text item.

### Claim 16

A non-transitory computer-readable storage medium, having computer instructions stored thereon, wherein the computer instructions are configured to cause a computer to execute a method for processing information, the method comprising:
recognizing text items in an image through an optical character recognition (OCR) technology or through a text recognition model based on a convolutional neural network;
classifying the text items into a first set of name text items and a second set of content text items based on semantics of the text items;
performing a matching operation between the first set and the second set based on a layout of the text items in the image, and determining matched name-content text items, wherein the matched name-content text items comprise a name text item in the first set and a content text item matching the name text item and in the second set; and
outputting the matched name-content text items;
wherein performing the matching operation between the first set and the second set comprises:
determining a candidate set comprising content text items to be matched with a name text item to be matched in the first set;
determining a candidate content text item nearest the name text item to be matched from the candidate set; and
determining that the candidate content text item satisfying a matching condition matches the name text item to be matched;
wherein the matching condition comprises at least one of:
a positional relationship between the candidate content text item and the name text item to be matched conforming to an information arrangement of the image;
a distance between the candidate content text item and the name text item to be matched being less than a distance threshold; or
a difference between a reference angle and an intersection angle between a reference direction and a line connecting center points of the candidate content text item and the name text item to be matched being less than an angle threshold.