# US 12,008,319 B2 · Method and apparatus for selecting answers to idiom fill-in-the-blank questions, and computer device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,319 B2
- Document ID: US12008319B2
- Status: Granted
- Kind: B2
- Application number: 17/613,506
- Filed: 2020-11-30
- Published: 2022-08-18
- Granted: 2024-06-11
- Earliest claimed: 2020-09-04
- Assignees: Ping An Technology (Shenzhen) Co., Ltd.
- Applicants: Ping An Technology (Shenzhen) Co., Ltd.
- Inventors: Xiang Liu; Xiuling Chen
- Web page: https://patentgrep.com/patent/US12008319B2

## CPC

- G06F 40/16: Automatic learning of transformation rules, e.g. from examples
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 16/3329: Natural language query formulation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 40/30: Semantic analysis
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06Q 50/20: Education
- G09B 7/02: of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
- G09B 7/06: of the multiple-choice answer-type, i.e. where a given question is provided with a series of answers and a choice has to be made from the answers
- G09B 19/06: Foreign languages (with audible presentation of material to be studied G09B5/04)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

Disclosed are a method and apparatus for selecting answers to idiom fill-in-the-blank questions, a computer device, and a storage medium. The method includes: obtaining a question text of idiom fill-in-the-blank questions, the question text including a fill-in-the-blank text and n candidate idioms, and the fill-in-the-blank text including m fill-in-the-blanks to be filled in with the candidate idioms; obtaining an explanatory text of all the candidate idioms; obtaining, through an idiom selection fill-in-the-blank model, a confidence that each fill-in-the-blank is filled in with each candidate idiom; selecting m idioms from the n candidate idioms to form multiple groups of answers; calculating a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers; and obtaining a group of answers with the highest confidence sum as answers to the idiom fill-in-the-blank questions. The present application implements answers to idiom fill-in-the-blank questions with high accuracy.

## Claims

### Claim 1

A method for selecting answers to idiom fill-in-the-blank questions, comprising the following steps:
obtaining a question text of idiom fill-in-the-blank questions, wherein the question text comprises a fill-in-the-blank text and n candidate idioms, and the fill-in-the-blank text comprises m fill-in-the-blanks to be filled in with the candidate idioms;
obtaining an explanatory text of all the candidate idioms;
inputting the fill-in-the-blank text and the explanatory text into a pre-trained idiom selection fill-in-the-blank model, to obtain a confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms;
selecting m idioms from the n candidate idioms, and randomly arranging the m idioms to fill in the m fill-in-the-blanks to form a plurality of groups of answers, wherein in each group of answers, the candidate idioms can be selected to fill in the fill-in-the-blanks only once at most;
calculating, based on a KM algorithm and the confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers; and
obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions.

### Claim 2 (depends on 1)

The method for selecting answers to idiom fill-in-the-blank questions according to claim 1, before the step of inputting the fill-in-the-blank text and the explanatory text into a pre-trained idiom selection fill-in-the-blank model, to obtain a confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, comprising:
obtaining training samples, wherein the training samples comprise a plurality of corpus texts of idiom fill-in-the-blank questions with selected answers and an explanatory text of all idioms in an idiom library; and
inputting the training samples into a BERT language model-based natural language neural network for training to obtain the idiom selection fill-in-the-blank model, wherein the natural language neural network comprises a network output layer and a convolutional layer, the convolutional layer is formed by stacking a plurality of convolutional networks in sequence, an input layer of a latter convolutional network is connected to output layers of all former convolutional networks, an output layer of each of the convolutional networks is connected to an output layer of the BERT language model, and an output layer of the last convolutional network is connected to the network output layer.

### Claim 3 (depends on 2)

The method for selecting answers to idiom fill-in-the-blank questions according to claim 2, wherein the step of inputting the training samples into a BERT language model-based natural language neural network for training to obtain the idiom selection fill-in-the-blank model comprises:
inputting the training samples into the BERT language model, and extracting a feature matrix of the training samples based on the BERT language model; and
inputting the feature matrix into the natural language neural network for iterative training to obtain the idiom selection fill-in-the-blank model.

### Claim 4 (depends on 1)

The method for selecting answers to idiom fill-in-the-blank questions according to claim 1, before the step of calculating, based on a KM algorithm and the confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers, comprising:
separately determining whether there is at least one candidate idiom in each group of answers for which the confidence that each of the fill-in-the-blanks is filled in with the at least one candidate idiom is less than a first threshold; and
if yes, eliminating corresponding answers, and skipping calculating a sum of the confidences that each of the fill-in-the-blanks is filled in with the at least one candidate idiom in the eliminated answers.

### Claim 5 (depends on 1)

The method for selecting answers to idiom fill-in-the-blank questions according to claim 1, after the step of obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions, further comprising:
filling in the fill-in-the-blank text with the target answers to obtain an answer text;
obtaining a score for the answer text given by a user, and determining whether the score is greater than a second threshold, wherein the user scores the answer text based on the standard answers;
if the score is greater than the second threshold, forming a training set from the answer text and an explanatory text of all candidate idioms of the target answers, and inputting the training set into the idiom selection fill-in-the-blank model for retraining; and
saving the retrained idiom selection fill-in-the-blank model to a blockchain.

### Claim 6 (depends on 1)

The method for selecting answers to idiom fill-in-the-blank questions according to claim 1, wherein the idiom fill-in-the-blank questions are used to test a student; and
after the step of obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions, the method further comprises:
obtaining a test text uploaded by a terminal of the student, wherein the test text is a text in which the student fills in the fill-in-the-blank text of the idiom fill-in-the-blank questions with student answers; and when the terminal of the student uploads the test text, sequentially splicing the student answers with which the idiom fill-in-the-blank questions are filled in to obtain a first idiom combination, performing hash calculation on the first idiom combination based on the blockchain to obtain a corresponding hash value, and adding the hash value as an identification code to the test text;
extracting all student answers in the test text, and sequentially splicing all the student answers to obtain a second idiom combination;
performing hash calculation on the second idiom combination based on the blockchain to obtain a corresponding verification hash value;
identifying the identification code in the test text, and verifying whether the verification hash value is consistent with the identification code; and
if yes, comparing the student answers with the target answers one by one to obtain accuracy of the student answers.

### Claim 7

A computer device, comprising a memory and a processor, wherein a computer program is stored in the memory; and when the processor executes the computer program, the computer device implements a method for selecting answers to idiom fill-in-the-blank questions by:
obtaining a question text of idiom fill-in-the-blank questions, wherein the question text comprises a fill-in-the-blank text and n candidate idioms, and the fill-in-the-blank text comprises m fill-in-the-blanks to be filled in with the candidate idioms;
obtaining an explanatory text of all the candidate idioms;
inputting the fill-in-the-blank text and the explanatory text into a pre-trained idiom selection fill-in-the-blank model, to obtain a confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms;
selecting m idioms from the n candidate idioms, and randomly arranging the m idioms to fill in the m fill-in-the-blanks to form a plurality of groups of answers, wherein in each group of answers, the candidate idioms can be selected to fill in the fill-in-the-blanks only once at most;
calculating, based on a KM algorithm and the confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers; and
obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions.

### Claim 8 (depends on 7)

The computer device according to claim 7, wherein before the step of inputting the fill-in-the-blank text and the explanatory text into a pre-trained idiom selection fill-in-the-blank model, to obtain a confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, the computer device is configured to:
obtain training samples, wherein the training samples comprise a plurality of corpus texts of idiom fill-in-the-blank questions with selected answers and an explanatory text of all idioms in an idiom library; and
input the training samples into a BERT language model-based natural language neural network for training to obtain the idiom selection fill-in-the-blank model, wherein the natural language neural network comprises a network output layer and a convolutional layer, the convolutional layer is formed by stacking a plurality of convolutional networks in sequence, an input layer of a latter convolutional network is connected to output layers of all former convolutional networks, an output layer of each of the convolutional networks is connected to an output layer of the BERT language model, and an output layer of the last convolutional network is connected to the network output layer.

### Claim 9 (depends on 8)

The computer device according to claim 8, wherein the computer device is specifically configured to:
input the training samples into the BERT language model, and extract a feature matrix of the training samples based on the BERT language model; and
input the feature matrix into the natural language neural network for iterative training to obtain the idiom selection fill-in-the-blank model.

### Claim 10 (depends on 7)

The computer device according to claim 7, wherein before the step of calculating, based on a KM algorithm and the confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers, the computer device is configured to:
separately determine whether there is at least one candidate idiom in each group of answers for which the confidence that each of the fill-in-the-blanks is filled in with the at least one candidate idiom is less than a first threshold; and
if yes, eliminate corresponding answers, and skip calculating a sum of the confidences that each of the fill-in-the-blanks is filled in with the at least one candidate idiom in the eliminated answers.

### Claim 11 (depends on 7)

The computer device according to claim 7, wherein after the step of obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions, the computer device is further configured to:
fill in the fill-in-the-blank text with the target answers to obtain an answer text;
obtain a score for the answer text given by a user, and determine whether the score is greater than a second threshold, wherein the user scores the answer text based on the standard answers;
if the score is greater than the second threshold, form a training set from the answer text and an explanatory text of all candidate idioms of the target answers, and input the training set into the idiom selection fill-in-the-blank model for retraining; and
save the retrained idiom selection fill-in-the-blank model to a blockchain.

### Claim 12 (depends on 7)

The computer device according to claim 7, wherein the idiom fill-in-the-blank questions are used to test a student; and
after the step of obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions, the computer device is further configured to:
obtain a test text uploaded by a terminal of the student, wherein the test text is a text in which the student fills in the fill-in-the-blank text of the idiom fill-in-the-blank questions with student answers; and when the terminal of the student uploads the test text, sequentially splice the student answers with which the idiom fill-in-the-blank questions are filled in to obtain a first idiom combination, perform hash calculation on the first idiom combination based on the blockchain to obtain a corresponding hash value, and add the hash value as an identification code to the test text;
extract all student answers in the test text, and sequentially splice all the student answers to obtain a second idiom combination;
perform hash calculation on the second idiom combination based on the blockchain to obtain a corresponding verification hash value;
identify the identification code in the test text, and verify whether the verification hash value is consistent with the identification code; and
if yes, compare the student answers with the target answers one by one to obtain accuracy of the student answers.

### Claim 13

A non-transitory computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, a method for selecting answers to idiom fill-in-the-blank questions is implemented, and the method comprises the following steps:
obtaining a question text of idiom fill-in-the-blank questions, wherein the question text comprises a fill-in-the-blank text and n candidate idioms, and the fill-in-the-blank text comprises m fill-in-the-blanks to be filled in with the candidate idioms;
obtaining an explanatory text of all the candidate idioms;
inputting the fill-in-the-blank text and the explanatory text into a pre-trained idiom selection fill-in-the-blank model, to obtain a confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms;
selecting m idioms from the n candidate idioms, and randomly arranging the m idioms to fill in the m fill-in-the-blanks to form a plurality of groups of answers, wherein in each group of answers, the candidate idioms can be selected to fill in the fill-in-the-blanks only once at most;
calculating, based on a KM algorithm and the confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers; and
obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein before the step of inputting the fill-in-the-blank text and the explanatory text into a pre-trained idiom selection fill-in-the-blank model, to obtain a confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, the method comprises:
obtaining training samples, wherein the training samples comprise a plurality of corpus texts of idiom fill-in-the-blank questions with selected answers and an explanatory text of all idioms in an idiom library; and
inputting the training samples into a BERT language model-based natural language neural network for training to obtain the idiom selection fill-in-the-blank model, wherein the natural language neural network comprises a network output layer and a convolutional layer, the convolutional layer is formed by stacking a plurality of convolutional networks in sequence, an input layer of a latter convolutional network is connected to output layers of all former convolutional networks, an output layer of each of the convolutional networks is connected to an output layer of the BERT language model, and an output layer of the last convolutional network is connected to the network output layer.

### Claim 15 (depends on 14)

The non-transitory computer-readable storage medium according to claim 14, wherein the step of inputting the training samples into a BERT language model-based natural language neural network for training to obtain the idiom selection fill-in-the-blank model comprises:
inputting the training samples into the BERT language model, and extracting a feature matrix of the training samples based on the BERT language model; and
inputting the feature matrix into the natural language neural network for iterative training to obtain the idiom selection fill-in-the-blank model.

### Claim 16 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein before the step of calculating, based on a KM algorithm and the confidence that each of the fill-in-the-blanks is filled in with each of the candidate idioms, a sum of the confidences that the fill-in-the-blanks are filled in with the candidate idioms in each group of answers, the method comprises:
separately determining whether there is at least one candidate idiom in each group of answers for which the confidence that each of the fill-in-the-blanks is filled in with the at least one candidate idiom is less than a first threshold; and
if yes, eliminating corresponding answers, and skipping calculating a sum of the confidences that each of the fill-in-the-blanks is filled in with the at least one candidate idiom in the eliminated answers.

### Claim 17 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein after the step of obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions, the method further comprises:
filling in the fill-in-the-blank text with the target answers to obtain an answer text;
obtaining a score for the answer text given by a user, and determining whether the score is greater than a second threshold, wherein the user scores the answer text based on the standard answers;
if the score is greater than the second threshold, forming a training set from the answer text and an explanatory text of all candidate idioms of the target answers, and inputting the training set into the idiom selection fill-in-the-blank model for retraining; and
saving the retrained idiom selection fill-in-the-blank model to a blockchain.

### Claim 18 (depends on 13)

The non-transitory computer-readable storage medium according to claim 13, wherein the idiom fill-in-the-blank questions are used to test a student; and
after the step of obtaining a group of answers with the highest confidence sum as target answers, and outputting the target answers as answers to the idiom fill-in-the-blank questions, the method further comprises:
obtaining a test text uploaded by a terminal of the student, wherein the test text is a text in which the student fills in the fill-in-the-blank text of the idiom fill-in-the-blank questions with student answers; and when the terminal of the student uploads the test text, sequentially splicing the student answers with which the idiom fill-in-the-blank questions are filled in to obtain a first idiom combination, performing hash calculation on the first idiom combination based on the blockchain to obtain a corresponding hash value, and adding the hash value as an identification code to the test text;
extracting all student answers in the test text, and sequentially splicing all the student answers to obtain a second idiom combination;
performing hash calculation on the second idiom combination based on the blockchain to obtain a corresponding verification hash value;
identifying the identification code in the test text, and verifying whether the verification hash value is consistent with the identification code; and
if yes, comparing the student answers with the target answers one by one to obtain accuracy of the student answers.