# US 2021/0241644 A1 · Apparatus, method and recording medium storing command for supporting learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241644 A1
- Document ID: US20210241644A1
- Status: Published application
- Kind: A1
- Application number: 16/780,086
- Filed: 2020-02-03
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: ST Unitas Co., Ltd.
- Applicants: ST Unitas Co., Ltd.
- Inventors: Sung Hyuk Yoon; Bon Jun Koo; Joo Young Yoon; Hwe Chul Cho; Se Won Cho
- Web page: https://patentgrep.com/patent/US20210241644A1

## CPC

- G09B 7/04: characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question or supplying a further explanation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06V 30/40: Document-oriented image-based pattern recognition
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/10: Character recognition
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus for supporting learning of a user is provided. The apparatus extracts, from a problem image, a first layout including textual information and a second layout including illustrative information, determines a first similarity between the first layout and textual information of a second problem stored, determines a second similarity between the second layout and illustrative information of the second problem, determines a third similarity between the first problem and the second problem by combining the first similarity and the second similarity, determines whether the second problem corresponds to the first problem, and transmits information indicating an answer or solution corresponding to the second problem to the user device.

## Claims

### Claim 1

An apparatus comprising:
a transceiver configured to receive a problem image of a first problem from an user device;
one or more processors; and
one or more memories configured to store commands that cause the one or more processors to perform an operation when the commands are executed by the one or more processors, and information related to a plurality of problems,
wherein the one or more processors are configured to:
extract, from the problem image, a first layout including an area of the problem image in which textual information of the first problem is located and a second layout including an area of the problem image in which illustrative information of the first problem is located;
determine a first similarity between the first layout and textual information of a second problem stored in the one or more memories;
determine a second similarity between the second layout and illustrative information of the second problem;
determine a third similarity between the first problem and the second problem by combining the first similarity and the second similarity;
determine whether the second problem corresponds to the first problem based on whether the third similarity is larger than or equal to a predetermined reference similarity; and
upon determining that the second problem corresponds to the first problem, control the transceiver to transmit information indicating an answer or solution corresponding to the second problem to the user device.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are further configured to:
input the first layout to a first neural network model that is trained to derive a first vector representation from an image having textual information;
obtain the first vector representation of the first layout from the first neural network model; and
determine the first similarity by comparing the first vector representation of the first layout with a previously stored vector representation of the textual information of the second problem.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the one or more processors are further configured to:
pre-process the first layout by replacing a proper noun or a constant of the textual information of the first layout with a placeholder, before inputting the first layout to the first neural network model.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein the one or more processors are further configured to:
input the second layout to a second neural network model that is trained to derive a second vector representation from an image having illustrative information;
obtain the second vector representation of the second layout from the second neural network model; and
determine the second similarity by comparing the second vector representation of the second layout with a previously stored vector representation of the illustrative information of the second problem.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are further configured to:
determine a first combination factor to be applied to the first similarity and a second combination factor to be applied to the second similarity based on predetermined criteria; and
determine the third similarity by combining the first similarity to which the first combination factor is applied and the second similarity to which the second combination factor is applied.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the predetermined criteria are that:
as an amount of the textual information of the first layout compared to a size of the first layout decreases, the first combination factor decreases,
as the number of problems having a certain similarity or more with the first layout stored in the one or more memories decreases, the first combination factor decreases,
as an amount of the illustrative information of the second layout compared to a size of the second layout decreases, the second combination factor decreases, and
as the number of problems having a certain similarity or more with the second layout stored in the one or more memories decreases, the second combination factor decreases.

### Claim 7 (depends on 4)

The apparatus of claim 4, wherein the one or more processors are further configured to:
extract, from the problem image, an area of the problem image in which each piece of information of the first problem is located as one or more layouts; and
determine each of the one or more layouts as the first layout or the second layout by inputting the one or more layouts to a third neural network model that is trained to distinguish between textual information and illustrative information.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein at least one of the first neural network model, the second neural network model, and the third neural network model is stored in a server, and
wherein the one or more processors control the transceiver to communicate with the server.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are further configured to:
store the problem image of the first problem in the one or more memories upon determining that no problem corresponds to the first problem among the plurality of problems.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the textual information of the first problem includes text or a mathematical expression of the first problem, and
wherein the illustrative information of the first problem includes a drawing, a picture, a table, or a graph of the first problem.

### Claim 11

A method performed by an apparatus including a transceiver communicating with a user device, one or more processors, and one or more memories storing commands that cause the one or more processors to perform an operation when the commands are executed by the one or more processors, and information related to a plurality of problems, the method comprising:
receiving, by the transceiver, a problem image of a first problem from the user device;
extracting, from the problem image by the one or more processors, a first layout including an area of the problem image in which textual information of the first problem is located and a second layout including an area of the problem image in which illustrative information of the first problem is located;
determining, by the one or more processors, a first similarity between the first layout and textual information of a second problem stored in the one or more memories;
determining, by the one or more processors, a second similarity between the second layout and illustrative information of the second problem;
determining, by the one or more processors, a third similarity between the first problem and the second problem by combining the first similarity and the second similarity;
determining, by the one or more processors, whether the second problem corresponds to the first problem based on whether the third similarity is larger than or equal to a predetermined reference similarity; and
upon determining that the second problem corresponds to the first problem, transmitting, by the transceiver, information indicating an answer or solution corresponding to the second problem to the user device.

### Claim 12 (depends on 11)

The method of claim 11, wherein determining the first similarity includes:
inputting the first layout to a first neural network model that is trained to derive a first vector representation from an image having textual information;
obtaining the first vector representation of the first layout from the first neural network model; and
determining the first similarity by comparing the first vector representation of the first layout with a previously stored vector representation of the textual information of the second problem.

### Claim 13 (depends on 12)

The method of claim 12, wherein determining the first similarity includes,
pre-processing the first layout by replacing a proper noun or a constant of the textual information of the first layout with a placeholder, before inputting the first layout to the first neural network model.

### Claim 14 (depends on 12)

The method of claim 12, wherein determining the second similarity includes:
inputting the second layout to a second neural network model that is trained to derive a second vector representation from an image having illustrative information;
obtaining the second vector representation of the second layout from the second neural network model; and
determining the second similarity by comparing the second vector representation of the second layout with a previously stored vector representation of the illustrative information of the second problem.

### Claim 15 (depends on 11)

The method of claim 11, wherein determining the third similarity includes:
determining a first combination factor to be applied to the first similarity and a second combination factor to be applied to the second similarity based on predetermined criteria; and
determining the third similarity by combining the first similarity to which the first combination factor is applied and the second similarity to which the second combination factor is applied.

### Claim 16 (depends on 15)

The method of claim 15, wherein the predetermined criteria are that:
as an amount of the textual information of the first layout compared to a size of the first layout decreases, the first combination factor decreases,
as the number of problems having a certain similarity or more with the first layout stored in the one or more memories decreases, the first combination factor decreases,
as an amount of the illustrative information of the second layout compared to a size of the second layout decreases, the second combination factor decreases, and
as the number of problems having a certain similarity or more with the second layout stored in the one or more memories decreases, the second combination factor decreases.

### Claim 17 (depends on 14)

The method of claim 14, wherein extracting the first layout and the second layout includes:
extracting, from the problem image, an area of the problem image in which each piece of information of the first problem is located as one or more layouts; and
determining each of the one or more layouts as the first layout or the second layout by inputting the one or more layouts to a third neural network model that is trained to distinguish between textual information and illustrative information.

### Claim 18 (depends on 11)

The method of claim 11, wherein the method further comprises:
storing, by the one or more processors, the problem image of the first problem in the one or more memories upon determining that no problem corresponds to the first problem among the plurality of problems.

### Claim 19 (depends on 11)

The method of claim 11, wherein the textual information of the first problem includes text or a mathematical expression of the first problem, and
wherein the illustrative information of the first problem includes a drawing, a picture, a table or a graph of the first problem.

### Claim 20

A non-transitory computer-readable recording medium storing commands that cause one or more processors to perform an operation when the commands are executed by the one or more processors, the commands comprising:
extracting, from a problem image of a first problem, a first layout including an area of the problem image in which textual information of the first problem is located and a second layout including an area of the problem image in which illustrative information of the first problem is located;
determining a first similarity between the first layout and textual information of a second problem stored in one or more memories;
determining a second similarity between the second layout and illustrative information of the second problem;
determining a third similarity between the first problem and the second problem by combining the first similarity and the second similarity;
determining whether the second problem corresponds to the first problem based on whether the third similarity is larger than or equal to a predetermined reference similarity; and
upon determining that the second problem corresponds to the first problem, controlling a transceiver to transmit information indicating an answer or solution corresponding to the second problem to an user device.