# US 2021/0241042 A1 · Classifier evaluation device, classifier evaluation method, and non-transitory computer readable recording medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241042 A1 - Document ID: US20210241042A1 - Status: Published application - Kind: A1 - Application number: 17/268,546 - Filed: 2019-08-14 - Published: 2021-08-05 - Earliest claimed: 2018-08-15 - Assignees: Nippon Telegraph and Telephone Corporation - Applicants: Nippon Telegraph and Telephone Corporation - Inventors: Takaaki HASEGAWA; Yoshiaki NODA; Setsuo YAMADA - Web page: https://patentgrep.com/patent/US20210241042A1 ## CPC - G06N 20/00: Machine learning - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The disclosure allows quick and accurate confirmation of the degree to which a presently used classifier (model) conforms to data for which no ground truth exists. The classifier evaluation device (1) comprises: a data count obtainment unit (18) for obtaining a data count of input data to be made a classification target; a correction frequency counter (17) for counting a correction frequency of the classifiers, from correction information of classification results for the classifiers; and a correction rate calculation unit (19) for calculating, based on, the correction frequency and the data count of input data a correction rate for each of the classifiers. ## Claims ### Claim 1 A classifier evaluation device for evaluating classifiers performing classification of input data, the classifier evaluation device comprising: a computer that obtains a data count of input data to be made a classification target, counts a correction frequency of the classifiers, from correction information on classification results for the classifiers, and calculates, based on the correction frequency and the data count of input data, a correction rate for each of the classifiers. ### Claim 2 (depends on 1) The classifier evaluation device according to claim 1, wherein the computer counts the correction frequency made after an update date of the classifiers. ### Claim 3 (depends on 1) The classifier evaluation device according to claim 1, wherein the computer deletes the correction information each time the classifiers are updated. ### Claim 4 (depends on 1) The classifier evaluation device according to claim 1, wherein the computer counts a frequency of modifications and deletions, or the frequency of additions as the correction frequency. ### Claim 5 (depends on 1) The classifier evaluation device according to claim 1, wherein the computer issues a notification in a case in which the correction rate exceeds a preset threshold. ### Claim 6 (depends on 5) The classifier evaluation device according to claim 5, wherein the classifiers are based on models, and the computer replaces the model with a model trained with the correction information. ### Claim 7 (depends on 1) The classifier evaluation device according to claim 1, wherein the computer generates the correction information in a case in which the classification result is corrected via a correction interface. ### Claim 8 (depends on 7) The classifier evaluation device according to claim 7, wherein the correction interface includes a button for adding a classification result, a button for deleting a classification result, and a region for inputting a post-correction classification result. ### Claim 9 A classifier evaluation method for evaluating classifiers performing classification of input data, the method comprising: obtaining a data count of input data to be made a classification target; counting a correction frequency of the classifiers, from correction information on classification results for the classifiers; and calculating, based on the correction frequency and the data count of input data, a correction rate for each of the classifiers. ### Claim 10 (depends on 1) A non-transitory computer readable recording medium recording a program for causing a computer to function as a classifier evaluation device according to claim 1. ### Claim 11 (depends on 2) The classifier evaluation device according to claim 2, wherein the computer counts a frequency of modifications and deletions, or the frequency of additions as the rectification frequency. ### Claim 12 (depends on 3) The classifier evaluation device according to claim 3, wherein the computer counts a frequency of modifications and deletions, or the frequency of additions as the rectification frequency. ### Claim 13 (depends on 2) The classifier evaluation device according to claim 2, wherein the computer issues a notification in a case in which the correction rate exceeds a preset threshold. ### Claim 14 (depends on 3) The classifier evaluation device according to claim 3, wherein the computer issues a notification in a case in which the correction rate exceeds a preset threshold. ### Claim 15 (depends on 4) The classifier evaluation device according to claim 4, wherein the computer issues a notification in a case in which the correction rate exceeds a preset threshold. ### Claim 16 (depends on 2) The classifier evaluation device according to claim 2, wherein the computer generates the correction information in a case in which the classification result is corrected via a correction interface. ### Claim 17 (depends on 3) The classifier evaluation device according to claim 3, wherein the computer generates the correction information in a case in which the classification result is corrected via a correction interface. ### Claim 18 (depends on 4) The classifier evaluation device according to claim 4, wherein the computer generates the correction information in a case in which the classification result is corrected via a correction interface. ### Claim 19 (depends on 5) The classifier evaluation device according to claim 5, wherein the computer generates the correction information in a case in which the classification result is corrected via a correction interface. ### Claim 20 (depends on 6) The classifier evaluation device according to claim 6, wherein the computer generates the correction information in a case in which the classification result is corrected via a correction interface.