US 9,754,208 B2Granted

Automatic rule coaching

AssigneeWal-Mart Stores, Inc.

InventorsJun Xie, Chong Sun, Fan Yang, Narasimhan Rampalli

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method of validating rules configured to be utilized in an information extraction application, including: receiving a plurality of labeled samples in a training database; for each of the rules in the rule database: (a) determining, for each of the data points of the plurality of labeled samples in the training database to which the rule applies, whether applying the rule to the data point has a positive or negative impact on matching an output for the data point based on the rule to the assured output of the labeled sample corresponding to the data point; (b) generating positive impact information for the rule based on the positive voters; (c) generating negative impact information for the rule based on the negative voters; and (d) determining a metric for the rule based on the quantity of the negative voters and the quantity of the positive voters; ranking the rules based on the metrics corresponding to the rules; and sending to a user for refinement one or more flagged rules of the rules that have a lowest ranking of the metric. Other embodiments are provided.

Claims20

Dependent claims

A method of validating rules configured to be utilized in an information extraction application, the rules being stored in a rules database, the method being implemented via execution of computer instructions configured to run at one or more processing modules and configured to be stored at one or more non-transitory memory storage modules, the method comprising:

receiving a plurality of labeled samples in a training database, each of the plurality of labeled samples comprising a different data point and an assured output, the assured output corresponding to the different data point for the information extraction application;
for each of the rules in the rules database:
determining, for each data point of the different data points of the plurality of labeled samples in the training database to which the rule applies, whether applying the rule to the data point has a positive impact on matching an output for the data point based on the rule to the assured output of the labeled sample corresponding to the data point, such that the data point is a positive voter, or whether applying the rule to the data point has a negative impact on matching the output for the data point based on the rule to the assured output of the labeled sample corresponding to the data point, such that the data point is a negative voter;
generating positive impact information for the rule based on the positive voters, wherein the positive impact information comprises a quantity of the positive voters;
generating negative impact information for the rule based on the negative voters, wherein the negative impact information comprises a quantity of the negative voters;
and
determining a metric for the rule based on the quantity of the negative voters and the quantity of the positive voters;
ranking the rules based on the metrics corresponding to the rules;
sending to a user for refinement one or more flagged rules of the rules that have a lowest ranking of the metric;
receiving from the user one or more refined rules;
generating a first output for a first data point in an information database based on the rules in the rules database, the rules in the rules database comprising the one or more refined rules, the plurality of labeled samples in the training database being devoid of the first data point;
receiving a request for information from a second user;
and
presenting the first output to the second user in response to the request.

11 more elements

A method of validating rules configured to be utilized in an information extraction application, the method being implemented via execution of computer instructions configured to run at one or more processing modules and configured to be stored at one or more non-transitory memory storage modules, the method comprising:

sending to a user a first data point for the information extraction application;
receiving from the user a first assured output corresponding to the first data point for the information extraction application based on human knowledge of the user;
storing the first data point and the first assured output as a first labeled sample in a training database, the training database comprising a plurality of labeled samples each for a different data point and an assured output, the assured output corresponding to the different data point for the information extraction application;
generating a first output for the first data point based on a first set of rules in a rules database comprising the rules configured to be utilized in the information extraction application;
sending to the user the first output for the first data point;
receiving from the user one of:
(1) a first new rule for the information extraction application of the first data point based on the human knowledge of the user, or (2) a first updated existing rule that is a modification of one of the first set of rules in the rules database, one of the first new rule or the first updated existing rule being a user-inputted rule;
storing the user-inputted rule in the rules database;
sending to the user an updated output for the first data point based on the rules in the rules database;
the rules in the rules database comprising the user-inputted rule and the first set of rules;
determining, for each data point of the different data points of the plurality of labeled samples in the training database to which the user-inputted rule applies, whether applying the user-inputted rule to the data point has a positive impact on matching an output for the data point based on the user-inputted rule to the assured output of the labeled sample corresponding to the data point, such that the data point is a positive voter, or whether applying the user-inputted rule to the data point has a negative impact on matching the output for the data point based on the user-inputted rule to the assured output of the labeled sample corresponding to the data point, such that the data point is a negative voter;
generating positive impact information for the user-inputted rule based on the positive voters;
generating negative impact information for the user-inputted rule based on the negative voters;
and
sending to the user the positive and negative impact information for the user-inputted rule.

12 more elements

A system for validating rules configured to be utilized in an information extraction application, the rules being stored in a rules database, the system comprising:

one or more processing modules;
and
one or more non-transitory memory storage modules storing computing instructions configured to run on the one or more processing modules and perform:
receiving a plurality of labeled samples in a training database, each of the plurality of labeled samples comprising a different data point and an assured output, the assured output corresponding to the different data point for the information extraction application;
for each of the rules in the rules database:
determining, for each data point of the different data points of the plurality of labeled samples in the training database to which the rule applies, whether applying the rule to the data point has a positive impact on matching an output for the data point based on the rule to the assured output of the labeled sample corresponding to the data point, such that the data point is a positive voter, or whether applying the rule to the data point has a negative impact on matching the output for the data point based on the rule to the assured output of the labeled sample corresponding to the data point, such that the data point is a negative voter;
generating positive impact information for the rule based on the positive voters, wherein the positive impact information comprises a quantity of the positive voters;
generating negative impact information for the rule based on the negative voters, wherein the negative impact information comprises a quantity of the negative voters;
and
determining a metric for the rule based on the quantity of the negative voters and the quantity of the positive voters;
ranking the rules based on the metrics corresponding to the rules;
sending to a user for refinement one or more flagged rules of the rules that have a lowest ranking of the metric;
receiving from the user one or more refined rules;
generating a first output for a first data point in an information database based on the rules in the rules database, the rules in the rules database comprising the one or more refined rules, the plurality of labeled samples in the training database being devoid of the first data point;
receiving a request for information from a second user;
and
presenting the first output to the second user in response to the request.

14 more elements

Dependencies are inferred from the claim text.

Facts

Application: 14/475,470Filed 2 Sep 2014
Published: US 2016/0063386 A13 Mar 2016
Granted: 5 Sep 2017
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06N 5/02
Computing arrangements using knowledge-based models: knowledge representation; Symbolic representation
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/2411
Relating to the classification model, e.g. parametric or non-parametric approaches: based on the proximity to a decision surface, e.g. support…
G06N 5/025
Knowledge engineering; Knowledge acquisition: extracting rules from data