US 10,013,654 B1Granted

Cooperatively operating a network of supervised learning processors to concurrently distribute supervised learning processor training and provide predictive responses to input data

AssigneeOJO Labs, Inc.

InventorsJoshua Howard Levy, Jacy Myles Legault, David Robert Rubin, John Kenneth Berkowitz, David Ross Pratt

Life of this patent

Estimating the term…

Loading patent…

Abstract

A supervised learning processing (SLP) system and non-transitory, computer program product provides cooperative operation of a network of supervised learning processors to concurrently distribute supervised learning processor training, generate predictions, and provide prediction driven responses to input objects, such as NL statements. The SLP system includes SLP stages that are distributed across multiple SLP subsystems. Concurrently training SLP's provides accurate predictions of input objects and responses thereto, the SLP system and non-transitory, computer program product enhance the network by providing high quality value predictions and responses and avoiding potential training and operational delays. The SLP system can enhance the network of SLP subsystems by providing flexibility to incorporate multiple SLP models into the network and train at least a proper subset of the SLP models while concurrently using the SLP system and non-transitory, computer program product in commercial operation.

Claims30

Dependent claims

A method of cooperatively operating a network of supervised learning processor subsystems to concurrently distribute supervised learning processor training and provide predictive responses to input objects, the method comprising:

in an electronic, supervised learning processing system:
receiving an input object from a requestor device;
storing the input object in a memory;
(A) receiving and accessing contextual data associated with the input object to contextualize the input object;
(B) associating the contextual data with the input object;
(C) supervised learning processing the input object and contextual data using a plurality of supervised learning processing stages distributed between at least a first order supervised learning processor subsystem and second order supervised learning processor subsystem having an equal or higher, predictive output quality value than the first supervised learning processor subsystem, wherein:
(i) each supervised learning processing stage comprises one or more supervised learning processor models,
(ii) each model of the first supervised learning processor subsystem (a) processes at least the input object and contextual data to determine a predictive response to the input object and (b) determines a quality value of the predictive response;
and
(iii) the second order supervised learning processing subsystem determines and finalizes an output of each supervised learning processing stage for each predictive response whose quality value is below a pre-determined threshold;
(D) training at least one model of the first order supervised learning processor subsystem with the determined and finalized output from the second order supervised learning processor subsystem;
(E) determining a response to the input object using at least a combined output from each supervised learning processing stage, wherein enhancement of the network of supervised learning processor subsystems includes concurrently distributing training and cooperatively operating the supervised learning processor subsystems to determine the response to the statement;
and
(F) providing the response to the input object to the user of the requestor device.

11 more elements

An apparatus comprising a cooperatively operated network of supervised learning processor subsystems to concurrently distribute supervised learning processor training and provide predictive responses to input objects, the apparatus comprising:

one or more data processors;
a memory, coupled to the data processors, having code stored therein to cause the one or more data processors to:
receive an input object from a requestor device;
store the input object in the memory;
(A) receive and access contextual data associated with the input object to contextualize the input object;
(B) associate the contextual data with the input object;
(C) supervised learning process the input object and contextual data using a plurality of supervised learning processing stages distributed between at least a first order supervised learning processor subsystem and second order supervised learning processor subsystem having an equal or higher, predictive output quality value than the first supervised learning processor subsystem, wherein:
(i) each supervised learning processing stage comprises one or more supervised learning processor models,
(ii) each model of the first supervised learning processor subsystem (a) processes at least the input object and contextual data to determine a predictive response to the input object and (b) determines a quality value of the predictive response;
and
(iii) the second order supervised learning processing subsystem determines and finalizes an output of each supervised learning processing stage for each predictive response whose quality value is below a pre-determined threshold;
(D) train at least one model of the first order supervised learning processor subsystem with the determined and finalized output from the second order supervised learning processor subsystem;
(E) determine a response to the input object using at least a combined output from each supervised learning processing stage, wherein enhancement of the network of supervised learning processor subsystems includes concurrently distributing training and cooperatively operating the supervised learning processor subsystems to determine the response to the statement;
and
(F) provide the response to the input object to the user of the requestor device.

12 more elements

A non-transitory, computer program product comprising code stored therein and executable by a cooperatively operated a network of supervised learning processor subsystems to concurrently distribute supervised learning processor training and provide predictive responses to input objects by:

receiving an input object from a requestor device;
storing the input object in a memory;
(A) receiving and accessing contextual data associated with the input object to contextualize the input object;
(B) associating the contextual data with the input object;
(C) supervised learning processing the input object and contextual data using a plurality of supervised learning processing stages distributed between at least a first order supervised learning processor subsystem and second order supervised learning processor subsystem having an equal or higher, predictive output quality value than the first supervised learning processor subsystem, wherein:
(i) each supervised learning processing stage comprises one or more supervised learning processor models,
(ii) each model of the first supervised learning processor subsystem (a) processes at least the input object and contextual data to determine a predictive response to the input object and (b) determines a quality value of the predictive response;
and
(iii) the second order supervised learning processing subsystem determines and finalizes an output of each supervised learning processing stage for each predictive response whose quality value is below a pre-determined threshold;
(D) training at least one model of the first order supervised learning processor subsystem with the determined and finalized output from the second order supervised learning processor subsystem;
(E) determining a response to the input object using at least a combined output from each supervised learning processing stage, wherein enhancement of the network of supervised learning processor subsystems includes concurrently distributing training and cooperatively operating the supervised learning processor subsystems to determine the response to the statement;
and
(F) providing the response to the input object to the user of the requestor device.

10 more elements

A non-transitory, computer program product of distributing training and concurrent operations within a network of computer systems:

in an electronic, first natural language processing system:
receiving a natural language input statement from a user of a requestor device;
storing the statement in a memory;
receiving and accessing contextual data associated with the user to contextualize the statement;
associating the contextual data with the statement;
processing the input statement and associated contextual data with a domain classifier model to determine one or more domains of the statement and determine a quality value for each determined domain;
determine if the highest domain quality value exceeds a predetermined domain threshold;
and
if the highest quality value does not exceed the predetermined domain threshold, transmitting the statement, contextual data, and each determined domain and quality value to a remotely located, second natural language processing system, wherein the second natural language processing system has access to a higher level natural language processor than the first natural language processor;
in the second natural language processing system:
reviewing the statement, each determined domain, and associated contextual data;
determining a domain of the statement;
and
transmitting the determined domain to the first natural language processing system;
in the first natural language processing system:
receiving the determined domain from the second natural language processing system;
training the domain classifier model with the received determined domain;
processing the input statement, associated contextual data, and determined domain with an intent classifier model to determine an intent of the statement and determine a quality value for the determined intent;
determine if the intent quality value exceeds a predetermined intent threshold;
if the quality value does not exceed the predetermined intent threshold, transmitting the statement, contextual data, the domain, and the determined intent to the second natural language processing system;
in the second natural language processing system:
reviewing the statement, the determined domain, the determined intent, and associated contextual data;
determining an intent of the statement;
and
transmitting the determined intent to the first natural language processing system;
in the first natural language processing system:
receiving the determined intent from the second natural language processing system;
training the intent classifier model with the received determined intent;
processing the input statement, associated contextual data, and determined domain and intent with an entities extraction model to extract language entities from the statement and determine an entities quality value for each determined entity;
determine if the highest entities quality value exceeds a predetermined entities threshold;
if the highest quality value does not exceed the predetermined entities threshold, transmitting the statement, contextual data, and the determined domain, intent, and entities to the second natural language processing system;
in the second natural language processing system:
reviewing the statement, each determined domain and intent, and associated contextual data;
determining entities of the statement;
and
transmitting the determined entities to the first natural language processing system;
in the first natural language processing system:
receiving the determined entities from the second natural language processing system;
training the entities extraction model with the received determined entities;
processing the input statement, associated contextual data, and determined domain, intent, and entities with a normalize entities model to normalize the entities of the statement and determine a normalized entities quality value for the normalized entities;
determine if the highest normalized entities quality value exceeds a predetermined normalized entities threshold;
if the highest quality value does not exceed the predetermined normalized entities threshold, transmitting the statement, contextual data, and the determined domain, intent, entities, and normalized entities to the second natural language processing system;
in the second natural language processing system:
reviewing the statement, the determined domain, intent, and entities and associated contextual data;
determining normalized entities of the statement;
and
transmitting the determined normalized entities to the first natural language processing system;
in the first natural language processing system:
training the normalized entities model with the received normalized entities;
processing the domain, intent, entities, and normalized entities with a fulfillment model and a response model to determine a response to the input statement;
and
transmitting the response to the requestor device.

49 more elements

Dependencies are inferred from the claim text.