# US 2021/0342683 A1 · Companion analysis network in deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342683 A1
- Document ID: US20210342683A1
- Status: Published application
- Kind: A1
- Application number: 16/620,177
- Filed: 2019-08-08
- Published: 2021-11-04
- Earliest claimed: 2018-08-09
- Assignees: D5Ai LLC
- Applicants: D5Ai LLC
- Inventors: James K. BAKER
- Web page: https://patentgrep.com/patent/US20210342683A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0499: Feedforward networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 20/20: Ensemble learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for analyzing a first machine learning system via a second machine learning system. The first machine learning system comprising a first objective function. The method includes connecting the first machine learning system to an input of the second machine learning system, which includes a second objective function for analyzing an internal characteristic of the first machine learning system. The method further includes providing a data item to the first machine learning system, collecting internal characteristic data from the first machine learning system associated with the internal characteristic, computing partial derivatives of the first objective function through the first machine learning system with respect to the data item, and computing partial derivatives of the second objective function through both the second machine learning system and the first machine learning system with respect to the collected internal characteristic data.

## Claims

### Claim 1

A computer-implemented method for analyzing a first machine learning system via a second machine learning system, the first machine learning system comprising a first objective function, the method comprising:
connecting, by a computer system, the first machine learning system to an input of the second machine learning system;
wherein the second machine learning system comprises a second objective function for analyzing an internal characteristic of the first machine learning system;
providing, by the computer system, a data item to the first machine learning system;
collecting, by the computer system, internal characteristic data from the first machine learning system associated with the internal characteristic;
computing, by the computer system, partial derivatives of the first objective function through the first machine learning system with respect to the data item; and
computing, by the computer system, partial derivatives of the second objective function through both the second machine learning system and the first machine learning system with respect to the collected internal characteristic data.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein connecting the first machine learning system to the second machine learning system comprises:
adding, by the computer system, an additional output node to the first machine learning system; and
connecting, by the computer system, the additional output node to an input of the second machine learning system.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein connecting the first machine learning system to the second machine learning system comprises:
connecting, by the computer system, an internal element of the first machine learning system to an input of the second machine learning system.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the first machine learning system comprises a neural network and the internal element comprises a node of the neural network.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the internal characteristic data comprises a latent variable.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein:
the first machine learning system comprises a neural network comprising an inner node; and
the latent variable comprises an activation value of the inner node.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the internal characteristic data comprises a derivative calculated with respect to a parameter of the first machine learning system.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein:
the first machine learning system comprises a neural network comprising a plurality of output nodes; and
the derivative comprises a partial derivative of a difference between a largest activation and a second largest activation of the plurality of output nodes for the data item.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein:
the first machine learning system comprises a neural network comprising an inner node; and
the derivative comprises a partial derivative of the first objective function with respect to an activation value of the inner node.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, wherein:
the first machine learning system comprises a neural network comprising a node; and
the derivative comprises a partial derivative of the first objective function with respect to an input to the node.

### Claim 11 (depends on 7)

The computer-implemented method of claim 7, wherein:
the first machine learning system comprises a neural network;
the neural network comprises a first node, a second node, and an arc connecting the first node and the second node; and
the derivative comprises a partial derivative of the first objective function with respect to a connection weight of the arc.

### Claim 12 (depends on 7)

The computer-implemented method of claim 7, wherein:
the first machine learning system is one of an ensemble of machine learning systems; and
the derivative comprises a partial derivative of a difference between an output of the first machine learning system and a correct output of the ensemble of machine learning systems.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the internal characteristic data comprises a satisfaction score sat(n) defined as:
sat(n)=sign(A(n)−0.5)*sign(−δ(n)),
where A(n) is an output activation of a node n of the first machine learning system for the data item.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the internal characteristic data comprises a comparison between a first value of a latent variable for a current training iteration and a second value of the latent variable for a prior training iteration of the first machine learning system.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein the internal characteristic data comprises a comparison between a first derivative calculated with respect to a parameter of the first machine learning system for a current training iteration and a second derivative calculated with respect to the parameter of the first machine learning system for a prior training iteration of the first machine learning system.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein the first machine learning system comprises a first neural network and the second machine learning system comprises a second neural network.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein computing partial derivatives of the first objective function and computing partial derivatives of the second objective function each comprise performing a back propagation calculation.

### Claim 18 (depends on 1)

The computer-implemented method of claim 1, wherein computing partial derivatives of the first objective function and computing partial derivatives of the second objective function each comprise performing a numerical estimation calculation.

### Claim 19 (depends on 1)

The computer-implemented method of claim 1, further comprising updating, by the computer system, learned parameters of the first machine learning system.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, wherein the internal characteristic is selected to cause the second machine learning system to alter an output of the first machine learning system when updating the learned parameters of the first machine learning system.

### Claim 21

A computer system for analyzing a first machine learning system via a second machine learning system, the computer system comprising:
a processor; and
a memory coupled to the processor, the memory storing:
the first machine learning system comprising a first objective function;
the second machine learning system comprising a second objective function; and
instructions that, when executed by the processor, cause the computer system to:
connect the first machine learning system to an input of the second machine learning system;
wherein the second machine learning system comprises a second objective function for analyzing an internal characteristic of the first machine learning system;
provide a data item to the first machine learning system;
collect internal characteristic data from the first machine learning system associated with the internal characteristic;
compute partial derivatives of the first objective function through the first machine learning system with respect to the data item; and
compute partial derivatives of the second objective function through both the second machine learning system and the first machine learning system with respect to the collected internal characteristic data.

### Claim 22 (depends on 21)

The computer system of claim 21, wherein the instructions cause the computer system to connect the first machine learning system to the second machine learning system by causing the computer system to:
add an additional output node to the first machine learning system; and
connect the additional output node to an input of the second machine learning system.

### Claim 23 (depends on 21)

The computer system of claim 21, wherein the instructions cause the computer system to connect the first machine learning system to the second machine learning system by causing the computer system to:
connect an internal element of the first machine learning system to an input of the second machine learning system.

### Claim 24 (depends on 23)

The computer system of claim 23, wherein the first machine learning system comprises a neural network and the internal element comprises a node of the neural network.

### Claim 25 (depends on 21)

The computer system of claim 21, wherein the internal characteristic data comprises a latent variable.

### Claim 26 (depends on 25)

The computer system of claim 25, wherein:
the first machine learning system comprises a neural network comprising an inner node; and
the latent variable comprises an activation value of the inner node.

### Claim 27 (depends on 21)

The computer system of claim 21, wherein the internal characteristic data comprises a derivative calculated with respect to a parameter of the first machine learning system.

### Claim 28 (depends on 27)

The computer system of claim 27, wherein:
the first machine learning system comprises a neural network comprising a plurality of output nodes; and
the derivative comprises a partial derivative of a difference between a largest activation and a second largest activation of the plurality of output nodes for the data item.

### Claim 29 (depends on 27)

The computer system of claim 27, wherein:
the first machine learning system comprises a neural network comprising an inner node; and
the derivative comprises a partial derivative of the first objective function with respect to an activation value of the inner node.

### Claim 30 (depends on 27)

The computer system of claim 27, wherein:
the first machine learning system comprises a neural network comprising a node; and
the derivative comprises a partial derivative of the first objective function with respect to an input to the node.

### Claim 31 (depends on 27)

The computer system of claim 27, wherein:
the first machine learning system comprises a neural network;
the neural network comprises a first node, a second node, and an arc connecting the first node and the second node; and
the derivative comprises a partial derivative of the first objective function with respect to a connection weight of the arc.

### Claim 32 (depends on 27)

The computer system of claim 27, wherein:
the first machine learning system is one of an ensemble of machine learning systems; and
the derivative comprises a partial derivative of a difference between an output of the first machine learning system and a correct output of the ensemble of machine learning systems.

### Claim 33 (depends on 21)

The computer system of claim 21, wherein the internal characteristic data comprises a satisfaction score sat(n) defined as:
sat(n)=sign(A(n)−0.5)*sign(−δ(n)),
where A(n) is an output activation of a node n of the first machine learning system for the data item.

### Claim 34 (depends on 21)

The computer system of claim 21, wherein the internal characteristic data comprises a comparison between a first value of a latent variable for a current training iteration and a second value of the latent variable for a prior training iteration of the first machine learning system.

### Claim 35 (depends on 21)

The computer system of claim 21, wherein the internal characteristic data comprises a comparison between a first derivative calculated with respect to a parameter of the first machine learning system for a current training iteration and a second derivative calculated with respect to the parameter of the first machine learning system for a prior training iteration of the first machine learning system.

### Claim 36 (depends on 21)

The computer system of claim 21, wherein the first machine learning system comprises a first neural network and the second machine learning system comprises a second neural network.

### Claim 37 (depends on 36)

The computer system of claim 36, wherein the instructions cause the computer system to compute partial derivatives of the first objective function and compute partial derivatives of the second objective function via a back propagation calculation.

### Claim 38 (depends on 21)

The computer system of claim 21, wherein the instructions cause the computer system to compute partial derivatives of the first objective function and compute partial derivatives of the second objective function via a numerical estimation calculation.

### Claim 39 (depends on 21)

The computer system of claim 21, further the instructions, when executed by the processor, further cause the computer system to update learned parameters of the first machine learning system.

### Claim 40 (depends on 39)

The computer system of claim 39, wherein the internal characteristic is selected to cause the second machine learning system to alter an output of the first machine learning system when updating the learned parameters of the first machine learning system.
