# US 2014/0156569 A1 · Method and apparatus for improving resilience in customized program learning network computational environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0156569 A1
- Document ID: US20140156569A1
- Status: Published application
- Kind: A1
- Application number: 13/921,124
- Filed: 2013-06-18
- Published: 2014-06-05
- Earliest claimed: 2012-06-19
- Assignees: Georges Harik
- Inventors: Georges Harik
- Web page: https://patentgrep.com/patent/US20140156569A1

## CPC

- G06N 20/00: Machine learning
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An apparatus and a method are provided for learning a program with a large number of parameters. In one embodiment, a method not only distorts the input values, but also distorts some of the parameters in the program model. Such an approach not only forces the learned program to acquire parameter values to predict missing or desired data, but also to correct errors in the input data and the program parameters themselves, thereby rendering the learned program more resilient to overfitting and falling into local optima.

## Claims

### Claim 1

In a computational environment, a method for learning a program that is characterized by a number of parameters, the computing environment allowing the program to receive an input vector, an expected output vector and values of the parameters, the method comprising of:
choosing an input distortion percentage and a parameter distortion percentage each having a value between zero and a respective predetermined maximum and;
blanking out or randomizing the input distortion percentage of values in the input vector;
blanking out or randomizing the parameter distortion percentage of the values of the parameters of the program; and
applying a learning technique to the program, using the expected output vector, the distorted input vector and distorted parameters.

### Claim 2 (depends on 1)

The method of claim 1, wherein the expected output vector is generated by a training algorithm.

### Claim 3 (depends on 1)

The method of claim 1 wherein, in which the blanking out or randomizing steps, the input values or parameter values are randomized based on one or more predetermined random distributions.

### Claim 4 (depends on 1)

The method of claim 1, wherein the learning technique applied comprises the step of minimizing a loss function computed over a sample of the input.

### Claim 5 (depends on 4)

The method of claim 4, wherein the step of minimizing the loss function is carried out using gradient descent, or second order minimization methods such as newton or quasi-newton methods over the loss function.

### Claim 6 (depends on 1)

The method of claim 1 wherein the program uses a control flow determined by a structure or value of the input vector or the state values in the program.

### Claim 7 (depends on 6)

The method of claim 6, wherein the program comprises linear transformations among values in the input vector, values of the parameters, and state values of the program.

### Claim 8 (depends on 6)

The method of claim 6, wherein the program uses one or more nonlinear functions among values in the input vector, values of parameters, and state values of the program.

### Claim 9 (depends on 6)

The method of claim 6, wherein the nonlinear functions include a sigmoid function and a rectified linear function.

### Claim 10 (depends on 1)

The method of claim 1, wherein the computational environment includes a plurality of computational resources each assigned to process a fraction of the input set.

### Claim 11

An apparatus for learning a program that is characterized by a number of parameters, the apparatus receiving an input vector, an expected output vector, and values of the parameters, the apparatus comprising:
an input randomizing module;
a parameter randomizing module;
a parameter updater and
an expected output module, wherein the apparatus carries out a method comprising:
choosing an input distortion percentage and a parameter distortion percentage each having a value between zero and a respective predetermined maximum and;
in the input randomizing module, blanking out or randomizing the input distortion percentage of values in the input vector;
in the parameter randomizing module, blanking out or randomizing the parameter distortion percentage of the values of the parameters of the program; and
applying a learning technique to the program, using the expected output vector, the distorted input vector and distorted parameters.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the expected output vector is generated by a training algorithm.

### Claim 13 (depends on 11)

The apparatus of claim 11 wherein, in the blanking out or randomizing steps, the input values or parameter values are randomized based on one or more predetermined random distributions.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein the learning technique applied comprises the step of minimizing a loss function computed over a sample of the input.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the step of minimizing the loss function is carried out using gradient descent over the loss function.

### Claim 16 (depends on 11)

The apparatus of claim 11 wherein the program uses a control flow determined by a structure or value of the input vector or the state values in the program.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the program comprises linear transformations among values in the input vector, values of the parameters, and state values of the program.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the program uses one or more nonlinear functions among values in the input vector, values of parameters, and state values of the program.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the nonlinear functions include a sigmoid function and a rectified linear function.

### Claim 20 (depends on 11)

The apparatus of claim 11, wherein the computational environment includes a plurality of computational resources each assigned to process a fraction of the input set.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein the computational resources comprise graphics processors.