# US 2019/0171954 A1 · Inferencing and learning based on sensorimotor input data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171954 A1
- Document ID: US20190171954A1
- Status: Published application
- Kind: A1
- Application number: 16/268,148
- Filed: 2019-02-05
- Published: 2019-06-06
- Earliest claimed: 2016-05-13
- Assignees: Numenta, Inc.
- Applicants: Numenta, Inc.
- Inventors: Jeffrey C. Hawkins; Subutai Ahmad
- Web page: https://patentgrep.com/patent/US20190171954A1

## CPC

- G06V 10/82: using neural networks
- G06F 18/251: {of input or preprocessed data}
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N 5/046: Forward inferencing; Production systems
- G06V 10/803: {of input or preprocessed data}
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 10/955: {using specific electronic processors}
- G05B 2219/40053: Pick 3-D object from pile of objects
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

One or more multi-layer systems are used to perform inference. A multi-layer system may correspond to a node that receives a set of sensory input data for hierarchical processing, and may be grouped to perform processing for sensory input data. Inference systems at lower layers of a multi-layer system pass representation of objects to inference systems at higher layers. Each inference system can perform inference and form their own versions of representations of objects, regardless of the level and layer of the inference systems. The set of candidate objects for each inference system is updated to those consistent with feature-location representations for the sensors as well as object representations at lower layers. The set of candidate objects is also updated to those consistent with candidate objects from other inference systems, such as inference systems at other layers of the hierarchy or inference systems included in other multi-layer systems.

## Claims

### Claim 1

A computer-implemented method of performing inference using a plurality of inference systems, comprising:
generating, by a first input processor in a first inference system at a first layer of a multi-layer system, a first input representation by processing first sensory input indicating potential of a first pair of a first location and a first feature at the first location being detected;
determining, by a first output processor in the first inference system, a first output representation corresponding to the first input representation by processing the first output representation, the first output representation indicating a first set of candidates associated with the first pair;
generating, by a second input processor in a second inference system at a second layer of the multi-layer system higher than the first layer, a second input representation corresponding to at least the first output representation of the first output processor by processing the first output representation; and
determining, by a second output processor in the second inference system, a second output representation corresponding to the second input representation by processing the second input representation, the second output representation indicating a second set of candidates likely to be associated with at least the first pair.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the first sensory input is received from a first sensor, wherein the first input representation is generated by further processing the first sensory input, and wherein the second input representation is generated by further processing the first sensory input at the second input processor.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the subset of cells in the third inference system is activated by processing second sensory input received from a second sensor of a modality different from a modality of the first sensor.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating, by a third input processor in a third inference system at the first layer, a third input representation by processing second sensory input indicating potential of a second pair of a second feature and a second location corresponding to the second feature;
determining, by a third output processor in the third inference system, a third output representation corresponding to the third input representation, the third output representation indicating a third set of candidates likely to be associated with the second pair,
wherein the first output representation is an activation state of a subset of output cells in the first output processor and the third output representation is an activation state of a subset of output cells in the third output processor, and wherein the second input representation at the second layer is generated by processing at least the activation states of the subset of output cells in the first output representation and the activation states of the subset of output cells in the third output representation.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the first output representation is an activation state of a subset of output cells in the first output processor, and wherein the second output representation is an activation state of a subset of output cells in the second output processor.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
sending the activation states of the subset of the output cells in the first output processor to columns of input cells in the second input processor to generate the second input representation.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, further comprising:
activating a subset of cells in a third inference system; and
sending the activation states of the subset of the cells in the third output processor to one or more of the output cells of the first output processor or the second output processor, activation of the one or more of the output cells in the first output processor or the second output processor based further on the activation states of the subset of the cells in the third inference system.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the third inference system is included in another multi-layer system different from the multi-layer system.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the first output representation is determined by further processing activation states of output cells in a third inference system.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the third inference system is at a level other than the first level.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the second output representation is determined by further processing activation states of output cells in a third inference system at a level other than the second level.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the third inference system is at a level other than the second level.

### Claim 13

A non-transitory computer readable storage medium comprising instructions when executed by a processor cause the processor to instantiate a method of performing inference, comprising:
generating, by a first input processor in a first inference system at a first layer of a multi-layer system, a first input representation by processing first sensory input indicating potential of a first pair of a first location and a first feature at the first location being detected;
determining, by a first output processor in the first inference system, a first output representation corresponding to the first input representation by processing the first output representation, the first output representation indicating a first set of candidates associated with the first pair;
generating, by a second input processor in a second inference system at a second layer of the multi-layer system higher than the first layer, a second input representation corresponding to at least the first output representation of the first output processor by processing the first output representation; and
determining, by a second output processor in the second inference system, a second output representation corresponding to the second input representation by processing the second input representation, the second output representation indicating a second set of candidates likely to be associated with at least the first pair.

### Claim 14 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the first sensory input is received from a first sensor, wherein the first input representation is generated by further processing the first sensory input, and wherein the second input representation is generated by further processing the first sensory input at the second input processor.

### Claim 15 (depends on 14)

The non-transitory computer readable storage medium of claim 14, wherein the subset of cells in the third inference system is activated by processing second sensory input received from a second sensor of a modality different from a modality of the first sensor.

### Claim 16 (depends on 13)

The non-transitory computer readable storage medium of claim 13, further comprising:
generating, by a third input processor in a third inference system at the first layer, a third input representation by processing second sensory input indicating potential of a second pair of a second feature and a second location corresponding to the second feature;
determining, by a third output processor in the third inference system, a third output representation corresponding to the third input representation, the third output representation indicating a third set of candidates likely to be associated with the second pair,
wherein the first output representation is an activation state of a subset of output cells in the first output processor and the third output representation is an activation state of a subset of output cells in the third output processor, and wherein the second input representation at the second layer is generated by processing at least the activation states of the subset of output cells in the first output representation and the activation states of the subset of output cells in the third output representation.

### Claim 17 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the first output representation is an activation state of a subset of output cells in the first output processor, and wherein the second output representation is an activation state of a subset of output cells in the second output processor.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, further comprising:
sending the activation states of the subset of the output cells in the first output processor to columns of input cells in the second input processor to generate the second input representation.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium of claim 17, further comprising:
activating a subset of cells in a third inference system; and
sending the activation states of the subset of the cells in the third output processor to one or more of the output cells of the first output processor or the second output processor, activation of the one or more of the output cells in the first output processor or the second output processor based further on the activation states of the subset of the cells in the third inference system.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein the third inference system is included in another multi-layer system different from the multi-layer system.

### Claim 21 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the first output representation is determined by further processing activation states of output cells in a third inference system.

### Claim 22 (depends on 21)

The non-transitory computer readable storage medium of claim 21, wherein the third inference system is at a level other than the first level.

### Claim 23 (depends on 13)

The non-transitory computer readable storage medium of claim 13, wherein the second output representation is determined by further processing activation states of output cells in a third inference system at a level other than the second level.

### Claim 24 (depends on 23)

The non-transitory computer readable storage medium of claim 23, wherein the third inference system is at a level other than the second level.

### Claim 25

A computing device, comprising:
a first input circuit of a first inference system at a first layer of a multi-layer system, the first input circuit configured to generate a first input representation by processing first sensory input indicating potential of a first pair of a first location and a first feature at the first location being detected;
a first output circuit of the first inference system coupled to the first input circuit, the first output circuit configured to determine a first output representation corresponding to the first input representation by processing the first output representation, the first output representation indicating a first set of candidates associated with the first pair;
a second input circuit of a second inference system at a second layer of the multi-layer system higher than the first layer, the second input circuit configured to generate a second input representation corresponding to at least the first output representation of the first output processor by processing the first output representation; and
a second output circuit of the second inference system configured to determine a second output representation corresponding to the second input representation by processing the second input representation, the second output representation indicating a second set of candidates likely to be associated with at least the first pair.

### Claim 26 (depends on 25)

The computing device of claim 25, wherein the first sensory input is generated by a first sensor, wherein the first input circuit is further configured to generate the first input representation by further processing the first sensory input, and wherein the second input circuit is further configured to generate the second input representation by further processing the first sensory input at the second input processor.

### Claim 27 (depends on 25)

The computing device of claim 25, further comprising:
a third input circuit of a third inference system at the first layer, the third input circuit configured to generate a third input representation by processing second sensory input indicating potential of a second pair of a second feature and a second location corresponding to the second feature;
a third output circuit of the third inference system coupled to the third input circuit, the third output circuit configured to determine a third output representation corresponding to the third input representation, the third output representation indicating a third set of candidates likely to be associated with the second pair,
wherein the first output representation is an activation state of a subset of output cells in the first output circuit and the third output representation is an activation state of a subset of output cells in the third output circuit, and wherein second input representation at the second layer is generated by processing at least the activation states of the subset of output cells in the first output representation and the activation states of the subset of output cells in the third output representation.
