# US 2022/0032932 A1 · Image sensor for processing sensor data to reduce data traffic to host system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0032932 A1
- Document ID: US20220032932A1
- Status: Published application
- Kind: A1
- Application number: 16/942,438
- Filed: 2020-07-29
- Published: 2022-02-03
- Earliest claimed: 2020-07-29
- Assignees: Micron Technology, Inc.
- Applicants: Micron Technology, Inc.
- Inventors: Poorna Kale; Amit Gattani
- Web page: https://patentgrep.com/patent/US20220032932A1

## CPC

- B60W 50/045: {Monitoring control system parameters}
- G06N 3/063: using electronic means
- G06F 13/4282: {on a serial bus, e.g. I2C bus, SPI bus (on daisy chain buses G06F13/4247)}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/24: Classification techniques
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher}
- G06V 10/7792: {the supervisor being an automated module, e.g. "intelligent oracle"}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2510/18: Braking system
- B60W 2510/20: Steering systems
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems, methods and apparatus of integrated image sensing devices. In one example, a system includes a sensor that generates data. A memory device stores the generated data, and further stores a first portion of an artificial neural network (ANN). A host interface of the system is configured to communicate with a host system that stores a second portion of the ANN. The memory device can be stacked with the sensor. The memory device includes an inference engine configured to generate inference results using the stored data as input to the first portion of the ANN. The host interface is further configured to send the inference results to the host system for processing by the host system using the second portion of the ANN.

## Claims

### Claim 1

A system comprising:
a first sensor configured to generate first data;
a memory device configured to store the generated first data, and a first portion of an artificial neural network (ANN); and
a host interface configured to communicate with a host system that stores a second portion of the ANN;
wherein the memory device is stacked with the first sensor, and the memory device comprises an inference engine configured to generate inference results using the stored first data as input to the first portion of the ANN; and
wherein the host interface is further configured to send the inference results to the host system for processing by the host system using the second portion of the ANN.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a camera that includes the first sensor, wherein the first data is an image stream including frames.

### Claim 3 (depends on 2)

The system of claim 2, wherein:
the first data is generated by the first sensor at a rate that exceeds a communication bandwidth between the host interface and the host system; and
the host system accesses the inference results by sending commands to the host interface.

### Claim 4 (depends on 1)

The system of claim 1, wherein:
the inference engine is a first inference engine and the generated inference results are first inference results;
the system further comprises a controller configured to send the first inference results to the host system;
the host system comprises a second inference engine configured to generate second inference results from the first inference results; and
the host system sends a write command executed by the controller to store the second inference results in the memory device.

### Claim 5 (depends on 4)

The system of claim 4, wherein:
the memory device further stores the first inference results; and
the controller is further configured to execute read commands received from the host system via the host interface, the read commands to access the stored first inference results.

### Claim 6 (depends on 1)

The system of claim 1, wherein the host system sends the first portion to the host interface in response to evaluating at least one of prior inference results received from the host interface, or image data received from the host interface.

### Claim 7 (depends on 1)

The system of claim 1, wherein the first data is in other than an image form, and the system further comprises a converter configured to convert the first data to an image form to provide as the input to the first portion of the ANN.

### Claim 8 (depends on 1)

The system of claim 1, wherein the first sensor is an image sensor, the first data is first image data, and the system further comprises:
a first sensing device that includes the image sensor and the memory device; and
a second sensing device configured to send second image data to the first sensing device, wherein the second sensing device includes a second sensor configured to generate sensor data in other than an image form, and a controller configured to convert the sensor data to the second image data.

### Claim 9 (depends on 8)

The system of claim 8, wherein the sensor data comprises sound data generated by at least one microphone.

### Claim 10 (depends on 8)

The system of claim 8, wherein the second sensor is a lidar sensor, a radar sensor, or a sonar sensor.

### Claim 11 (depends on 8)

The system of claim 8, wherein the inference engine uses the second image data as additional input to generate the inference results.

### Claim 12 (depends on 1)

The system of claim 1, further comprising a package that encapsulates the first sensor and the memory device.

### Claim 13 (depends on 1)

The system of claim 1, wherein:
the first sensor is formed on a first integrated circuit die;
the memory device is formed on at least one second integrated circuit die; and
the first integrated circuit die and the at least one second integrated circuit die are stacked to form a three-dimensional integrated circuit.

### Claim 14 (depends on 1)

The system of claim 1, wherein:
the host interface is further configured to receive a write command from the host system to store the first portion of the ANN in the memory device;
the host system sends the write command in response to evaluating prior data generated by the first sensor; and
the evaluating is performed using the second portion of the ANN.

### Claim 15 (depends on 14)

The system of claim 14, wherein the inference results include at least one of image classification, object detection, or pixel segmentation.

### Claim 16 (depends on 15)

The system of claim 15, wherein the system further comprises:
a control for at least one of steering, braking, or acceleration of a vehicle;
wherein the vehicle generates input for the control based on the inference results from the inference engine.

### Claim 17 (depends on 1)

The system of claim 1, wherein the host interface is further configured to communicate with the host system in accordance with a serial communication protocol.

### Claim 18

A method comprising:
collecting data from a sensor;
storing, by a memory device stacked with the sensor, the collected data;
storing, by the memory device, a first portion of an artificial neural network (ANN);
communicating, by a host interface, with a host system that stores a second portion of the ANN;
generating, by an inference engine of the memory device, inference results using the stored collected data as input to the first portion of the ANN; and
sending, by the host interface, the inference results to the host system for processing by the host system using the second portion of the ANN.

### Claim 19 (depends on 18)

The method of claim 18, wherein the collected data is sound data, the method further comprising converting the sound data to at least one image prior to use as the input to the first portion of the ANN.

### Claim 20

A non-transitory computer-readable medium storing instructions which, when executed on at least one computing device, cause the at least one computing device to:
store at least a portion of an artificial neural network (ANN) received from a computing system;
collect data from a sensing device;
store the collected data;
generate inference results using the collected data as input to the at least a portion of the ANN; and
send, by a communication interface, the inference results to the computing system for processing.
