# US 2019/0057314 A1 · Joint processing for embedded data inference

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0057314 A1
- Document ID: US20190057314A1
- Status: Published application
- Kind: A1
- Application number: 15/770,487
- Filed: 2016-10-25
- Published: 2019-02-21
- Earliest claimed: 2015-10-26
- Assignees: Netradyne, Inc.
- Applicants: Netradyne Inc
- Inventors: David Jonathan JULIAN; Avneesh AGRAWAL
- Web page: https://patentgrep.com/patent/US20190057314A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06F 8/65: Updates (security arrangements therefor G06F21/57)
- G06F 18/2431: Multiple classes
- G06F 18/25: Fusion techniques
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 5/043: Distributed expert systems; Blackboards
- G06N 20/00: Machine learning
- G06V 30/2504: {Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches}
- H04L 67/34: {involving the movement of software or configuration parameters (network booting or remote initial program loading [RIPL] G06F9/4416)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Systems and methods are provided for embedded data inference. The systems and methods may process camera and other sensor data in by leveraging processing and storage capacity of one or more devices nearby or in the cloud to augment or update the sensor processing of an embedded device. The joint processing may be used in stationary cameras or in vehicular systems such as cars and drones, and may improve crop assessments, navigation, and safety.

## Claims

### Claim 1

A method for updating a model on a machine, comprising:
obtaining sensor data at a first device; wherein the sensor data were captured at a sensor attached to the machine, and wherein the first device is proximate to the sensor;
identifying, at the first device, an object and determining a class of the object based on the sensor data and an embedded model on the first device;
sending communication data based on the sensor data to a second device;
determining a first model at a second device based on identifying from the communication data at least one of:
one or more additional classes of the object,
one or more additional objects, or
a context of the machine;
sending model information about the first model from the second device to the first device; and
updating the embedded model on the first device based on the model information.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the first model further comprises selecting a model from a model library.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the first model further comprises compressing a model from a model library.

### Claim 4 (depends on 1)

The method of claim 1, wherein the model information includes an indication to turn parts of the embedded model on or off.

### Claim 5 (depends on 1)

The method of claim 1, wherein the model information includes the first model, and further comprising: modifying the first model at the first device based on local data stored on the first device, wherein the local data includes at least one of local training data, local parameters, local capabilities, or local compression.

### Claim at position 6

6-8. (canceled)

### Claim 9 (depends on 1)

The method of claim 1, wherein the sensor data is based on at least one of visual processing, GPS measurements, inertial sensor measurements, audio inputs, inputs selected by a user, or a behavior of the user.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
determining system parameters of the first device; wherein the system parameters include at least one of compute capabilities, application latency targets, or some combination thereof; and wherein the determining a first model is based in part on the determined system parameters.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first model and the embedded model consist essentially of deep neural networks, support vector machines, random forests, recurrent neural networks, or reinforcement learning networks.

### Claim 12

(canceled)

### Claim 13 (depends on 1)

The method of claim 1, wherein the context of the machine is determined at the first device; and wherein the communication data comprises the determined context.

### Claim 14 (depends on 1)

The method of claim 1, wherein the context of the machine is determined at the second device; and wherein the communication data is based on at least one of sensor data, processed sensor data, or some combination thereof.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
obtaining a second communication data from a third device at the second device; and wherein the determining a context of the first device is further based on the second communication data obtained from the third device.

### Claim 16 (depends on 1)

The method of claim 1, wherein the context is based on at least one of weather, geography, road types, sign types, route type, motion type, location, lighting conditions, sun glare, sensor availability, computational resource availability, latency requirements, bandwidth availability, or some combination thereof.

### Claim at position 15

17-23. (canceled)

### Claim 24

A method of joint processing for data inference, comprising:
processing sensor data on a first device with a first model to create first inference data, wherein the sensor data were captured at a sensor, and wherein the first device is proximate to the sensor;
sending first communication data based on the sensor data to a second device;
processing the first communication data at the second device using a second model to create second inference data; and
sending second communication data based on the second inference data from the second device to the first device.

### Claim 25 (depends on 24)

The method of claim 24, further comprising:
determining at the second device how densely to sample an area around the first device; wherein the second communication data comprises an indication of an area of interest; and
adjusting the density of sampling based on the indication; wherein the adjusting includes more densely sampling the area of interest, less densely sampling the area of interest, or returning to a previously sampled area for additional sampling.

### Claim 26 (depends on 24)

The method of claim 24, further comprising:
determining at the second device how densely to sample sensor data from a time period of interest; wherein the second communication data comprises an indication of the time period of interest;
processing a portion of recorded sensor data on the first device to create third inference data; wherein the portion is determined based on the indication; and
sending third communication data to the second device based on the third inference data.

### Claim 27 (depends on 24)

The method of claim 24, wherein the second communication data further comprises anticipatory focus locations.

### Claim 28 (depends on 24)

The method of claim 24, wherein the second communication data further comprises an indication of a future time period of interest.

### Claim 29 (depends on 24)

The method of claim 24, wherein the processing sensor data comprises processing an anomaly detection algorithm, and further comprising:
detecting a putative anomaly based on the first inference data, and wherein the sending first communication data is triggered by the detecting, and wherein the second inference data comprises a classification of the putative anomaly.

### Claim at position 22

30-37. (canceled)

### Claim 38 (depends on 24)

The method of claim 24, wherein the processing of first communication data comprises a full image search, wherein processing sensor data further comprises tracking objects based on the location estimate from the second communication data, and wherein the processing sensor data further comprises a full image search at an image resolution that is lower than the image resolution of the full image search processed on the second device.

### Claim 39 (depends on 24)

The method of claim 24, wherein the processing sensor data is applied at a first resolution for a first portion of the sensor data and at a second resolution for a second portion of the sensor data.

### Claim 40 (depends on 39)

The method of claim 39, further comprising:
selecting the first portion of sensor data based on the first inference data and the second inference data, wherein the first inference data comprises a first location estimate of at least one object and the second inference data comprises a second location estimate of the at least one object; and
comparing the first location estimate and the second location estimate to create an agreement value; and wherein the first resolution is based on the agreement value.

### Claim at position 26

41-55. (canceled)

### Claim 56

A computer program product for updating a machine learning model on a device, the computer program product comprising:
a non-transitory computer-readable medium having program code recorded thereon, the program code comprising:
program code to obtain sensor data on a first device;
process sensor data on the first device with a first machine learning model to create first classification data;
send first communication data based on the sensor data to a second device;
process the first communication data at the second device using a second machine learning model to create second classification data;
determine differences between the first classification data and the second classification data; and
update the first model based on the determined differences.
