# US 2025/0148277 A1 · Entropy-based online learning with active sparse layer update for on-device training with resource-constrained devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148277 A1
- Document ID: US20250148277A1
- Status: Published application
- Kind: A1
- Application number: 18/501,409
- Filed: 2023-11-03
- Published: 2025-05-08
- Earliest claimed: 2023-11-03
- Assignees: Dell Products L.P.
- Applicants: Dell Products L.P.
- Inventors: Jonathan Mendes De Almeida; Renam Castro Da Silva; Victor da Cruz Ferreira
- Web page: https://patentgrep.com/patent/US20250148277A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques are disclosed for sparse layer-wise training of neural networks. An example system includes at least one processing device including a processor coupled to a memory. The at least one processing device can be configured to implement the following steps: obtaining class predictions while saving activations for only a number ‘k’ layers of a neural network, using the class predictions to calculate a layer shallowness measure for the neural network, using the layer shallowness measure to determine a number ‘u’ of layers to update in the neural network, and partially updating the neural network by training only the number ‘u’ layers of the neural network.

## Claims

### Claim 1

A system comprising:
at least one processing device including a processor coupled to a memory;
the at least one processing device being configured to implement the following steps:
obtaining class predictions while saving activations for only a number ‘k’ layers of a neural network;
using the class predictions to calculate a layer shallowness measure for the neural network;
using the layer shallowness measure to determine a number ‘u’ of layers to update in the neural network; and
partially updating the neural network by training only the number ‘u’ layers of the neural network.

### Claim 2 (depends on 1)

The system of claim 1, wherein the layer shallowness measure is an adaptive partial model backpropagation measure.

### Claim 3 (depends on 2)

The system of claim 2, wherein the layer shallowness measure is determined dynamically as the neural network is retrained.

### Claim 4 (depends on 1)

The system of claim 1, wherein the layer shallowness measure is determined dynamically by detecting drift in the neural network.

### Claim 5 (depends on 4)

The system of claim 4, wherein the drift is detected using entropy values determined based on classes predicted by the neural network.

### Claim 6 (depends on 5)

The system of claim 5, wherein the number ‘u’ of layers to update is determined using the entropy values.

### Claim 7 (depends on 6)

The system of claim 6, wherein the number ‘u’ of layers to update is determined by:
dividing the entropy values into a plurality of ranges; and
determining a count of entropy values that fall into each range.

### Claim 8 (depends on 1)

The system of claim 1, wherein ‘k’ and ‘u’ are fewer than all layers of the neural network.

### Claim 9 (depends on 1)

The system of claim 1, wherein the activations are saved for the last ‘k’ layers of the neural network.

### Claim 10 (depends on 1)

The system of claim 1, wherein the neural network is trained using sparse training to update the last ‘u’ layers of the neural network.

### Claim 11 (depends on 1)

The system of claim 1, wherein the at least one processing device is further configured to implement the following steps:
after partially updating the neural network, updating the number ‘k’ to have the value of the number ‘u’.

### Claim 12 (depends on 1)

The system of claim 1, wherein the steps are performed on a resource-constrained device.

### Claim 13 (depends on 11)

The system of claim 11, wherein the resource-constrained device is an edge node of an edge network.

### Claim 14 (depends on 1)

The system of claim 1, wherein the neural network is a classifier model.

### Claim 15

A method comprising:
obtaining class predictions while saving activations for only a number ‘k’ layers of a neural network;
using the class predictions to calculate a layer shallowness measure for the neural network;
using the layer shallowness measure to determine a number ‘u’ of layers to update in the neural network; and
partially updating the neural network by training only the number ‘u’ layers of the neural network.

### Claim 16 (depends on 15)

The method of claim 15, wherein the layer shallowness measure is an adaptive partial model backpropagation measure.

### Claim 17 (depends on 16)

The method of claim 16, wherein the layer shallowness measure is determined dynamically as the neural network is retrained.

### Claim 18 (depends on 15)

The method of claim 15, wherein the layer shallowness measure is determined dynamically by detecting drift in the neural network.

### Claim 19 (depends on 18)

The method of claim 18, wherein the drift is detected using entropy values determined based on classes predicted by the neural network.

### Claim 20

A non-transitory processor-readable storage medium having stored thereon program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:
obtaining class predictions while saving activations for only a number ‘k’ layers of a neural network;
using the class predictions to calculate a layer shallowness measure for the neural network;
using the layer shallowness measure to determine a number ‘u’ of layers to update in the neural network; and
partially updating the neural network by training only the number ‘u’ layers of the neural network.
