US 11,455,538 B2Granted

Correctness preserving optimization of deep neural networks

AssigneeGM Global Technology Operations LLC

InventorsPrakash Mohan Peranandam, Ramesh Sethu, Alena Rodionova

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method for reducing the number of neurons in a trained deep neural network (DNN) includes classifying layer types in a plurality of hidden layers; evaluating the accuracy of the DNN using a validation set of data; and generating a layer specific ranking of neurons, wherein the generating includes: analyzing, using the validation set of data for one or more of the plurality of hidden layers, the activation function for each neuron in the analyzed layers to determine an activation score for each neuron; and ranking, on a layer type basis, each neuron in the analyzed layers based on the neuron's activation score to generate a layer specific ranking of neurons. The method further includes removing a number of lower ranked neurons from the DNN that does not result in the DNN after the removal of selected lower ranked neurons to fall outside of an accuracy threshold limit.

Claims20

Dependent claims

A processor-implemented method for reducing the number of neurons and their interconnections in a trained deep neural network (DNN) used in a vehicle, the method comprising:

identifying and classifying layer types in a plurality of hidden layers of the DNN;
evaluating the accuracy of the DNN using a validation set of data;
generating a layer specific ranking of neurons, the generating comprising:
analyzing, using the validation set of data for one or more of the plurality of hidden layers, an activation function for each neuron in the analyzed one or more layers to determine an activation score for each neuron corresponding to an analyzed activation function;
and
ranking, on a layer type basis, each neuron in the analyzed one or more layers based on the activation score for the neuron to generate a layer specific ranking of neurons, the ranking including generating a normalized neuron activation output score for each neuron and ranking the neurons in a layer based on its normalized activation output score;
selecting a number of lower ranked neurons of a specific type from the DNN for removal that does not result in the DNN after the removal of selected lower ranked neurons to fall outside of an accuracy threshold limit, the selecting a number of lower ranked neurons from the DNN for removal comprising iteratively analyzing and selecting lower ranked neurons from the DNN for removal until an iteration threshold is met, the iteratively selecting lower ranked neurons from the DNN for removal comprising:
designating some of the lower ranked neurons for removal;
comparing the accuracy of a newer DNN with the lower ranked neurons designated for removal removed to the DNN without neurons removed;
when the accuracy of the newer DNN is not within accuracy threshold limits, performing another iteration of the selecting while designating fewer neurons for removal if an iteration limit has not been met or selecting for removal the neurons designated for removal from the latest prior iteration wherein the accuracy of the newer DNN was within accuracy threshold limits if the iteration limit has been met;
and
when the accuracy of the newer DNN is within accuracy threshold limits, performing another iteration of the selecting while designating more neurons for removal if an iteration limit has not been met or selecting for removal the neurons designated for removal in the current iteration if the iteration limit has been met;
and
removing the neurons selected for removal from the DNN;
wherein the DNN with the removed neurons is configured to be used to perform perception tasks in a vehicle.

12 more elements

A vehicle perception system comprising one or more processors configured by programming instructions on non-transitory computer readable media to implement a trained deep neural network (DNN) with removed neurons, the DNN configured by a processing system configured to:

identify and classify layer types in a plurality of hidden layers of the DNN;
evaluate the accuracy of the DNN using a validation set of data;
generate a layer specific ranking of neurons, to generate the processing system is configured to:
analyze, using the validation set of data for one or more of the plurality of hidden layers, an activation function for each neuron in the analyzed one or more layers to determine an activation score for each neuron corresponding to an analyzed activation function;
and
rank, on a layer type basis, each neuron in the analyzed one or more layers based on the activation score for the neuron to generate a layer specific ranking of neurons, the ranking including generating a normalized neuron activation output score for each neuron and ranking the neurons in a layer based on its normalized activation output score;
select a number of lower ranked neurons from the DNN for removal that does not result in the DNN after the removal of selected lower ranked neurons to fall outside of an accuracy threshold limit, wherein to select a number of lower ranked neurons from the DNN for removal the processing system is configured to iteratively select lower ranked neurons from the DNN for removal until an iteration threshold is met, wherein to iteratively select lower ranked neurons from the DNN for removal the processing system is configured to:
designate some of the lower ranked neurons for removal;
compare the accuracy of a newer DNN with the lower ranked neurons designated for removal removed to the DNN without neurons removed;
when the accuracy of the newer DNN is not within accuracy threshold limits, perform another iteration of the selecting while designating fewer neurons for removal if an iteration limit has not been met or select for removal the neurons designated for removal from the latest prior iteration wherein the accuracy of the newer DNN was within accuracy threshold limits if the iteration limit has been met;
and
when the accuracy of the newer DNN is within accuracy threshold limits, perform another iteration of the selecting while designating more neurons for removal if an iteration limit has not been met or select for removal the neurons designated for removal in the current iteration if the iteration limit has been met;
and
remove the neurons selected for removal from the DNN;
wherein the DNN with the removed neurons is configured to perform perception tasks in a vehicle.

12 more elements

A processing system comprising one or processors configured by programming instructions on non-transitory computer readable media to reduce the number of neurons in a trained deep neural network (DNN), the processing system configured to:

identify and classify layer types in a plurality of hidden layers of the DNN;
evaluate the accuracy of the DNN using a validation set of data;
generate a layer specific ranking of neurons, to generate the processing system is configured to:
analyze, using the validation set of data for one or more of the plurality of hidden layers, an activation function for each neuron in the analyzed one or more layers to determine an activation score for each neuron corresponding to an analyzed activation function;
and
rank, on a layer type basis, each neuron in the analyzed one or more layers based on the activation score for the neuron to generate a layer specific ranking of neurons, the ranking including generating a normalized neuron activation output score for each neuron and ranking the neurons in a layer based on its normalized activation output score;
select a number of lower ranked neurons from the DNN for removal that does not result in the DNN after the removal of selected lower ranked neurons to fall outside of an accuracy threshold limit, wherein to select a number of lower ranked neurons from the DNN for removal the processing system is configured to iteratively select lower ranked neurons from the DNN for removal until an iteration threshold is met, wherein to iteratively select lower ranked neurons from the DNN for removal the processing system is configured to:
designate some of the lower ranked neurons for removal;
compare the accuracy of a newer DNN with the lower ranked neurons designated for removal removed to the DNN without neurons removed;
when the accuracy of the newer DNN is not within accuracy threshold limits, perform another iteration of the selecting while designating fewer neurons for removal if an iteration limit has not been met or select for removal the neurons designated for removal from the latest prior iteration wherein the accuracy of the newer DNN was within accuracy threshold limits if the iteration limit has been met;
and
when the accuracy of the newer DNN is within accuracy threshold limits, perform another iteration of the selecting while designating more neurons for removal if an iteration limit has not been met or select for removal the neurons designated for removal in the current iteration if the iteration limit has been met;
and
remove the neurons selected for removal from the DNN;
wherein the DNN with the removed neurons is configured to perform perception tasks in a vehicle.

12 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/227,195Filed 20 Dec 2018
Published: US 2020/0202214 A125 Jun 2020
Granted: 27 Sep 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification13 codes

G06N 3/082
Learning methods: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
G06F 18/217
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: validation; Performance…
G06F 18/2414
Distances to cluster centroïds: smoothing the distance, e.g. radial basis function networks [RBFN]
G06N 3/04
Neural networks: architecture, e.g. interconnection topology