Back to results
US 9,792,492 B2Granted
Extracting gradient features from neural networks
AssigneeXerox Corporation
InventorsAlbert Gordo Soldevila, Adrien Gaidon, Florent C. Perronnin
Life of this patent
Estimating the term…
Abstract
A method for extracting a representation from an image includes inputting an image to a pre-trained neural network. The gradient of a loss function is computed with respect to parameters of the neural network, for the image. A gradient representation is extracted for the image based on the computed gradients, which can be used, for example, for classification or retrieval.
Claims18
A method for extracting a representation from an image, comprising:
- inputting an image to a pre-trained neural network comprising a sequence of layers;
- computing a gradient of a loss function with respect to parameters of the neural network for the image comprising:
- in a forward pass of the pre-trained neural network:
- extracting a set of forward features input to a selected one of the layers of the neural network which is not a final layer of the neural network, and
- generating a vector of label predictions for the input image from an output of the final layer,
- computing an error vector based on differences between the vector of label predictions and a standardized prediction vector,
- in a backward pass of the neural network, computing a set of backward features from the neural network, the backward features comprising a gradient of a loss function with respect to the output of the selected layer of the neural network, and
- combining the forward and the backward features through a tensor product operation to compute a set of gradient features for the image;
- extracting a vectorial gradient representation of the image comprising the computed set of gradient features;
- outputting the vectorial gradient representation or information based on the vectorial gradient representation, the output information including one of:
- a class for the input image output by a classifier trained using gradient representations of training images,
- a computed similarity between the input image and another image based on respective gradient representations of the input image and the other image, and
- a set of similar images retrieved based on similarity of their gradient representations,
- wherein at least one of the computing and the extracting is performed with a processor including hardware.
11 more elements
A method for computing a similarity between two images, comprising:
- with a processor including hardware, for each of two images:
- inputting the image to a pre-trained neural network,
- computing a forward representation in a forward pass of the neural network, and
- computing a backward representation comprising computing a gradient of a loss function with respect to output parameters of a layer of the neural network in a backward pass of the neural network;
- computing a forward similarity between the forward representations of the two images extracted in the forward pass of the neural network;
- computing a backward similarity between the backward representations of the two images extracted in the backward pass of a pre-trained neural network;
- computing the similarity between two images by combining the forward and backward similarities;
- and
- outputting the computed similarity.
6 more elements
A system for extracting a representation from an image, comprising:
- memory which stores a pre-trained neural network;
- a prediction component for predicting labels for an input image using a forward pass of the neural network, and
- a gradient computation component for computing the gradient of a loss function with respect to parameters of the neural network for the image based on the predicted labels on a backward pass of the neural network according to the expression:
- ∂
- E
- ∂
- W
- k
- =
- x
- k
- -
- 1
-
- [
- Y
- k
-
- [
- W
- k
- +
- 1
-
- ∂
- E
- ∂
- y
- k
- +
- 1
- ]
- ]
- T
- ,
- where Yk is a square diagonal matrix constructed with the elements of Π[yk>0],
- Xk−1 represents a respective forward feature input to a selected layer k of the neural network which is not the last layer, and
- ∂
- E
- ∂
- y
- k
- +
- 1
- is the partial derivative of the loss E with respect to the output parameters ykof the layer k +1 of the neural network, and
- Wk+1 are parameters of the layer k +1 of the neural network;
- a gradient representation generator for extracting a gradient representation of the image based on the computed gradient, the gradient representation being a vector including the computed gradient of the loss function with respect to the parameters of the neural network
- ∂
- E
- ∂
- W
- k
- ;
- an output component which outputs the gradient representation or information based thereon;
- and
- a processor in communication with the memory for implementing the prediction component, gradient computation component, gradient representation generator and output component.
53 more elements
Dependencies are inferred from the claim text.