Back to results
US 10,650,230 B2Granted
Image data extraction using neural networks
AssigneeSap SE
InventorReid Johnson
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure pertain to extracting data from images using neural networks. In one embodiment, an image is fit to a predetermined bounding window. The image is then processed with a convolutional neural network to produce a three dimensional data cube. Slices of the cube are processed by an encoder RNN, and the results concatenated. The concatenated results are processed by an attention layer with input from a downstream decoder RNN. The attention layer output is provided to the decoder RNN to generate a probability array where values in the probability array correspond to particular characters in a character set. The maximum value is selected, and translated into an output character. In one embodiment, an amount may be extracted from an image of a receipt.
Claims20
A computer implemented method comprising:
- receiving an image in a convolutional neural network, and in accordance therewith, producing an L×M×N three (3) dimensional array of data values, wherein L, M, and N are integers;
- sequentially processing each L×M slice of the three dimensional array of data values in an encoder recurrent neural network to produce N L×M outputs, wherein each L×M slice corresponds to M characters encoded by a L-length vector;
- concatenating the N L×M outputs to produce an L×(M*N) array of values;
- normalizing the M times N values in each row to produce an L×1 attention vector;
- for a plurality of cycles corresponding to a maximum number of characters to be extracted from the image:
- concatenating the L×1 attention vector with an L×1 internal state vector of a decoder recurrent neural network to produce a (2*L)×1 input vector to the decoder recurrent neural network;
- combining the (2*L)×1 input vector with an L×L weight matrix to produce the L×1 internal state vector;
- matrix multiplying the L×1 internal state vector by an J×L weight matrix to produce a J×1 vector of probability values, wherein each probability value in the J×1 vector corresponds to a likelihood for one character to be extracted of a character set having J distinct characters;
- and
- selecting the probability value having the greatest value;
- and
- translating the selected probability into a representation of a character;
- outputting the characters produced over the plurality of cycles.
10 more elements
A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer, the program comprising sets of instructions for:
- receiving an image in a convolutional neural network, and in accordance therewith, producing an L×M×N three (3) dimensional array of data values, wherein L, M, and N are integers;
- sequentially processing each L×M slice of the three-dimensional array of data values in an encoder recurrent neural network to produce N L×M outputs, wherein each L×M slice corresponds to M characters encoded by a L-length vector;
- concatenating the N L×M outputs to produce an L×(M*N) array of values;
- normalizing the M times N values in each row to produce an L×1 attention vector;
- for a plurality of cycles corresponding to a maximum number of characters to be extracted from the image:
- concatenating the L×1 attention vector with an L×1 internal state vector of a decoder recurrent neural network to produce a (2*L)×1 input vector to the decoder recurrent neural network;
- combining the (2*L)×1 input vector with an L×L weight matrix to produce the L×1 internal state vector;
- matrix multiplying the L×1 internal state vector by an J×L weight matrix to produce a J×1 vector of probability values, wherein each probability value in the J×1 vector corresponds to a likelihood for one character to be extracted of a character set having J distinct characters;
- and
- selecting the probability value having the greatest value;
- and
- translating the selected probability into a representation of a character;
- outputting the characters produced over the plurality of cycles.
10 more elements
A computer system comprising:
- a processor;
- and
- a non-transitory machine-readable medium storing a program executable by the processor, the program comprising sets of instructions for:
- receiving an image in a convolutional neural network, and in accordance therewith, producing an L×M×N three (3) dimensional array of data values, wherein L, M, and N are integers;
- sequentially processing each L×M slice of the three-dimensional array of data values in an encoder recurrent neural network to produce N L×M outputs, wherein each L×M slice corresponds to M characters encoded by a L-length vector;
- concatenating the N L×M outputs to produce an L×(M*N) array of values;
- normalizing the M times N values in each row to produce an L×1 attention vector;
- for a plurality of cycles corresponding to a maximum number of characters to be extracted from the image:
- concatenating the L×1 attention vector with an L×1 internal state vector of a decoder recurrent neural network to produce a (2*L)×1 input vector to the decoder recurrent neural network;
- combining the (2*L)×1 input vector with an L×L weight matrix to produce the L×1 internal state vector;
- matrix multiplying the L×1 internal state vector by an J×L weight matrix to produce a J×1 vector of probability values, wherein each probability value in the J×1 vector corresponds to a likelihood for one character to be extracted of a character set having J distinct characters;
- and
- selecting the probability value having the greatest value;
- and
- translating the selected probability into a representation of a character;
- outputting the characters produced over the plurality of cycles.
13 more elements
Dependencies are inferred from the claim text.