Enhanced beam search decoding for transformer-based OCR with probability score optimization

A system for optical character recognition of imagery, the system comprising:

• a processor;
• and
• a non-transitory computer-readable memory storing instructions that, when executed by the processor, cause the system to:
• receive an input image depicting an amount;
• process the input image through an encoder of a transformer-based optical character recognition model to generate encoded visual features;
• perform sequential decoding of the amount using a decoder of the transformer-based optical character recognition model, wherein performing the sequential decoding comprises:
• at each decoding step, generating, by the decoder, output values comprising raw logit scores for tokens in a vocabulary of possible tokens, the output values being conditioned on the encoded visual features and any previously decoded tokens;
• converting the output values from raw logit scores to normalized confidence scores using a normalization function;
• generating and maintaining a plurality of candidate output sequences based on the raw logit scores during the sequential decoding, each candidate output sequence comprising one or more decoded tokens, and wherein maintaining the plurality of candidate output sequences comprises, at each decoding step, expanding candidate output sequences based on the raw logit scores, pruning lower-scoring candidate output sequences based on average logit scores to retain a predetermined number of highest-scoring candidate output sequences, and then reranking the retained candidate output sequences based on logit scores calculated by averaging logit scores;
• for each candidate output sequence, calculating a cumulative confidence score by averaging the logit scores across all tokens of that candidate output sequence;
• selecting a final output sequence from among the plurality of candidate output sequences based on the cumulative confidence scores, the final output sequence having a maximum cumulative confidence score;
• and
• output the final output sequence as a recognized textual representation of the amount.

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

• receiving an input image depicting an amount;
• processing the input image through an encoder of a transformer-based optical character recognition model to generate encoded visual features;
• performing beam search decoding of the amount using a decoder of the transformer-based optical character recognition model, wherein the beam search decoding comprises:
• initializing a plurality of beams, each beam representing a candidate output sequence;
• at each decoding step:
• for each beam of the plurality of beams, generating, by the decoder, output values comprising raw logit scores for tokens in a vocabulary of possible next tokens, the output values being conditioned on the encoded visual features and tokens already present in that beam;
• converting the output values from raw logit scores to normalized confidence scores using a normalization function;
• expanding each beam based on the raw logit scores by appending possible next tokens selected according to the raw logit scores to create expanded candidate output sequences;
• pruning the expanded candidate output sequences based on average logit scores to retain a predetermined beam width number of highest-scoring expanded candidate output sequences as beams for a subsequent decoding step;
• for each retained expanded candidate output sequence, calculating a cumulative confidence score by averaging the logit scores across all tokens in that expanded candidate output sequence;
• and
• reranking the retained expanded candidate output sequences based on the cumulative confidence scores;
• continuing the beam search decoding until an end-of-sequence token is generated or a maximum sequence length is reached;
• selecting a final output sequence having a maximum cumulative confidence score from among the beams;
• and
• outputting the final output sequence as a recognized textual representation of the amount for use in remote processing.