# US 11,308,354 B1 · Residual context refinement network architecture for optical character recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,354 B1 - Document ID: US11308354B1 - Status: Granted - Kind: B1 - Application number: 16/834,997 - Filed: 2020-03-30 - Granted: 2022-04-19 - Earliest claimed: 2020-03-30 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Ron Litman; Oron Anschel; Shahar Tsiper; Roee Litman; Shai Mazor; Jonathan Wu; Raghavan Manmatha - Web page: https://patentgrep.com/patent/US11308354B1 ## CPC - G06V 30/147: {Determination of region of interest} - G06F 18/211: Selection of the most significant subset of features - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2163: {Partitioning the feature space} - G06F 18/24: Classification techniques - G06V 10/32: Normalisation of the pattern dimensions - G06V 30/18057: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques for recognizing text in an image are described. An exemplary method may include receiving a request to recognize text in an image; extracting features from the image and generating a visual feature sequence from the extracted features; performing selective contextual refinement at least one selective contextual refinement block of a stack of selective contextual refinement blocks to generate a text prediction by: generating a contextual feature map and combining the contextual feature map with the visual feature sequence into a visual feature space, and applying a selective decoder that utilizes a two-step attention on the visual feature space to generate a text prediction, wherein the two-step attention includes performing a 1-D self-attention computation to generate attentional features and decoding the attentional features to generate the text prediction; and outputting the generated text prediction. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a request to recognize text in an image; transforming the image into a normalized image; extracting features from the normalized image and generating a visual feature sequence from the extracted features; refining the visual feature sequence; performing selective contextual refinement at least one time to generate a text prediction by: generating a contextual feature map and concatenating the contextual feature map with the visual feature sequence into a visual feature space, and applying a selective decoder that utilizes a two-step attention on the visual feature space to generate a text prediction, wherein the two-step attention includes: obtaining an attention map from the visual feature space, performing an element-wise product between the attention map and the visual feature sequence to generate attentional features, and decoding the attentional features to generate the text prediction; and outputting the generated text prediction. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the received image is a crop of a large image, the crop focusing on the text image. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein refining the visual feature sequence comprises classifying each feature of the visual feature sequence. ### Claim 4 A computer-implemented method comprising: receiving a request to recognize text in an image; extracting features from the image and generating a visual feature sequence from the extracted features; performing selective contextual refinement at least one time to generate a text prediction by: generating a contextual feature map and combining the contextual feature map with the visual feature sequence into a visual feature space, and applying a selective decoder that utilizes a two-step attention on the visual feature space to generate a text prediction, wherein the two-step attention includes performing a 1-D self-attention computation to generate attentional features and decoding the attentional features to generate the text prediction; and outputting the generated text prediction. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the received image is a crop of a large image, the crop focusing on the text image. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, further comprising: using a machine learning model to crop the image before extracting features. ### Claim 7 (depends on 4) The computer-implemented method of claim 4, wherein the 1-D self-attention computation comprises: obtaining an attention map from the visual feature space; and performing an element-wise product between the visual feature sequence and the attention map to generate the attentional features. ### Claim 8 (depends on 4) The computer-implemented method of claim 4, wherein decoding the attentional features to generate the text prediction comprises using an attention decoder to compute a vector of attentional weights, combining the visual feature space into a vector, and feeding a recurrent cell of the attention decoder a combination of the vector and an embedding. ### Claim 9 (depends on 4) The computer-implemented method of claim 4, wherein the contextual feature map is generated using one or more bidirectional long short-term memory neural networks. ### Claim 10 (depends on 4) The computer-implemented method of claim 4, wherein the selective decoder was trained along with one or more selective decoders of a stack of selective-contextual refinement blocks. ### Claim 11 (depends on 4) The computer-implemented method of claim 4, further comprising: transforming the image into a normalized image. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein transforming the image into a normalized image is performed using thin plate spline transformation. ### Claim 13 (depends on 4) The computer-implemented method of claim 4, wherein the features from the image are extracted using a convolutional neural network. ### Claim 14 (depends on 4) The computer-implemented method of claim 4, wherein the request includes an indication of a location of the image in a provider network. ### Claim 15 A system comprising: a first one or more electronic devices to store an image in a multi-tenant provider network; and a second one or more electronic devices to implement an image text recognizing service in the multi-tenant provider network, the image text recognizing service including instructions that, upon execution, cause the image text recognizing service to: receive a request to recognize text in a stored image, extract features from the image and generating a visual feature sequence from the extracted features, and perform selective contextual refinement at least one time to generate a text prediction by: generating a contextual feature map and combining the contextual feature map with the visual feature sequence into a visual feature space, and applying a selective decoder that utilizes a two-step attention on the visual feature space to generate a text prediction, wherein the two-step attention includes performing a 1-D self-attention computation to generate attentional features and decoding the attentional features to generate the text prediction; and output the generated text prediction. ### Claim 16 (depends on 15) The system of claim 15, wherein image text recognizing service is further to transform the image into a normalized image. ### Claim 17 (depends on 15) The system of claim 15, wherein transforming the image into a normalized image is performed using thin plate spline transformation. ### Claim 18 (depends on 15) The system of claim 15, wherein the features from the image are extracted using a convolutional neural network. ### Claim 19 (depends on 15) The system of claim 15, wherein the request includes an indication of a location of the image in a provider network. ### Claim 20 (depends on 15) The system of claim 15, wherein the 1-D self-attention computation comprises: obtaining an attention map from the visual feature space; and performing an element-wise product between the visual feature sequence and the attention map to generate the attentional features.