# US 12,217,017 B2 · Translation of text depicted in images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,017 B2 - Document ID: US12217017B2 - Status: Granted - Kind: B2 - Application number: 17/791,409 - Filed: 2020-01-08 - Published: 2023-04-20 - Granted: 2025-02-04 - Earliest claimed: 2020-01-08 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Puneet Jain; Orhan Firat; Sihang Liang - Web page: https://patentgrep.com/patent/US12217017B2 ## CPC - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 10/82: using neural networks - G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images - G06F 40/42: Data-driven translation - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that translate text depicted in images from a source language into a target language. Methods can include obtaining a first image that depicts first text written in a source language. The first image is input into an image translation model, which includes a feature extractor and a decoder. The feature extractor accepts the first image as input and in response, generates a first set of image features that are a description of a portion of the first image in which the text is depicted is obtained. The first set of image features are input into a decoder. In response to the input first set of image features, the decoder outputs a second text that is a predicted translation of text in the source language that is represented by the first set of image features. ## Claims ### Claim 1 A computer implemented method, comprising: obtaining a first image that depicts first text written in a source language; inputting the first image into an image feature extractor of an image translation model that is trained, using a loss function, to extract, from an input image, a set of image features that are a description of a portion of the input image in which the text is depicted; obtaining, from the feature extractor and in response to inputting the first image into the feature extractor, a first set of image features representing the first text present in the first image; inputting the first set of image features into a decoder of the image translation model, wherein the decoder is trained, using the loss function, to infer text in a target language from an input set of image features, wherein the inferred text is a predicted translation of text represented by the input set of image features; and obtaining, from the decoder and in response to inputting the first set of image features into the decoder, a second text that is in a target language and is predicted to be a translation of the first text. ### Claim 2 (depends on 1) The computer implemented method of claim 1, further comprising: training the feature extractor using a set of input training images that depict training text in the source language and corresponding sets of training image features, wherein each set of training image features is a description of a portion of the input image in which the training text is depicted. ### Claim 3 (depends on 1) The computer implemented method of claim 1: wherein the decoder includes a text-to-text translation model. ### Claim 4 (depends on 3) The method of claim 3, further comprising training the decoder, wherein the training includes: training the text-to-text translation model to translate text written in the source language into text in the target language, wherein the text-to-text translation model is trained using a set of input training text data in the source language and a corresponding set of output training text data that is a translation of the input training text data from the source language into the target language; and training the trained text-to-text translation model to output text data in a target language that is a predicted translation of text represented by an input set of image features that represent text in an input image, wherein the trained text-to-text translation model is trained using a set of input training images that depict training text in a source language and a corresponding set of text data that is a translation in a target language of the training text depicted in the input training images. ### Claim 5 (depends on 1) The computer implemented method of claim 1, wherein the feature extractor is a convolution neural network (CNN) with 53 layers of convolution, residual, and pooling. ### Claim 6 (depends on 1) The computer implemented method of claim 1, wherein the decoder is a 6-layer multi-head transformer decoder and the text-to-text translation model is a transformer neural machine translation model. ### Claim 7 (depends on 1) The computer implemented method of claim 1, wherein the decoder is a multi-layer multi-head transformer decoder and the text-to-text translation model is a transformer neural machine translation model. ### Claim 8 A system, comprising: one or more memory devices storing instructions; and one or more data processing apparatus that are configured to interact with the one or more memory devices, and upon execution of the instructions, perform operations including: obtaining a first image that depicts first text written in a source language; inputting the first image into an image feature extractor of an image translation model that is trained, using a loss function, to extract, from an input image, a set of image features that are a description of a portion of the input image in which the text is depicted; obtaining, from the feature extractor and in response to inputting the first image into the feature extractor, a first set of image features representing the first text present in the first image; inputting the first set of image features into a decoder of the image translation model, wherein the decoder is trained, using the loss function, to infer text in a target language from an input set of image features, wherein the inferred text is a predicted translation of text represented by the input set of image features; and obtaining, from the decoder and in response to inputting the first set of image features into the decoder, a second text that is in a target language and is predicted to be a translation of the first text. ### Claim 9 (depends on 8) The system of claim 8, wherein the one or more data processing apparatus are configured to perform operations further comprising: training the feature extractor using a set of input training images that depict training text in the source language and corresponding sets of training image features, wherein each set of training image features is a description of a portion of the input image in which the training text is depicted. ### Claim 10 (depends on 8) The system of claim 8, wherein the decoder includes a text-to-text translation model. ### Claim 11 (depends on 10) The system of claim 10, wherein the one or more data processing apparatus are configured to perform operations further comprising training the decoder, the training includes: training the text-to-text translation model to translate text written in the source language into text in the target language, wherein the text-to-text translation model is trained using a set of input training text data in the source language and a corresponding set of output training text data that is a translation of the input training text data from the source language into the target language; and training the trained text-to-text translation model to output text data in a target language that is a predicted translation of text represented by an input set of image features that represent text in an input image, wherein the trained text-to-text translation model is trained using a set of input training images that depict training text in a source language and a corresponding set of text data that is a translation in a target language of the training text depicted in the input training images. ### Claim 12 (depends on 8) The system of claim 8, wherein the feature extractor is a convolution neural network (CNN) with 53 layers of convolution, residual, and pooling. ### Claim 13 (depends on 8) The system of claim 8, wherein the decoder is a 6-layer multi-head transformer decoder and the text-to-text translation model is a transformer neural machine translation model. ### Claim 14 A non-transitory computer readable medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations comprising: obtaining a first image that depicts first text written in a source language; inputting the first image into an image feature extractor of an image translation model that is trained, using a loss function, to extract, from an input image, a set of image features that are a description of a portion of the input image in which the text is depicted; obtaining, from the feature extractor and in response to inputting the first image into the feature extractor, a first set of image features representing the first text present in the first image; inputting the first set of image features into a decoder of the image translation model, wherein the decoder is trained, using the loss function, to infer text in a target language from an input set of image features, wherein the inferred text is a predicted translation of text represented by the input set of image features; and obtaining, from the decoder and in response to inputting the first set of image features into the decoder, a second text that is in a target language and is predicted to be a translation of the first text. ### Claim 15 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the instructions cause the one or more data processing apparatus to perform operations comprising: training the feature extractor using a set of input training images that depict training text in the source language and corresponding sets of training image features, wherein each set of training image features is a description of a portion of the input image in which the training text is depicted. ### Claim 16 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the decoder includes a text-to-text translation model. ### Claim 17 (depends on 16) The non-transitory computer readable medium of claim 16, wherein the instructions cause the one or more data processing apparatus to perform operations comprising training the decoder, wherein the training includes: training the text-to-text translation model to translate text written in the source language into text in the target language, wherein the text-to-text translation model is trained using a set of input training text data in the source language and a corresponding set of output training text data that is a translation of the input training text data from the source language into the target language; and training the trained text-to-text translation model to output text data in a target language that is a predicted translation of text represented by an input set of image features that represent text in an input image, wherein the trained text-to-text translation model is trained using a set of input training images that depict training text in a source language and a corresponding set of text data that is a translation in a target language of the training text depicted in the input training images. ### Claim 18 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the feature extractor is a convolution neural network (CNN) with 53 layers of convolution, residual, and pooling. ### Claim 19 (depends on 14) The non-transitory computer readable medium of claim 14, wherein the decoder is a 6-layer multi-head transformer decoder and the text-to-text translation model is a transformer neural machine translation model.