# US 12,762,882 B2 · Reading order with pointer transformer networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,882 B2 - Document ID: US12762882B2 - Status: Granted - Kind: B2 - Application number: 18/686,233 - Filed: 2022-08-25 - Published: 2024-08-01 - Granted: 2026-10-13 - Earliest claimed: 2021-09-01 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Henri Rebecq; Federico Tombari; Diego Martin Arroyo - Web page: https://patentgrep.com/patent/US12762882B2 ## CPC - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/82: using neural networks - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method including receiving an image representing a document including a plurality of layout components, identifying textual information associated with the plurality of layout components, identifying visual information associated with the plurality of layout components, combining the textual information with the visual information, and predicting a reading order of the plurality of layout components based on the combined textual information and visual information using a self-attention encoder/decoder. ## Claims ### Claim 1 A method comprising: receiving an image representing a document including a plurality of layout components; identifying textual information associated with the plurality of layout components; identifying visual information associated with the plurality of layout components; combining the textual information with the visual information; and predicting a reading order of the plurality of layout components based on the combined textual information and visual information using: a self-attention encoder configured to generate an embedding based on a first sequence associated with the plurality of layout components; and a self-attention decoder configured to generate a second sequence based on the embedding. ### Claim 2 (depends on 1) The method of claim 1, wherein the identifying of the textual information includes extracting text-based data from the image. ### Claim 3 (depends on 2) The method of claim 2, wherein the extracting of the text-based data includes using a neural network configured to generate an embedding representing the textual information. ### Claim 4 (depends on 3) The method of claim 3, wherein the neural network is a pretrained neural network that maps textual data to an embedding, and the embedding represents an element including the text-based data associated with a layout component of the plurality of layout components. ### Claim 5 (depends on 1) The method of claim 1, wherein the identifying of the visual information includes extracting visual-based data from the image. ### Claim 6 (depends on 5) The method of claim 5, wherein the extracting of the visual-based data includes using a neural network configured to generate an embedding including the visual information. ### Claim 7 (depends on 6) The method of claim 6, wherein the neural network includes a two-dimensional convolution operation, the embedding includes an array, and the array includes an element including the visual-based data associated with the plurality of layout components. ### Claim 8 (depends on 6) The method of claim 6, wherein the neural network includes a plurality of two-dimensional convolution operations, and the embedding includes an array including an element including the visual-based data associated with an associated layout component and the visual-based data associated with at least one additional layout component. ### Claim 9 (depends on 1) The method of claim 1, wherein the textual information is associated with a first embedding, the visual information is associated with a second embedding, and the combining of the textual information with the visual information includes concatenating the first embedding with the second embedding. ### Claim 10 (depends on 1) The method of claim 1, wherein the first sequence has a first order, and the second sequence has a second order different from the first order. ### Claim 11 (depends on 1) The method of claim 1, wherein the self-attention encoder is configured to: weight relationships between pairs of elements in a set, and generate an embedding for the elements. ### Claim 12 (depends on 1) The method of claim 1, wherein the self-attention encoder is configured to determine an influence of an element in an embedding based on the combined textual information and visual information. ### Claim 13 (depends on 1) The method of claim 1, wherein the self-attention decoder is configured to operate as an auto-regressive inference. ### Claim 14 (depends on 1) The method of claim 1, wherein the self-attention decoder is configured to auto-regressively predict a next layout component in the reading order associated with the plurality of layout components. ### Claim 15 (depends on 1) The method of claim 1, wherein the self-attention decoder is configured to perform a QKV outer product between elements of the self-attention encoder and inputs to the self-attention decoder. ### Claim 16 A non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing system to: receive an image representing a document including a plurality of layout components; identify textual information associated with the plurality of layout components; identify visual information associated with the plurality of layout components; combine the textual information with the visual information; and predict a reading order of the plurality of layout components based on the combined textual information and visual information using: a self-attention encoder configured to generate an embedding based on a first sequence associated with the plurality of layout components; and a self-attention decoder configured to generate a second sequence based on the embedding. ### Claim 17 An apparatus comprising: at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to: receive an image representing a document including a plurality of layout components; identify textual information associated with the plurality of layout components; identify visual information associated with the plurality of layout components; combine the textual information with the visual information; and predict a reading order of the plurality of layout components based on the combined textual information and visual information using: a self-attention encoder configured to generate an embedding based on a first sequence associated with the plurality of layout components; and a self-attention decoder configured to generate a second sequence based on the embedding. ### Claim 18 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the identifying of the visual information includes extracting visual-based data from the image, the extracting of the visual-based data includes using a neural network configured to generate an embedding including the visual information, the neural network includes a two-dimensional convolution operation, the embedding includes an array, and the array includes an element including the visual-based data associated with the plurality of layout components. ### Claim 19 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the identifying of the visual information includes extracting visual-based data from the image, the extracting of the visual-based data includes using a neural network configured to generate an embedding including the visual information, the neural network includes a plurality of two-dimensional convolution operations, and the embedding includes an array including an element including the visual-based data associated with an associated layout component and the visual-based data associated with at least one additional layout component. ### Claim 20 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the textual information is associated with a first embedding, the visual information is associated with a second embedding, and the combining of the textual information with the visual information includes concatenating the first embedding with the second embedding.