# US 10,372,821 B2 · Identification of reading order text segments with a probabilistic language model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,821 B2 - Document ID: US10372821B2 - Status: Granted - Kind: B2 - Application number: 15/462,684 - Filed: 2017-03-17 - Published: 2018-09-20 - Granted: 2019-08-06 - Earliest claimed: 2017-03-17 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Walter Chang; Trung Bui; Pranjal Daga; Michael Kraley; Hung Bui - Web page: https://patentgrep.com/patent/US10372821B2 ## CPC - G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking - G06F 40/114: Pagination - G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154) - G06F 40/131: Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces - G06F 40/151: Transformation - G06F 40/216: using statistical methods - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Certain embodiments identify a correct structured reading-order sequence of text segments extracted from a file. A probabilistic language model is generated from a large text corpus to comprise observed word sequence patterns for a given language. The language model measures whether splicing together a first text segment with another continuation text segment results in a phrase that is more likely than a phrase resulting from splicing together the first text segment with other continuation text segments. Sets of text segments, which include a first set with a first text segment and a first continuation text segment as well as a second set with the first text segment and a second continuation text segment, are provided to the probabilistic model. A score indicative of a likelihood of the set providing a correct structured reading-order sequence is obtained for each set of text segments. ## Claims ### Claim 1 A computer implemented method for identifying correct structured reading-order sequences of text segments extracted from a file, the method including one or more processing devices performing operations comprising: accessing a probabilistic language model of observed word sequence patterns that computes (i) a first likelihood that a phrase obtained by splicing a text segment together with a continuation text segment has a correct reading order and (ii) a second likelihood that an additional phrase obtained by splicing the text segment together with another continuation text segment has the correct reading order, wherein the probabilistic language model is (i) trained with tokenized data obtained from a first text corpus for a given language and (ii) validated against a second text corpus distinct from the first text corpus, the tokenized data comprising tokens respectively corresponding to sets of textually encoded characters generated from the first text corpus; providing, to the probabilistic language model, (i) a first set of text segments having a first text segment and a corresponding first continuation text segment and (ii) a second set of text segments having the first text segment and a corresponding second continuation text segment; computing, with the probabilistic language model, a first score and a second score, the first score indicating a likelihood of the first text segment and the corresponding first continuation text segment providing a correct structured reading-order sequence, the second score indicating a likelihood of the first text segment and the corresponding second continuation text segment providing the correct structured reading-order sequence; and outputting, based on the first score and the second score, a reading order text extraction for sequencing a set of text strings on a display device. ### Claim 2 (depends on 1) The computer implemented method of claim 1, wherein the probabilistic language model comprises an n-gram model trained on a text corpus. ### Claim 3 (depends on 1) The computer implemented method of claim 1, wherein the probabilistic language model comprises a recurrent neural network trained on a text corpus. ### Claim 4 (depends on 3) The computer implemented method of claim 3, wherein the recurrent neural network comprises a long short term memory system that learns language patterns over entire sentences. ### Claim 5 (depends on 4) The computer implemented method of claim 4, the operations further comprising reducing memory consumption by the long short term memory system by sampling selected target words from a noise distribution. ### Claim 6 (depends on 4) The computer implemented method of claim 4 wherein an output of the long short term memory system is normalized by a length of a corresponding set of text segments used to generate the output. ### Claim 7 (depends on 1) The computer implemented method of claim 1, further comprising identifying a set of text segments having the first text segment and the corresponding first continuation text segment as the correct structured reading-order sequence of text segments based on the first score being higher than the second score. ### Claim 8 A computing system comprising: a processing unit; and a non-transitory computer-readable medium communicatively coupled to the processing unit, wherein the processing unit is configured for executing program code stored on the non-transitory computer-readable medium and thereby performing operations comprising: accessing a probabilistic language model of observed word sequence patterns that computes (i) a first likelihood that a phrase obtained by splicing a text segment together with a continuation text segment has a correct reading order and (ii) a second likelihood that an additional phrase obtained by splicing the text segment together with another continuation text segment has the correct reading order, wherein the probabilistic language model is (i) trained with tokenized data obtained from a first text corpus for a given language and (ii) validated against a second text corpus distinct from the first text corpus, the tokenized data comprising tokens respectively corresponding to sets of textually encoded characters generated from the first text corpus, providing, to the probabilistic language model, (i) a first set of text segments having a first text segment and a corresponding first continuation text segment and (ii) a second set of text segments having the first text segment and a corresponding second continuation text segment, computing, with the probabilistic language model, a first score and a second score, the first score indicating a likelihood of the first text segment and the corresponding first continuation text segment providing a correct structured reading-order sequence, the second score indicating a likelihood of the first text segment and the corresponding second continuation text segment providing the correct structured reading-order sequence, and outputting, based on the first score and the second score, a reading order text extraction for sequencing a set of text strings on a display device. ### Claim 9 (depends on 8) The computing system of claim 8, wherein the probabilistic language model comprises a recurrent neural network trained on a text corpus. ### Claim 10 (depends on 9) The computing system of claim 9, wherein the recurrent neural network comprises a long short term memory system that learns language patterns over entire sentences. ### Claim 11 (depends on 10) The computing system of claim 10, the operations further comprising reducing memory consumption by the long short term memory system by sampling selected target words from a noise distribution. ### Claim 12 (depends on 10) The computing system of claim 10 wherein an output of the long short term memory system is normalized by a length of a corresponding set of text segments used to generate the output. ### Claim 13 (depends on 8) The computing system of claim 8, further comprising identifying a set of text segments having the first text segment and the corresponding first continuation text segment as a correct structured reading-order sequence of text segments based on the first score being higher than the second score. ### Claim 14 A non-transitory computer-readable medium having executable program code stored thereon that, when executed by a processing unit, configures the processing unit to perform operations comprising: accessing a probabilistic language model of observed word sequence patterns that computes (i) a first likelihood that a phrase obtained by splicing a text segment together with a continuation text segment has a correct reading order and (ii) a second likelihood that an additional phrase obtained by splicing the text segment together with another continuation text segment has the correct reading order, wherein the probabilistic language model is (i) trained with tokenized data obtained from a first text corpus for a given language and (ii) validated against a second text corpus distinct from the first text corpus, the tokenized data comprising tokens respectively corresponding to sets of textually encoded characters generated from the first text corpus; providing, to the probabilistic language model, (i) a first set of text segments having a first text segment and a corresponding first continuation text segment and (ii) a second set of text segments having the first text segment and a corresponding second continuation text segment; computing, with the probabilistic language model, a first score and a second score, the first score indicating a likelihood of the first text segment and the corresponding first continuation text segment providing a correct structured reading-order sequence, the second score indicating a likelihood of the first text segment and the corresponding second continuation text segment providing the correct structured reading-order sequence; and outputting, based on the first score and the second score, a reading order text extraction for sequencing a set of text strings on a display device. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the probabilistic language model comprises a recurrent neural network trained on a text corpus, wherein the recurrent neural network comprises a long short term memory system that learns language patterns over entire sentences. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, the operations further comprising reducing memory consumption by the long short term memory system by sampling selected target words from a noise distribution, wherein an output of the long short term memory system is normalized by a length of a corresponding set of text segments used to generate the output.