# US 12,118,294 B2 · Machine learning systems and methods for automatically tagging documents to enable accessibility to impaired individuals > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,294 B2 - Document ID: US12118294B2 - Status: Granted - Kind: B2 - Application number: 18/309,857 - Filed: 2023-05-01 - Published: 2023-10-05 - Granted: 2024-10-15 - Earliest claimed: 2020-02-14 - Assignees: Open Text Corporation - Applicants: Open Text Corporation - Inventors: David Comeau; Jeffrey Williams; Evgeny Kolesnikov; Michael Itkin; June Qiang; James Relunia; Brian Sue - Web page: https://patentgrep.com/patent/US12118294B2 ## CPC - G06F 40/16: Automatic learning of transformation rules, e.g. from examples - G06F 16/31: Indexing; Data structures therefor; Storage structures - G06F 40/154: Tree transformation for tree-structured or markup documents, e.g. XSLT, XSL-FO or stylesheets - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06V 30/413: Classification of content, e.g. text, photographs or tables - G06N 3/04: Architecture, e.g. interconnection topology - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems, methods, and products for auto tagging structured PDF documents that do not have accessibility tags. In one embodiment, structured PDF documents having accessibility tags are first parsed and analyzed to organize the visual components of the documents. The relationships of the identified objects to DOM elements (e.g., tags) are determined, and the objects and related DOM elements are stored in training files. The training files are used to train various classifiers. Untagged PDF documents are then parsed to identify included visual objects, and the classifiers are used to determine DOM elements that should be associated with visual objects identified in the untagged PDF documents. This information is used to construct a DOM structure corresponding to each untagged document. A new PDF is then generated corresponding to each untagged document using the generated DOM structure and visual object information. ## Claims ### Claim 1 A method comprising: obtaining a set of tagged pdf documents, each having a corresponding a document object model (DOM) structure; determining, for each tagged pdf document, relationships between graphical objects in the tagged pdf document and corresponding elements of the DOM structure of the tagged pdf document; generating, for each tagged pdf document, a corresponding training record identifying the determined relationships; training, using the training records, a machine learning model to determine DOM structure elements that are associated with graphical objects; obtaining a set of untagged PDF documents which do not contain corresponding DOM structures; and for each untagged PDF document, automatically generating, using the trained machine learning model, a corresponding tagged PDF document having one or more DOM structure elements corresponding to one or more graphical objects contained in the untagged PDF document. ### Claim 2 (depends on 1) The method of claim 1, further comprising pre-processing the tagged PDF documents to organize visual elements of each tagged PDF document into corresponding ones of the graphical objects, wherein the pre-processing of the tagged PDF documents includes: parsing the tagged PDF document to identify visual elements of the document; grouping the identified visual elements into visual objects; determining, for each of the visual objects, a corresponding visual bounding box; determining, for each of the visual objects, whether the visual object is a foreground element or a background element; and generating a cluster tree by performing a plurality of cuts which segment the PDF document into multiple visually separated pieces. ### Claim 3 (depends on 2) The method of claim 2, wherein grouping the identified visual elements into visual objects comprises grouping text elements and grouping image elements. ### Claim 4 (depends on 3) The method of claim 3, wherein the text elements comprise text characters and the text characters are grouped based on text size, font, position, and direction. ### Claim 5 (depends on 2) The method of claim 2, further comprising identifying the one or more graphical objects contained in each untagged PDF document in the same manner by which the one or more graphical objects contained in the tagged PDF document are identified. ### Claim 6 (depends on 2) The method of claim 2, wherein a size of the visual bounding box is different than an extent of the visual object. ### Claim 7 (depends on 2) The method of claim 2, further comprising pruning the cluster tree by recombining a plurality of leaves of the cluster tree. ### Claim 8 (depends on 2) The method of claim 2, wherein the processing of the tagged PDF documents includes: identifying groups of visual objects that are closely positioned visually in the PDF document; identifying elements of the DOM structure that correspond to the identified groups of visual objects and associating the identified elements of the DOM structure with the corresponding groups of visual objects; and identifying region segment features which are leaves on the cluster tree and storing indications of neighboring region segment features in the training records. ### Claim 9 (depends on 1) The method of claim 1, further comprising generating a plurality of training files, wherein each of the training files contains one or more of the generated training records, each training file corresponding to a distinct classifier of the machine learning model. ### Claim 10 (depends on 9) The method of claim 9, wherein the classifiers include: a text separator classifier; a cluster cut classifier; a cluster join classifier; a layout features classifier; a table cluster join classifier; a complex table cell classifier; and a region segment. ### Claim 11 A system comprising: a processor coupled to a memory that stores one or more instructions, the instructions executable by the processor to perform the method comprising: obtaining a set of tagged pdf documents, each having a corresponding a document object model (DOM) structure; determining, for each tagged pdf document, relationships between graphical objects in the tagged pdf document and corresponding elements of the DOM structure of the tagged pdf document; generating, for each tagged pdf document, a corresponding training record identifying the determined relationships; training, using the training records, a machine learning model to determine DOM structure elements that are associated with graphical objects; obtaining a set of untagged PDF documents which do not contain corresponding DOM structures; and for each untagged PDF document, automatically generating, using the trained machine learning model, a corresponding tagged PDF document having one or more DOM structure elements corresponding to one or more graphical objects contained in the untagged PDF document. ### Claim 12 (depends on 11) The system of claim 11, further comprising pre-processing the tagged PDF documents to organize visual elements of each tagged PDF document into corresponding ones of the graphical objects, wherein the pre-processing of the tagged PDF documents includes: parsing the tagged PDF document to identify visual elements of the document; grouping the identified visual elements into visual objects; determining, for each of the visual objects, a corresponding visual bounding box; determining, for each of the visual objects, whether the visual object is a foreground element or a background element; and generating a cluster tree by performing a plurality of cuts which segment the PDF document into multiple visually separated pieces. ### Claim 13 (depends on 12) The system of claim 12, wherein identifying the one or more graphical objects contained in the untagged PDF document is performed in the same manner by which the one or more graphical objects contained in the tagged PDF document are identified. ### Claim 14 (depends on 12) The system of claim 12, wherein the processing of the tagged PDF documents includes: identifying groups of visual objects that are closely positioned visually in the PDF document; identifying elements of the DOM structure that correspond to the identified groups of visual objects and associating the identified elements of the DOM structure with the corresponding groups of visual objects; and identifying region segment features which are leaves on the cluster tree and storing indications of neighboring region segment features in the training records. ### Claim 15 (depends on 11) The system of claim 11, further comprising generating a plurality of training files, wherein each of the training files contains one or more of the generated training records, each training file corresponding to a distinct classifier of the machine learning model, wherein the classifiers include one or more of: a text separator classifier; a cluster cut classifier; a cluster join classifier; a layout features classifier; a table cluster join classifier; a complex table cell classifier; and a region segment. ### Claim 16 A computer program product comprising a non-transitory computer-readable medium storing instructions executable by a processor to perform: obtaining a set of tagged pdf documents, each having a corresponding a document object model (DOM) structure; determining, for each tagged pdf document, relationships between graphical objects in the tagged pdf document and corresponding elements of the DOM structure of the tagged pdf document; generating, for each tagged pdf document, a corresponding training record identifying the determined relationships; training, using the training records, a machine learning model to determine DOM structure elements that are associated with graphical objects; obtaining a set of untagged PDF documents which do not contain corresponding DOM structures; and for each untagged PDF document, automatically generating, using the trained machine learning model, a corresponding tagged PDF document having one or more DOM structure elements corresponding to one or more graphical objects contained in the untagged PDF document. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein the pre-processing of the tagged PDF documents includes: parsing the tagged PDF document to identify visual elements of the document; grouping the identified visual elements into visual objects; determining, for each of the visual objects, a corresponding visual bounding box; determining, for each of the visual objects, whether the visual object is a foreground element or a background element; and generating a cluster tree by performing a plurality of cuts which segment the PDF document into multiple visually separated pieces. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein identifying the one or more graphical objects contained in the untagged PDF document is performed in the same manner by which the one or more graphical objects contained in the tagged PDF document are identified. ### Claim 19 (depends on 17) The computer program product of claim 17, wherein the processing of the tagged PDF documents includes: identifying groups of visual objects that are closely positioned visually in the PDF document; identifying elements of the DOM structure that correspond to the identified groups of visual objects and associating the identified elements of the DOM structure with the corresponding groups of visual objects; and identifying region segment features which are leaves on the cluster tree and storing indications of neighboring region segment features in the training records. ### Claim 20 (depends on 16) The computer program product of claim 16, further comprising generating a plurality of training files, wherein each of the training files contains one or more of the generated training records, each training file corresponding to a distinct classifier of the machine learning model, wherein the classifiers include one or more of: a text separator classifier; a cluster cut classifier; a cluster join classifier; a layout features classifier; a table cluster join classifier; a complex table cell classifier; and a region segment.