# US 2014/0208191 A1 · Grouping Fixed Format Document Elements to Preserve Graphical Data Semantics After Reflow > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0208191 A1 - Document ID: US20140208191A1 - Status: Published application - Kind: A1 - Application number: 13/745,279 - Filed: 2013-01-18 - Published: 2014-07-24 - Earliest claimed: 2013-01-18 - Assignees: Microsoft Corporation - Applicants: Microsoft Corporation - Inventors: Drazen Zaric; Milan Sesum; Milos Lazarevic; Aljosa Obuljen - Web page: https://patentgrep.com/patent/US20140208191A1 ## CPC - G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191) - G06F 40/10: Text processing (natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40) - G06F 40/131: Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces - G06K 15/1814: {including print-ready data, i.e. data already matched to the printing process} - G06F 40/106: Display of layout of documents; Previewing - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS ## Abstract Determining relationships between graphical elements in a fixed format document is provided. Graphical element sizes and their relative positions may be analyzed to determine whether two or more graphical elements should be aggregated together or whether the graphical elements should belong to different graphical groups. Graphs and figures comprising objects that are absolutely positioned may be detected, as well as objects where inter-element positions need to be preserved from regular document flow. Additionally, background objects may be differentiated from regular text flow when the objects overlap with text. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim 21 A method for converting a fixed format document containing a plurality of graphical elements into a flow format document, the method comprising: detecting a plurality of graphical elements on a page in a fixed format document; determining that a first graphical element is proximate to a second graphical element; and displaying the first graphical element and the second graphical element on the page as a first graphic aggregation in the flow format document. ### Claim 22 (depends on 21) The method of claim 21, wherein a graphical element comprises a path, an image, or text. ### Claim 23 (depends on 21) The method of claim 21, wherein determining that a first graphical element is proximate to a second graphical element further comprises: determining a shortest distance between the first graphical element and the second graphical element; and determining the shortest distance between the first graphical element and the second graphical element is less than a determined join threshold. ### Claim 24 (depends on 23) The method of claim 23, wherein determining a shortest distance between a first graphical element and a second graphical element comprises determining a shortest distance between a bounding box of the first graphical element and a bounding box of the second graphical element. ### Claim 25 (depends on 21) The method of claim 21, further comprising: determining that a line of text is proximate to the first graphic aggregation; and grouping the line of text with the first graphic aggregation in the flow format document. ### Claim 26 (depends on 25) The method of claim 25, wherein determining that a line of text is proximate to the first graphic aggregation comprises detecting that more than a predefined amount of the line of text is inside a bounding box of the first aggregation. ### Claim 27 (depends on 21) The method of claim 21 further comprising merging a first graphic aggregation and a second graphic aggregation if the first graphic aggregation and the second graphic aggregation intersect. ### Claim 27 (depends on 21) The method of claim 21, further comprising: determining a line of text semantically belongs with a graphical element in the first graphic aggregation; and grouping the line of text with the first graphic aggregation. ### Claim 28 (depends on 27) The method of claim 27, wherein merging the first graphic aggregation and the second graphic aggregation comprises: determining a number of graphical elements in the first graphic aggregation; determining a number of graphical elements in the second graphic aggregation; and moving the graphical elements from the graphic aggregation comprising fewer graphical elements to the graphic aggregation comprising more graphical elements. ### Claim 28 (depends on 27) The method of claim 27, wherein determining the line of text semantically belongs with a graphical element in the first graphic aggregation comprises: comparing a text line above or below the first graphic aggregation to a bounding box of the first graphic aggregation; determining the text line does not extend horizontally past the bounding box of the graphic aggregation by more than a predefined width; determining a vertical distance between the text line and the bounding box of the graphic aggregation is less than a predefined value; and expanding the graphic aggregation to include the text line. ### Claim 29 (depends on 27) The method of claim 27, wherein determining the line of text semantically belongs with a graphical element in the first graphic aggregation comprises: comparing a text line on a left side or a right side of the first graphic aggregation to a bounding box of the first graphic aggregation; determining the text line has a vertical overlap with the bounding box of the graphic aggregation; determining the text line does not extend horizontally past the bounding box of the graphic aggregation by more than a predefined width; and expanding the graphic aggregation bounding box to include the text line. ### Claim 30 A system for converting a fixed format document containing a plurality of graphical elements into a flow format document, the system comprising: one or more processors; and a memory coupled to the one or more processors, the one or more processors operable to: detect a plurality of graphical elements on a page in a fixed format document; determine that a first graphical element is proximate to a second graphical element; and display the first graphical element and the second graphical element on the page as a first graphic aggregation in the flow format document. ### Claim 31 (depends on 30) The system of claim 30, wherein a graphical element comprises a path, an image, or text. ### Claim 32 (depends on 30) The system of claim 30, wherein in determining that a first graphical element is proximate to a second graphical element, the one or more processors are further operable to: determine a shortest distance between the first graphical element and the second graphical element; and determine the shortest distance between the first graphical element and the second graphical element is less than a determined join threshold. ### Claim 33 (depends on 32) The system of claim 32, wherein in determining a shortest distance between a first graphical element and a second graphical element, the one or more processors are operable to determine a shortest distance between a bounding box of the first graphical element and a bounding box of the second graphical element. ### Claim 34 (depends on 30) The system of claim 30, wherein the one or more processors are further operable to: determine that a line of text is proximate to the first graphic aggregation; and group the line of text with the first graphic aggregation in the flow format document. ### Claim 35 (depends on 34) The system of claim 34, wherein in determining that a line of text is proximate to the first graphic aggregation, the one or more processors are further operable to detect that more than a predefined amount of the line of text is inside a bounding box of the first aggregation. ### Claim 36 (depends on 30) The system of claim 30, wherein the one or more processors are further operable to: determine that the first graphic aggregation and a second graphic aggregation intersect; and merge the first graphic aggregation and the second graphic aggregation into a single aggregation. ### Claim 37 (depends on 30) The system of claim 30, wherein the one or more processors are further operable to: determine a line of text semantically belongs with a graphical element in the first graphic aggregation; and group the line of text with the first graphic aggregation. ### Claim 38 A computer readable medium containing computer executable instructions which, when executed by a computer, perform a method for converting a fixed format document containing a plurality of graphical elements into a flow format document, the method comprising: detecting a plurality of graphical elements on a page in a fixed format document; determining that a first graphical element is proximate to a second graphical element; and displaying the first graphical element and the second graphical element on the page as a first graphic aggregation in the flow format document. ### Claim 39 (depends on 38) The computer readable medium of claim 38, the method further comprising: determining a line of text semantically belongs with a graphical element in the first graphic aggregation; and grouping the line of text with the first graphic aggregation. ### Claim 40 (depends on 38) The computer readable medium of claim 38, the method further comprising: determining that the first graphic aggregation and a second graphic aggregation intersect; and merging the first graphic aggregation and the second graphic aggregation into a single aggregation.