# US 12,112,561 B2 · Interactive visual representation of semantically related extracted data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,561 B2 - Document ID: US12112561B2 - Status: Granted - Kind: B2 - Application number: 17/534,105 - Filed: 2021-11-23 - Published: 2023-05-25 - Granted: 2024-10-08 - Earliest claimed: 2021-11-23 - Assignees: Figma, Inc. - Applicants: FIGMA, Inc. - Inventors: Amy Pham Le; Joren Lauwers; Richard Vaughan Stebbing; Ankur Goyal - Web page: https://patentgrep.com/patent/US12112561B2 ## CPC - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06V 30/1916: {Validation; Performance evaluation} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques are disclosed to provide an interactive visual representation of semantically related extracted data. In various embodiments, a plurality of data entities are extracted from a file, each entity comprising a key-value pair. One or more sets of related entities, each set comprising an occurrence of a defined repeating type of entity set, are identified among the plurality of data entities. Data associating the one or more sets of related entities with the file and the defined repeating type of entity set are stored. ## Claims ### Claim 1 A system, comprising: a communication interface configured to receive a file; and a processor coupled to the communication interface and configured to: extract a plurality of data entities from the file, each entity comprising a key-value pair; identify among the plurality of data entities one or more sets of related entities, each set comprising an occurrence of a defined repeating type of entity set; and store data associating the one or more sets of related entities with the file and the defined repeating type of entity set. ### Claim 2 (depends on 1) The system of claim 1, wherein the processor is further configured to use the stored data associating the one or more sets of related entities with the file and the defined repeating type of entity set to provide an extracted data validation user interface in which each of the one or more sets of related entities is displayed as a corresponding row. ### Claim 3 (depends on 2) The system of claim 2, wherein the extracted data validation user interface displays data extracted from a plurality of files and for each file the extracted data validation user interface includes a display of one or more rows, each row corresponding to and displaying extracted values associated with a corresponding set of related entities. ### Claim 4 (depends on 3) The system of claim 3, wherein extracted data validation user interface provides one or more features associated with a spreadsheet application. ### Claim 5 (depends on 3) The system of claim 3, wherein the extracted data validation user interface includes for at least a subset of entities displayed in at least a subset of rows a visual indication of a state of validation of one or both of a data value and a label associated with that entity. ### Claim 6 (depends on 5) The system of claim 5, wherein the visual indication includes a color code reflecting a degree of confidence. ### Claim 7 (depends on 5) The system of claim 5, wherein the extracted data validation user interface includes for each of at least said subset of entities displayed in at least said subset of rows a user-selectable control to access a file data validation interface for one or both of the entity and a corresponding one of said plurality of files with which the entity is associated. ### Claim 8 (depends on 7) The system of claim 7, wherein the file data validation interface includes a first region in which data representing the corresponding one of said plurality of files is displayed and a second region that includes an interactive data validation element to validate one or both of a data value and a label associated with an entity comprising the file. ### Claim 9 (depends on 8) The system of claim 8, wherein said interactive data validation element is displayed in response to selection of a representation of said entity within the first region. ### Claim 10 (depends on 8) The system of claim 8, wherein the processor is configured to receive via the interactive data validation element a user input indicating validation of one or both of the data value and the label associated with the entity; and the processor is further configured to update the extracted data validation user interface to reflect validation of the entity. ### Claim 11 (depends on 10) The system of claim 10, wherein the file comprises a first file included in said plurality of files and the validated entity comprises a first entity; and wherein the processor is configured to update the extracted data validation user interface to reflect validation of the first entity; and to update a second entity associated with a second file based on a reevaluation in real time of a confidence associated with a one or both of a predicted value and a predicted label associated with the second entity, the reevaluation being determined based at least in part on the validation of the first entity. ### Claim 12 (depends on 10) The system of claim 10, wherein the processor is further configured to update a data extraction model based at least in part on the validation of the entity. ### Claim 13 (depends on 12) The system of claim 12, wherein the processor is configured to update the data extraction model at least in part by providing data associated with validation of the entity to a machine learning process or module. ### Claim 14 (depends on 1) The system of claim 1, wherein the processor is further configured to receive a definition of said repeating type of entity set. ### Claim 15 (depends on 14) The system of claim 14, wherein the definition is received via a data extraction configuration user interface and wherein the processor is further configured to display the data extraction configuration user interface. ### Claim 16 (depends on 15) The system of claim 15, wherein the data extraction configuration user interface enables a user to select an instance of a set of related entities as represented in a displayed representation of a file with which the set of related entities is associated and to define the repeating type of entity set in relation to the selected instance of the set of related entities. ### Claim 17 A method, comprising: extracting a plurality of data entities from a file, each entity comprising a key-value pair; identifying among the plurality of data entities one or more sets of related entities, each set comprising an occurrence of a defined repeating type of entity set; and storing data associating the one or more sets of related entities with the file and the defined repeating type of entity set. ### Claim 18 (depends on 17) The method of claim 17, further comprising using the stored data associating the one or more sets of related entities with the file and the defined repeating type of entity set to provide an extracted data validation user interface in which each of the one or more sets of related entities is displayed as a corresponding row. ### Claim 19 (depends on 18) The method of claim 18, wherein the extracted data validation user interface displays data is extracted from a plurality of files and for each file the extracted data validation user interface includes a display of one or more rows, each row corresponding to and displaying extracted values associated with a corresponding set of related entities. ### Claim 20 A computer program product embodied in a non-transitory computer readable medium, comprising computer instructions for: extracting a plurality of data entities from a file, each entity comprising a key-value pair; identifying among the plurality of data entities one or more sets of related entities, each set comprising an occurrence of a defined repeating type of entity set; and storing data associating the one or more sets of related entities with the file and the defined repeating type of entity set.