Multimodal table extraction and semantic search in a machine learning platform for structuring data in organizations
AssigneeExIService Holdings, Inc.
InventorsChaithanya Manda, Anupam Kumar, Solmaz Torabi, Raman Kumar, Anish Goswami, Sidhant Agarwal, Md Sharique, Diksha Malhotra, Garimella Venkata BhanuTeja, Arvind Singh, Pavan Praneeth
Life of this patent
Estimating the term…
Abstract
Systems, methods, and computer-readable media for computer-assisted output validation in machine learning/artificial intelligence platforms are disclosed. An application instance includes one or more machine learning models used to generate searchable data structures based on multimodal inputs.
Claims19
A provider computing system associated with a provider entity and comprising at least one processor, at least one memory, and one or more non-transitory computer readable media, excluding transitory signals, storing instructions, which when executed by the at least one processor, perform operations for generating responses to natural-language queries regarding items in unstructured documents, the operations comprising:
- receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;
- performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising extracting, by a trained neural network, alphanumeric data from the unstructured data by generating a plurality of bounding boxes, each of the plurality of bounding boxes corresponding to a detected cell in a plurality of cells;
- generating a searchable data structure comprising the extracted alphanumeric data;
- using the searchable data structure, performing operations comprising:
- identifying, in the unstructured data, an image comprising a globally applicable item related to the alphanumeric data;
- extracting, from the image, the globally applicable item;
- causing the searchable data structure to store the globally applicable item relationally to each row in the alphanumeric data;
- generating a query response by performing, by a semantic similarity model, a semantic search using the globally applicable item or at least an additional portion of the searchable data structure;
- determining a set of coordinates in the unstructured document, wherein the set of coordinates corresponds to a query response item included in the alphanumeric data;
- including the determined set of coordinates in the query response;
- and
- transmitting query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.
A method for generating responses to natural-language queries regarding items in unstructured documents, the method comprising:
- receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;
- performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising generating an optimized model input comprising extracting alphanumeric data from the at least a portion of the document;
- generating a searchable data structure comprising the alphanumeric data;
- and
- for each of the at least one parsed document section and using the searchable data structure,
- identifying, in the unstructured data, an image comprising a globally applicable item related to the alphanumeric data;
- extracting, from the image, the globally applicable item;
- causing the searchable data structure to store the globally applicable item relationally to each row in the alphanumeric data;
- generating a query response by performing, by a semantic similarity model, a semantic search using the globally applicable item or at least an additional portion of the searchable data structure;
- and
- transmitting the query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.
One or more non-transitory computer readable media excluding transitory signals, the media storing instructions, which when executed by at least one processor, perform operations for generating responses to natural-language queries regarding items in unstructured documents, the operations comprising:
- receiving, at an application instance communicatively coupled to a subscriber computing system of a plurality of subscriber computing systems, a query and a document comprising unstructured data;
- performing pre-processing operations on at least a portion of the document comprising the unstructured data, the pre-processing operations comprising generating an optimized model input comprising at least one parsed document section that includes alphanumeric data by:
- using a computer vision machine learning model,
- detecting a first image in the unstructured data, wherein the first image comprises the alphanumeric data;
- generating a bounding box to encapsulate the first image;
- parsing, from the first image, a second image comprising the alphanumeric data, the second image defined by the bounding box;
- and
- extracting, by a trained neural network, the alphanumeric data from the second image;
- generating a searchable data structure comprising the alphanumeric data;
- and
- for each of the at least one parsed document section and using the searchable data structure,
- identifying, in the unstructured data, an image comprising a globally applicable item related to the alphanumeric data;
- extracting, from the image, the globally applicable item;
- causing the searchable data structure to store the globally applicable item relationally to each row in the alphanumeric data;
- generating a query response by performing, by a semantic similarity model, a semantic search using the globally applicable item or at least an additional portion of the searchable data structure;
- and
- transmitting the query response to a target application operated or hosted at least in part by the subscriber computing system or a subscriber entity associated with the subscriber computing system.
Dependencies are inferred from the claim text.