US 11,893,012 B1Granted

Content extraction using related entity group metadata from reference objects

AssigneeAmazon Technologies, Inc.

InventorsSrikar Appalaraju, Raghavan Manmatha, Bhargava Urala Kota

Life of this patent

Estimating the term…

Abstract

Representations of sets of descriptors of reference objects are stored in a repository, with individual descriptors including information about entities identified in the reference objects. In response to a request to extract content from a particular data object, a reference object which satisfies a similarity criterion with respect to the particular data object is identified from the repository using the descriptors. A structural comparison of the particular data object and the reference object is performed to determine an entity related to another entity identified in the particular data object.

Claims20

Dependent claims

A system, comprising:

one or more computing devices that implement an analytics service of a provider network;
wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices cause the one or more computing devices to:
store, in a repository of the analytics service, respective sets of descriptors corresponding to content identified within images of individual ones of one or more reference documents, wherein a particular descriptor comprises information about at least a text key within an image of a reference document;
in response to a first request to extract content from an image of a particular document, perform a first content extraction on the image of the particular document, to:
identify, using a first query directed to the repository, at least a first reference document, wherein the particular document is distinct from the one or more reference documents including the first reference document, and wherein a first set of descriptors of the first reference document meets a similarity criterion with respect to a second set of descriptors obtained from the image of the particular document, wherein the second set of descriptors is obtained using a first set of one or more machine learning models to which the image of the particular document has been provided as input, and wherein the first set of one or more machine learning models comprises an optical character recognition model;
and
based at least in part on a result of the first query, perform a layout-aware matching analysis of the particular document and the first reference document to project respective portions of the image of the particular document onto the first reference document, wherein the layout-aware matching analysis comprises computing an estimate of a difference in location of (a) a first key within a projected representation of the image of the particular document and (b) the first key within the first reference document, and the projected representation is used to extract a first predicted value corresponding to the first key from the particular document;
and
in response to a second request to extract content from an image of another document after the respective sets of descriptors have been stored, perform a second content extraction on the image of the other document, to:
execute a second query on the repository, wherein a result of the second query indicates that the repository does not comprise a reference document whose descriptors meet the similarity criterion with respect to descriptors obtained from the other document;
and
based at least in part on the result of the second query, utilize a second set of one or more machine learning models to extract, from the other document, a second predicted value corresponding to a second key, wherein the second set of one or more machine learning models comprises at least one model which is not used for the layout-aware matching analysis;
wherein the first content extraction achieves a performance improvement based at least in part on use of the layout-aware matching analysis of the particular document and the first reference document, to (a) improve an accuracy of extracted content or (b) reduce an amount of compute, memory, or storage resources used to extract content.

10 more elements

A computer-implemented method performed by an analytics service, comprising:

storing, in a repository, respective sets of descriptors associated with individual ones of one or more reference objects, wherein a particular descriptor of a particular set of descriptors comprises information pertaining to at least a key identified within a particular reference object;
in response to a first request to extract content from a particular data object, performing a content extraction on the particular data object, including:
identifying a first reference object using a result of a first query directed to the repository, wherein the particular data object is distinct from the one or more references objects including the first reference object and, wherein the first reference object meets a similarity criterion with respect to the particular data object, wherein a parameter of the first query is based at least in part on output generated by a first set of one or more machine learning models to which the particular data object has been provided as input, and wherein the result of the first query is based at least in part on analysis of one or more descriptors associated with the first reference object;
based at least in part on the result of a first query, performing a structural comparison of the particular data object and the first reference object to determine a projection of respective portions of the particular data object onto the first reference object;
extracting, using the projection, a first prediction of a particular value associated with a key in the particular data object;
and
transmitting the first prediction to one or more destinations;
wherein the content extraction achieves a performance improvement based at least in part on use of the structural comparison of the particular data object with the first reference object, to (a) improve an accuracy of extracted content or (b) reduce an amount of compute, memory, or storage resources used to extract content.

5 more elements

One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors implement an analytics service and cause the analytics service to:

store, in a repository, respective sets of descriptors associated with individual ones of one or more reference objects, wherein a particular descriptor of a particular set of descriptors comprises information pertaining to a particular entity identified within a particular reference object;
in response to a first request to extract content from a particular data object, perform a content extraction on the particular data object, to:
identify a first reference object using a result of a first query directed to the repository, wherein the particular data object is distinct from the one or more reference objects including the first reference object, wherein the first reference object meets a similarity criterion with respect to the particular data object, wherein a parameter of the first query is based at least in part on output generated by a first set of one or more machine learning models to which the particular data object has been provided as input, and wherein the result of the first query is based at least in part on analysis of one or more descriptors associated with the first reference object;
based at least in part on the result of a first query, perform a structural comparison of the particular data object and the first reference object to determine a projection of respective portions of the particular data object onto the first reference object;
extract, using the projection, a first prediction of another entity associated with the particular entity in the particular data object;
and
transmit the first prediction to one or more destination;
wherein the content extraction achieves a performance improvement based at least in part on use of the structural comparison of the particular data object with the first reference object, to (a) improve an accuracy of extracted content or (b) reduce an amount of compute, memory, or storage resources used to extract content.

5 more elements

Dependencies are inferred from the claim text.

patentgrep

Loading patent…

Abstract

A system, comprising:

one or more computing devices that implement an analytics service of a provider network;
wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices cause the one or more computing devices to:
store, in a repository of the analytics service, respective sets of descriptors corresponding to content identified within images of individual ones of one or more reference documents, wherein a particular descriptor comprises information about at least a text key within an image of a reference document;
in response to a first request to extract content from an image of a particular document, perform a first content extraction on the image of the particular document, to:
identify, using a first query directed to the repository, at least a first reference document, wherein the particular document is distinct from the one or more reference documents including the first reference document, and wherein a first set of descriptors of the first reference document meets a similarity criterion with respect to a second set of descriptors obtained from the image of the particular document, wherein the second set of descriptors is obtained using a first set of one or more machine learning models to which the image of the particular document has been provided as input, and wherein the first set of one or more machine learning models comprises an optical character recognition model;
and
based at least in part on a result of the first query, perform a layout-aware matching analysis of the particular document and the first reference document to project respective portions of the image of the particular document onto the first reference document, wherein the layout-aware matching analysis comprises computing an estimate of a difference in location of (a) a first key within a projected representation of the image of the particular document and (b) the first key within the first reference document, and the projected representation is used to extract a first predicted value corresponding to the first key from the particular document;
and
in response to a second request to extract content from an image of another document after the respective sets of descriptors have been stored, perform a second content extraction on the image of the other document, to:
execute a second query on the repository, wherein a result of the second query indicates that the repository does not comprise a reference document whose descriptors meet the similarity criterion with respect to descriptors obtained from the other document;
and
based at least in part on the result of the second query, utilize a second set of one or more machine learning models to extract, from the other document, a second predicted value corresponding to a second key, wherein the second set of one or more machine learning models comprises at least one model which is not used for the layout-aware matching analysis;
wherein the first content extraction achieves a performance improvement based at least in part on use of the layout-aware matching analysis of the particular document and the first reference document, to (a) improve an accuracy of extracted content or (b) reduce an amount of compute, memory, or storage resources used to extract content.

10 more elements