# US 12,222,975 B2 · Triggering execution of machine learning based prediction of document metadata > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,975 B2 - Document ID: US12222975B2 - Status: Granted - Kind: B2 - Application number: 18/096,994 - Filed: 2023-01-13 - Published: 2024-07-18 - Granted: 2025-02-11 - Earliest claimed: 2023-01-13 - Assignees: DocuSign, Inc. - Applicants: DocuSign, Inc. - Inventors: Kaushik Narayanan; David Matthew Wong; David Lange; Vinay Jethava; Qing Zheng; Mohammad Mehdi Ghanimifard; Pontus Lindstrom; Gowtham Rangarajan Raman; David DeBarr; Yan He - Web page: https://patentgrep.com/patent/US12222975B2 ## CPC - G06F 16/383: using metadata automatically derived from the content - G06F 16/34: Browsing; Visualisation therefor (browsing or visualisation for clustering or classification G06F16/358) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A system predicts metadata attributes associated with documents using machine learning models. The document may represent an interaction between entities. The system trains machine learning models to predict scores indicating whether a token or a sequence of token of a document represents a metadata attribute. The metadata prediction is used to annotate the document and display to users. The system receives user feedback via the user interface and uses the user feedback to evaluate or retrain the model. The system generates training data by receiving a set of annotated documents and comparing the annotated documents against other documents to identify matching documents. The system determines when to execute the machine learning based metadata prediction based on steps of document workflow executed by the system. ## Claims ### Claim 1 A computer-implemented method for machine learning based prediction of document metadata, the computer-implemented method comprising: execute, by a document management system, document workflows for one or more documents, wherein each of the one or more documents represents an interaction between a plurality of entities, each document workflow comprising a plurality of steps; receive one or more trigger criteria, each trigger criterion specifying conditions for triggering execution of machine learning based prediction of metadata attributes for a document; execute, for each document of the one or more documents: receiving status of execution of a particular step of document workflow for the document; evaluating a trigger criterion associated with the document based on the received status of execution of the particular step; responsive to the evaluation of the trigger criterion indicating that metadata prediction should be triggered, send request for prediction of metadata attributes based on the document by executing one or more machine learning models trained to predict a likelihood that a portion of an input document represents a metadata attribute describing an interaction between a plurality of entities, wherein the portion of the input document comprises a token or sequence of tokens from the document associated with the metadata attribute; annotate the document with one or more metadata attributes predicted using the one or more machine learning models; and cause for display, the annotated document via a user interface. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein a trigger criterion specifies a type of action associated with the document performed in a particular step of the document workflow. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the steps of the document workflow include one or more of: a document upload step, a document update step, a document signing step, an identity verification step, or a step for configuring and presenting a form for receiving information. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein a document workflow for a document is performed by execution of a document workflow specification comprising a sequence of steps associated with the document, the document workflow executed on a cloud platform. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: receive user feedback via the user interface, the user feedback comprising one or more of: (1) a correction of a predicted metadata attribute or (2) an approval of a predicted metadata attribute. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: use the user feedback for evaluation of the one or more machine learning models. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising: use the user feedback for generating training data for retraining the one or more machine learning models. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein a machine learning model predicts a score indicating that the portion of the input document represents a date associated with the interaction between the plurality of entities. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein a machine learning model predicts a role of an entity from the plurality of entities. ### Claim 10 A non-transitory computer-readable storage medium storing executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform operations comprising: execute, by a document management system, document workflows for one or more documents, wherein each of the one or more documents represents an interaction between a plurality of entities, each document workflow comprising a plurality of steps; receive one or more trigger criteria, each trigger criterion specifying conditions for triggering execution of machine learning based prediction of metadata attributes for a document; execute, for each document of the one or more documents: receive status of execution of a particular step of document workflow for the document; evaluate a trigger criterion associated with the document based on the received status of execution of the particular step; responsive to the evaluation of the trigger criterion indicating that metadata prediction should be triggered, send request for prediction of metadata attributes based on the document by executing one or more machine learning models trained to predict a likelihood that a portion of an input document represents a metadata attribute describing an interaction between a plurality of entities, wherein the portion of the input document comprises a token or sequence of tokens from the document associated with the metadata attribute; annotate the document with one or more metadata attributes predicted using the one or more machine learning models; and cause for display, the annotated document via a user interface. ### Claim 11 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein a trigger criterion specifies a type of action associated with the document performed in a particular step of the document workflow. ### Claim 12 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the steps of the document workflow include one or more of: a document upload step, a document update step, a document signing step, an identity verification step, or a step for configuring and presenting a form for receiving information. ### Claim 13 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein a document workflow for a document is performed by execution of a document workflow specification comprising a sequence of steps associated with the document, the document workflow executed on a cloud platform. ### Claim 14 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein the instructions further cause the one or more computer processors to perform steps comprising: receive user feedback via the user interface, the user feedback comprising one or more of: (1) a correction of a predicted metadata attribute or (2) an approval of a predicted metadata attribute. ### Claim 15 (depends on 14) The non-transitory computer-readable storage medium of claim 14, wherein the instructions further cause the one or more computer processors to perform steps comprising: use the user feedback for evaluation of the one or more machine learning models. ### Claim 16 (depends on 14) The non-transitory computer-readable storage medium of claim 14, wherein the instructions further cause the one or more computer processors to perform steps comprising: use the user feedback for generating training data for retraining the one or more machine learning models. ### Claim 17 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein a machine learning model predicts a score indicating that the portion of the input document represents a date associated with the interaction between the plurality of entities. ### Claim 18 (depends on 10) The non-transitory computer-readable storage medium of claim 10, wherein a machine learning model predicts a role of an entity from the plurality of entities. ### Claim 19 A computer system comprising: one or more computer processors; and a non-transitory computer-readable storage medium storing executable instructions that, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising: execute, by a document management system, document workflows for one or more documents, wherein each of the one or more documents represents an interaction between a plurality of entities, each document workflow comprising a plurality of steps; receive one or more trigger criteria, each trigger criterion specifying conditions for triggering execution of machine learning based prediction of metadata attributes for a document; execute, for each document of the one or more documents: receive status of execution of a particular step of document workflow for the document; evaluate a trigger criterion associated with the document based on the received status of execution of the particular step; responsive to the evaluation of the trigger criterion indicating that metadata prediction should be triggered, send request for prediction of metadata attributes based on the document by executing one or more machine learning models trained to predict a likelihood that a portion of an input document represents a metadata attribute describing an interaction between a plurality of entities, wherein the portion of the input document comprises a token or sequence of tokens from the document associated with the metadata attribute; annotate the document with one or more metadata attributes predicted using the one or more machine learning models; and cause for display, the annotated document via a user interface. ### Claim 20 (depends on 19) The computer system of claim 19, wherein the instructions further cause the one or more computer processors to perform steps comprising: receive user feedback via the user interface, the user feedback comprising one or more of: (1) a correction of a predicted metadata attribute or (2) an approval of a predicted metadata attribute.