# US 2023/0054068 A1 · Systems and methods for abstractive document summarization with entity coverage control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0054068 A1 - Document ID: US20230054068A1 - Status: Published application - Kind: A1 - Application number: 17/589,522 - Filed: 2022-01-31 - Published: 2023-02-23 - Earliest claimed: 2021-08-06 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Haopeng Zheng; Semih Yavuz; Wojciech Kryscinski; Kazuma Hashimoto; Yingbo Zhou - Web page: https://patentgrep.com/patent/US20230054068A1 ## CPC - G06F 16/345: {Summarisation for human users} - G06F 40/166: Editing, e.g. inserting or deleting - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/08: Learning methods - G06N 20/00: Machine learning - G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154) - G06F 40/279: Recognition of textual entities - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Embodiments described herein provide document summarization systems and methods that utilize fine-tuning of pre-trained abstractive summarization models to produce summaries that more faithfully track the content of the documents. Such abstractive summarization models may be pre-trained using a corpus consisting of pairs of articles and associated summaries. For each article-summary pair, a pseudo label or control code is generated and represents a faithfulness of the summary with respect to the article. The pre-trained model is then fine-tuned based on the article-summary pairs and the corresponding control codes. The resulting fine-tuned models then provide improved faithfulness in document summarization tasks. ## Claims ### Claim 1 A method for abstractive summarization of a document, the method comprising: receiving, via a data interface, a training dataset comprising a plurality of articles and a plurality of summaries corresponding to the plurality of articles; generating a plurality of article-summary pairs by pairing each article with at least one associated summary; computing, for an article-summary pair, an entity coverage precision metric based on a number of entity mentions in a corresponding summary or a corresponding article; determining a pseudo label indicating a faithfulness level of the corresponding article and the corresponding summary based on the computed entity coverage precision metric; prepending the article with the determined pseudo label as a training input to a summarization model; generating, by the summarization model, an output summary conditioned on both the article and the prepended pseudo label; and updating the summarization model based on a training objective comparing the output summary and the corresponding summary. ### Claim 2 (depends on 1) The method of claim 1, wherein the summarization model is based at least in part on a Bidirectional and Auto-Regressive Transformer (BART) summarization model. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of articles comprise a first article and a second article, each associated with a domain-specific database. ### Claim 4 (depends on 1) The method of claim 1, wherein the plurality of articles comprises an encyclopedia article. ### Claim 5 (depends on 4) The method of claim 4, further comprising: generating the output summary conditioned on the encyclopedia article and the prepended pseudo label. ### Claim 6 (depends on 1) The method of claim 1, wherein the entity coverage precision metric between the summary and the article is computed as a first number of entities mentioned by both the summary and the article divided by a second number of entities mentioned by the summary. ### Claim 7 (depends on 1) The method of claim 1, wherein the determining the pseudo label indicating the faithfulness level of the corresponding article and the corresponding summary based on the at least one computed entity coverage precision metric for the training dataset comprises: computing a plurality of entity coverage precision metrics corresponding to the plurality of article-summary pairs; binning the plurality of entity coverage precision metrics to determine a plurality of binned pseudo labels; computing an entity coverage precision metric for the corresponding article and the corresponding summary; and assigning a binned pseudo label of the plurality of binned pseudo labels to the corresponding article and the corresponding summary based on the entity coverage precision metric for the corresponding article and the corresponding summary. ### Claim 8 A system for abstractive summarization of a document, the system comprising: a non-transitory memory; and one or more processor coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising: receiving, via a data interface, a training dataset comprising a plurality of articles and a plurality of summaries corresponding to the plurality of articles; generating a plurality of article-summary pairs by pairing each article with at least one associated summary; computing, for an article-summary pair, an entity coverage precision metric based on a number of entity mentions in a corresponding summary or a corresponding article; determining a pseudo label indicating a faithfulness level of the corresponding article and the corresponding summary based on the computed entity coverage precision metric; prepending the article with the determined pseudo label as a training input to a summarization model; generating, by the summarization model, an output summary conditioned on both the article and the prepended pseudo label; and updating the summarization model based on a training objective comparing the output summary and the corresponding summary. ### Claim 9 (depends on 8) The system of claim 8, the plurality of articles comprise a first article and a second article, each associated with a domain-specific database. ### Claim 10 (depends on 8) The system of claim 8, wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with a domain-specific database. ### Claim 11 (depends on 8) The system of claim 8, wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with an encyclopedia article. ### Claim 12 (depends on 11) The system of claim 11, wherein the operations further comprise generating the output summary conditioned on the encyclopedia article and the prepended pseudo label. ### Claim 13 (depends on 8) The system of claim 8, wherein the entity coverage precision metric between the summary and the article is computed as a first number of entities mentioned by both the summary and the article divided by a second number of entities mentioned by the summary. ### Claim 14 (depends on 8) The system of claim 8, wherein the determining the pseudo label indicating the faithfulness level of the corresponding article and the corresponding summary based on the at least one computed entity coverage precision metric for the training dataset comprises: computing a plurality of entity coverage precision metrics corresponding to the plurality of article-summary pairs; binning the plurality of entity coverage precision metrics to determine a plurality of binned pseudo labels; computing an entity coverage precision metric for the corresponding article and the corresponding summary; and assigning a binned pseudo label of the plurality of binned pseudo labels to the corresponding article and the corresponding summary based on the entity coverage precision metric for the corresponding article and the corresponding summary. ### Claim 15 A non-transitory, machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising: receiving, via a data interface, a training dataset comprising a plurality of articles and a plurality of summaries associated with the plurality of articles; generating a plurality of article-summary pairs by pairing each article with at least one associated summary; computing, for an article-summary pair, an entity coverage precision metric based on a number of entity mentions in a corresponding summary or a corresponding article; determining a pseudo label indicating a faithfulness level of the corresponding article and the corresponding summary based on the computed entity coverage precision metric; prepending the article with the determined pseudo label as a training input to a summarization model; generating, by the summarization model, an output summary conditioned on both the article and the prepended pseudo label; and updating the summarization model based on a training objective comparing the output summary and the corresponding summary. ### Claim 16 (depends on 15) The non-transitory, machine-readable medium of claim 15, wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with a domain-specific database. ### Claim 17 (depends on 15) The non-transitory, machine-readable medium of claim 15, wherein the training dataset comprises a plurality of articles and a plurality of summaries each associated with an encyclopedia article. ### Claim 18 (depends on 18) The non-transitory, machine-readable medium of claim 18, wherein the operations further comprise generating the plurality of summaries associated with the encyclopedia article. ### Claim 19 (depends on 15) The non-transitory, machine-readable medium of claim 15, wherein the entity coverage precision metric between the summary and the article is computed as a first number of entities mentioned by both the summary and the article divided by a second number of entities mentioned by the summary. ### Claim 20 (depends on 15) The non-transitory, machine-readable medium of claim 15, wherein the determining the pseudo label indicating the faithfulness level of the corresponding article and the corresponding summary based on the at least one computed entity coverage precision metric for the training dataset comprises: computing a plurality of entity coverage precision metrics corresponding to the plurality of article-summary pairs; binning the plurality of entity coverage precision metrics to determine a plurality of binned pseudo labels; computing an entity coverage precision metric for the corresponding article and the corresponding summary; and assigning a binned pseudo label of the plurality of binned pseudo labels to the corresponding article and the corresponding summary based on the entity coverage precision metric for the corresponding article and the corresponding summary.