Back to results
US 12,591,612 B1Granted
Multimodal web page content classification with knowledge graph-based explainability
AssigneePalo Alto Networks, Inc.
InventorsZeyu You, Wei Wang
Life of this patent
Estimating the term…
Abstract
A multimodal web page content classification system (“classification system”) uses summarization, extraction, and knowledge graph generation to classify web page content. For each of multiple modalities, the classification system summarizes web page content corresponding to that modality and then identifies key phrase/concept pairs in that modality summary. The classification system builds a knowledge graph from the key phrase/concept pairs as well as known concept/category pairs. Finally, the classification system invokes a classifier on a representation of the knowledge graph to obtain a category classification, resulting in an interpretable classification via the modality summaries and knowledge graph.
Claims20
A method comprising:
- generating a plurality of summaries of content of a web page, wherein each of the plurality of summaries corresponds to one of a plurality of modalities of web page content, wherein generating the plurality of summaries of the content for the web page comprises invoking a first large language model with a first prompt comprising a descriptor of each modality of the plurality of modalities of web page content and task instructions to summarize the modality for the content of the web page;
- annotating the plurality of summaries with one or more concepts corresponding to a plurality of phrases in the plurality of summaries, wherein annotating the plurality of summaries comprises invoking a transformer model to,
- identify the plurality of phrases in the plurality of summaries;
- and
- determine concepts in the one or more concepts corresponding to each of the plurality of phrases;
- generating a knowledge graph that represents relationships between the plurality of summaries, the one or more concepts, and a plurality of categories of web content,
- wherein each node in the knowledge graph indicates at least one of a summary in the plurality of summaries, a category in the plurality of categories, and a concept in the one or more concepts, and
- wherein each edge in the knowledge graph indicates at least one of,
- a relationship between a phrase of the plurality of phrases and a concept for the phrase in the one or more concepts, wherein the phrase is in a summary of the plurality of summaries that is represented by a node, and
- a relationship between a concept in the one or more concepts and a category in the plurality of categories;
- and
- categorizing the content of the web page based, at least in part, on the knowledge graph.
9 more elements
A non-transitory machine-readable medium having program code stored thereon, the program code comprising instructions to:
- generate a plurality of summaries of content of a web page, wherein each of the plurality of summaries corresponds to one of a plurality of modalities of web page content, wherein the instructions to generate the plurality of summaries of the content for the web page comprise instructions to invoke a first large language model with a first prompt comprising a descriptor of each modality of the plurality of modalities of web page content and task instructions to summarize the modality for the content of the web page;
- annotate the plurality of summaries with one or more concepts corresponding to a plurality of phrases in the plurality of summaries, wherein the instructions to annotate the plurality of summaries comprise instructions to invoke a transformer model to,
- identify the plurality of phrases in the plurality of summaries;
- and
- determine concepts in the one or more concepts corresponding to each of the plurality of phrases;
- build a knowledge graph that represents relationships between the plurality of summaries, the one or more concepts, and a plurality of categories of web content,
- wherein each node in the knowledge graph indicates at least one of a summary in the plurality of summaries, a category in the plurality of categories, and a concept in the one or more concepts, and
- wherein each edge in the knowledge graph indicates at least one of,
- a relationship between a phrase of the plurality of phrases and a concept for the phrase in the one or more concepts, wherein the phrase is in a summary of the plurality of summaries that is represented by a node, and
- a relationship between a concept in the one or more concepts and a category in the plurality of categories;
- and
- categorize the content of the web page based, at least in part, on the knowledge graph.
9 more elements
An apparatus comprising:
- a processor;
- and
- a machine-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to:
- generate a plurality of summaries of content of a web page, wherein each of the plurality of summaries corresponds to one of a plurality of modalities of web page content, wherein the instructions to generate the plurality of summaries of the content for the web page comprise instructions executable by the processor to cause the apparatus to invoke a first large language model with a first prompt comprising a descriptor of each modality of the plurality of modalities of web page content and task instructions to summarize the modality for the content of the web page;
- annotate the plurality of summaries with one or more concepts corresponding to a plurality of phrases in the plurality of summaries, wherein the instructions to annotate the plurality of summaries comprise instructions executable by the processor to cause the apparatus to invoke a transformer model to,
- identify the plurality of phrases in the plurality of summaries;
- and
- determine concepts in the one or more concepts corresponding to each of the plurality of phrases;
- build a knowledge graph that represents relationships between the plurality of summaries, the one or more concepts, and a plurality of categories of web content,
- wherein each node in the knowledge graph indicates at least one of a summary in the plurality of summaries, a category in the plurality of categories, and a concept in the one or more concepts, and
- wherein each edge in the knowledge graph indicates at least one of,
- a relationship between a phrase of the plurality of phrases and a concept for the phrase in the one or more concepts, wherein the phrase is in a summary of the plurality of summaries that is represented by a node, and
- a relationship between a concept in the one or more concepts and a category in the plurality of categories;
- and
- categorize the content of the web page based, at least in part, on the knowledge graph.
12 more elements
Dependencies are inferred from the claim text.