# US 2023/0267277 A1 · Systems and methods for using document activity logs to train machine-learned models for determining document relevance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267277 A1
- Document ID: US20230267277A1
- Status: Published application
- Kind: A1
- Application number: 18/010,727
- Filed: 2020-06-15
- Published: 2023-08-24
- Earliest claimed: 2020-06-15
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Weize Kong; Michael Bendersky; Marc Najork; Rama Kumar Pasumarthi; Zhen Qin; Rolf Jagerman
- Web page: https://patentgrep.com/patent/US20230267277A1

## CPC

- G06F 16/95: Retrieval from the web
- G06F 40/30: Semantic analysis
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 16/9537: Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
- G06F 16/9538: Presentation of query results
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods of the present disclosure are directed to a method for training a machine-learned semantic matching model. The method can include obtaining a first and second document and a first and second activity log. The method can include determining, based on the first document activity log and the second document activity log, a relation label indicative of whether the documents are related. The method can include inputting the documents into the model to receive a semantic similarity value representing an estimated semantic similarity between the first document and the second document. The method can include evaluating a loss function that evaluates a difference between the relation label and the semantic similarity value. The method can include modifying values of parameters of the model based on the loss function.

## Claims

### Claim 1

A computer-implemented method for training a machine-learned semantic matching model, the method comprising:
obtaining, by one or more computing devices, a first document, a first document activity log associated with the first document, a second document, and a second document activity log associated with the second document;
determining, by the one or more computing devices based at least in part on the first document activity log and the second document activity log, a relation label indicative of whether the first document and the second document are related;
inputting, by the one or more computing devices, the first document and the second document into the machine-learned semantic matching model to receive, from the machine-learned semantic matching model, a semantic similarity value representing an estimated semantic similarity between the first document and the second document;
evaluating, by the one or more computing devices, a loss function that evaluates a difference between the relation label and the semantic similarity value; and
modifying, by the one or more computing devices, one or more values of one or more parameters of the machine-learned semantic matching model based on the loss function.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the machine-learned semantic matching model is configured to generate the semantic similarity value representing the estimated semantic similarity between the first document and the second document by:
determining a first content embedding for the first document based on at least a portion of content of the first document;
determining a second content embedding for the second document based on at least a portion of content of the second document; and
generating, based on the first content embedding and the second content embedding, the semantic similarity value representing the estimated semantic similarity between the first document and the second document.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the content of the first document and the content of the second document respectively comprise at least one of:
first image data and second image data;
first video data and second video data;
first textual data and second textual data; or
first file data and second file data.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein:
the content of the first document comprises first textual content data and the content of the second document comprises second textual data;
the first content embedding comprises a first textual embedding and the second content embedding comprises a second textual embedding; and
determining the first textual embedding and the second textual embedding comprises, for each of the first textual embedding and the second textual embedding:
selecting one or more character subsets from the textual data of the corresponding document based at least in part on an appearance frequency for each character subset of a plurality of character subsets of the textual data of the document, each of the plurality of character subsets comprising one or more characters from the text of the document; and
averaging the one or more character subsets to determine the textual embedding.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the first document activity log and the second document activity log respectively describe one or more access events associated with the first document and the second document, each of the one or more access events comprising an access type and an access time.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the access type comprises:
a document sharing event;
a document opening event;
a document renaming event;
a document annotation event;
a document moving event;
a document downloading event;
a document editing event; or
a document creation event.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein the relation label is further based at least in part on a difference in time between a first document access time described by the first document activity log and a second document access time described by the second document activity log.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6, wherein the relation label is further based at least in part on a first document access type described by the first document activity log and a second document access type described by the second document activity log.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the relation label comprises a scalar value indicative of a degree of relatedness between the first document and the second document.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining, by the one or more computing devices, a search query from a user and a search result document corresponding to the search query;
inputting, by the one or more computing devices, the search query and the search result document to the machine-learned semantic matching model to obtain a second semantic similarity value representing an estimated semantic similarity between the search query and the search result document; and
ranking, by the one or more computing devices, based on the second semantic similarity value, the search result document among a plurality of ranked search result documents corresponding to the search query.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising providing, by the one or more computing devices, the plurality of ranked search result documents to the user.

### Claim 12

A computing system for determining semantic similarity between documents, comprising:
one or more processors;
a machine-learned semantic matching model, the model configured to:
obtain a first document and a second document, wherein the first document and second document are different and distinct from one another; and
generate, based on the first document and the second document, a semantic similarity value representing an estimated semantic similarity between the first document and the second document; and
one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining a search query;
in response to obtaining the search query, retrieving a search result document corresponding to the search query;
inputting the search query and the search result document into the machine-learned semantic matching model, wherein the machine-learned semantic matching model has been trained based at least in part on training data comprising one or more document pairs and one or more respectively associated relation labels, each of the one or more relation labels generated based on a comparison between a pair of activity logs for the document pair respectively associated with the relation label;
receiving, from the machine-learned semantic matching model, the semantic similarity value representing the estimated semantic similarity between the search query and the search result document; and
ranking, based at least in part on the semantic similarity value, the search result document among a plurality of ranked search result documents.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein the operations further comprise providing the plurality of ranked search result documents to a user.

### Claim 14 (depends on 12)

The computing system of claim 12, wherein the operations further comprise reclustering the plurality of ranked search results based on the semantic similarity value.

### Claim 15 (depends on 12)

The computing system of claim 12, wherein content of the search result document comprises at least one of:
image data;
video data;
textual data; or
file data.

### Claim 16 (depends on 12)

The computing system of claim 12, wherein the machine-learned semantic matching model is configured to generate, based on the first document and the second document, the semantic similarity value by:
obtaining the first document comprising the search query and the second document comprising the search result document;
determining a first content embedding for the first document based on at least a portion of content of the first document;
determining a second content embedding for the second document based on at least a portion of content of the second document; and
generating, based on the first content embedding and the second content embedding, the semantic similarity value representing the estimated semantic similarity between the first document and the second document.

### Claim 17 (depends on 16)

The computing system of claim 16, wherein:
the content of the first document comprises first textual data and the content of the second document comprises second textual data;
the first content embedding comprises a first textual embedding and the second content embedding comprises a second textual embedding; and
determining the first textual embedding and the second textual embedding comprises, for each of the first textual embedding and the second textual embedding:
selecting one or more character subsets from the textual data of the corresponding document based at least in part on an appearance frequency for each character subset of a plurality of character subsets of the textual data of the document, each of the plurality of character subsets comprising one or more characters from the text of the document;
mapping each of the one or more selected character subsets to one or more respective n-gram embeddings; and
averaging the one or more n-gram embeddings to determine the textual embedding.

### Claim 18

One or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining a first document, a first document activity log associated with the first document, a second document, and a second document activity log associated with the second document;
determining, based at least in part on the first document activity log and the second document activity log, a relation label indicative of whether the first document and the second document are related;
inputting the first document and the second document into a machine-learned semantic matching model to receive, from the machine-learned semantic matching model, a semantic similarity value representing an estimated semantic similarity between the first document and the second document;
evaluating a loss function that evaluates a difference between the relation label and the semantic similarity value; and
modifying one or more values of one or more parameters of the machine-learned semantic matching model based on the loss function.

### Claim 19 (depends on 18)

The one or more tangible, non-transitory computer readable media of claim 18, wherein the machine-learned semantic matching model is configured to generate the semantic similarity value representing the estimated semantic similarity between the first document and the second document by:
determining a first content embedding for the first document based on at least a portion of content of the first document;
determining a second content embedding for the second document based on at least a portion of content of the second document; and
generating, based on the first content embedding and the second content embedding, the semantic similarity value representing the estimated semantic similarity between the first document and the second document.

### Claim 20 (depends on 19)

The one or more tangible, non-transitory computer readable media of claim 19, wherein:
the content of the first document comprises first textual content data and the content of the second document comprises second textual data;
the first content embedding comprises a first textual embedding and the second content embedding comprises a second textual embedding; and
determining the first textual embedding and the second textual embedding comprises, for each of the first textual embedding and the second textual embedding:
selecting one or more character subsets from the textual data of the document based at least in part on an appearance frequency for each character subset of a plurality of character subsets of the textual data of the document, each of the plurality of character subsets comprising one or more characters from the text of the document;
mapping each of the one or more selected character subsets to one or more respective n-gram embeddings; and
averaging the one or more n-gram embeddings to determine the textual embedding.
