# US 2021/0064653 A1 · Predicting topics of potential relevance based on retrieved/created digital media files

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0064653 A1
- Document ID: US20210064653A1
- Status: Published application
- Kind: A1
- Application number: 17/097,110
- Filed: 2020-11-13
- Published: 2021-03-04
- Earliest claimed: 2018-06-21
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Robert Rose; Qun Cao
- Web page: https://patentgrep.com/patent/US20210064653A1

## CPC

- G06F 16/5838: {using colour}
- G06F 16/908: using metadata automatically derived from the content
- G06F 16/144: {Query formulation}
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Implementations are described herein for leveraging digital media files retrieved and/or created by users to predict/determine topics of potential relevance to the users. In various implementations, digital media file(s) created and/or retrieved by a user with a client device may be applied as input across trained machine learning model(s), which in some cases are local to the client device, to generate output that indicates object(s) detected in the digital media file(s). Data indicative of the indicated object(s) may be provided to a remote computing system without providing the digital media file(s) themselves. In some implementations, information associated with the indicated object(s) may be retrieved and proactively output to the user. In some implementations, a frequency at which objects occur across a corpus of digital media files may be considered when determining a likelihood that a detected object is potentially relevant to a user.

## Claims

### Claim 1

A method implemented using one or more processors of a client device operated by a user, comprising:
obtaining one or more digital media files created by the user with the client device;
at the client device, locally applying the one or more digital media files as input across one or more trained machine learning models that are stored in memory of the client device, wherein the one or more trained machine learning models generate output indicative of one or more objects detected in each of the one or more digital media files;
providing, to a remote computing system, data indicative of the detected one or more objects, without providing the one or more digital media files, wherein the providing causes the remote computing system to identify one or more topics of potential relevance to the user based on the detected one or more objects;
receiving, from the same remote computing system or a different remote computing system, information associated with the one or more topics; and
proactively outputting the information associated with the one or more topics to the user at the same client device or a different client device operated by the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein the providing causes the remote computing system to formulate a search query based on the one or more topics, and the information associated with the one or more topics comprises information that is responsive to the formulated search query.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
receiving, from the remote computing system, the search query; and
locally issuing the search query in order to receive, in response to the issuing, the information that is responsive to the search query.

### Claim 4 (depends on 2)

The method of claim 2, wherein the remote computing system uses the formulated search query itself to obtain the information that is responsive to the formulated search query.

### Claim 5 (depends on 1)

The method of claim 1, wherein the providing causes the remote computing system to:
determine a frequency at which each of the one or more objects occurs across a corpus of media files provided by a population of users; and
identify one or more topics of potential relevance to the user based on the frequencies at which the one or more objects occur across the corpus, wherein a frequency at which a given object occurs across the corpus is inversely related to a likelihood a topic is identified from the object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more digital media files comprise digital images.

### Claim 7 (depends on 6)

The method of claim 6, wherein one or more of the trained machine learning models comprises a convolutional neural network.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more digital media files comprise digital audio files, and the detected one or more objects comprise detected mentions of the one or more objects in the audio data.

### Claim 9

A client device comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to:
obtain one or more digital media files created by the user with the client device;
at the client device, locally apply the one or more digital media files as input across one or more trained machine learning models that are stored in memory of the client device, wherein the one or more trained machine learning models generate output indicative of one or more objects detected in each of the one or more digital media files;
provide, to a remote computing system, data indicative of the detected one or more objects, without providing the one or more digital media files, wherein the providing causes the remote computing system to identify one or more topics of potential relevance to the user based on the detected one or more objects;
receive, from the same remote computing system or a different remote computing system, information associated with the one or more topics; and
proactively output the information associated with the one or more topics to the user at the same client device or a different client device operated by the user.

### Claim 10 (depends on 9)

The client device of claim 9, wherein the data indicative of the detected one or more objects causes the remote computing system to formulate a search query based on the one or more topics, and the information associated with the one or more topics comprises information that is responsive to the formulated search query.

### Claim 11 (depends on 10)

The client device of claim 10, further comprising instructions to:
receive, from the remote computing system, the search query; and
locally issue the search query in order to receive the information that is responsive to the search query.

### Claim 12 (depends on 10)

The client device of claim 10, wherein the remote computing system uses the formulated search query itself to obtain the information that is responsive to the formulated search query.

### Claim 13 (depends on 9)

The client device of claim 9, wherein the data indicative of the detected one or more objects causes the remote computing system to:
determine a frequency at which each of the one or more objects occurs across a corpus of media files provided by a population of users; and
identify one or more topics of potential relevance to the user based on the frequencies at which the one or more objects occur across the corpus, wherein a frequency at which a given object occurs across the corpus is inversely related to a likelihood a topic is identified from the object.

### Claim 14 (depends on 9)

The client device of claim 9, wherein the one or more digital media files comprise digital images.

### Claim 15 (depends on 14)

The client device of claim 14, wherein one or more of the trained machine learning models comprises a convolutional neural network.

### Claim 16 (depends on 9)

The client device of claim 9, wherein the one or more digital media files comprise digital audio files, and the detected one or more objects comprise detected mentions of the one or more objects in the audio data.

### Claim 17

At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors of a client device, cause the one or more processors to perform the following operations:
obtaining one or more digital media files created by the user with the client device;
at the client device, locally applying the one or more digital media files as input across one or more trained machine learning models that are stored in memory of the client device, wherein the one or more trained machine learning models generate output indicative of one or more objects detected in each of the one or more digital media files;
providing, to a remote computing system, data indicative of the detected one or more objects, without providing the one or more digital media files, wherein the providing causes the remote computing system to identify one or more topics of potential relevance to the user based on the detected one or more objects;
receiving, from the same remote computing system or a different remote computing system, information associated with the one or more topics; and
proactively outputting the information associated with the one or more topics to the user at the same client device or a different client device operated by the user.

### Claim 18 (depends on 17)

The at least one non-transitory computer-readable medium of claim 17, wherein the providing causes the remote computing system to formulate a search query based on the one or more topics, and the information associated with the one or more topics comprises information that is responsive to the formulated search query.

### Claim 19 (depends on 18)

The at least one non-transitory computer-readable medium of claim 18, further comprising:
receiving, from the remote computing system, the search query; and
locally issuing the search query in order to receive, in response to the issuing, the information that is responsive to the search query.

### Claim 20 (depends on 18)

The at least one non-transitory computer-readable medium of claim 18, wherein the remote computing system uses the formulated search query itself to obtain the information that is responsive to the formulated search query.
