# US 2022/0035856 A1 · Computing Systems and Methods for Cataloging, Retrieving, and Organizing User-Generated Content Associated with Objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0035856 A1
- Document ID: US20220035856A1
- Status: Published application
- Kind: A1
- Application number: 17/314,345
- Filed: 2019-01-31
- Published: 2022-02-03
- Earliest claimed: 2018-11-07
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ariel Gilder; Piotr Zielinski; Alexandros Panagopoulos
- Web page: https://patentgrep.com/patent/US20220035856A1

## CPC

- G06F 16/434: {using image data, e.g. images, photos, pictures taken by a user}
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 16/55: Clustering; Classification
- G06F 16/583: using metadata automatically derived from the content
- G06F 16/5854: {using shape and object relationship}
- G06F 16/587: using geographical or spatial information, e.g. location
- G06F 16/908: using metadata automatically derived from the content
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure provides computing systems and methods for cataloging, retrieving, and/or organizing user-generated content associated with objects. Aspects of the disclosure are directed to a systems and methods which utilize computers to enable users to interact with libraries of user-generated content associated with cataloged objects. For example, a user can capture one or more images of a real-world object, label or otherwise annotate the object with various types of user-generated content and organize the object and its associated content into one or more libraries. The user-generated content can then be provided to other users upon the receipt of images of the same object or an object displaying similar features.

## Claims

### Claim 1

A computing system for cataloging and retrieving user-generated data associated with objects, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instruction that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving a dataset that comprises one or more images captured by a user that depict an object;
processing, using an artificial neural network, the dataset to generate an embedding associated with the object;
accessing one or more libraries of a plurality of libraries of user-generated content, wherein each of the plurality of libraries comprises one or more cataloged embeddings respectively associated with one or more cataloged objects, wherein each cataloged embedding has been generated from one or more images of the associated cataloged object captured by one or more other users, wherein one or more items of user-generated content are associated with each cataloged object, and wherein the one or more libraries are a subset of the plurality of libraries;
identifying, based at least in part on a comparison of the embedding with one or more cataloged embeddings included in the one or more libraries, a closest cataloged embedding of the one or more cataloged embeddings included in the one or more libraries; and
providing, for presentation to the user, at least one item of user-generated content that is associated with the closest cataloged embedding of the one or more cataloged embeddings included in the one or more libraries.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein:
receiving the dataset that comprises the one or more images comprises receiving, via a pre-defined application programming interface, the dataset that comprises the one or more images; and
providing the at least item of user-generated content associated with the closest cataloged embedding comprises providing, via the pre-defined application programming interface, the at least item of user-generated content associated with the closest cataloged embedding for presentation to the user.

### Claim 3 (depends on 1)

The computing system of claim 1 wherein:
the dataset further comprises data descriptive of a current location of the user at a time the one or more images were captured; and
accessing the one or more libraries of the plurality of libraries comprises selecting the one or more libraries from the plurality of libraries based at least in part on the current location of the user at the time the one or more images were captured.

### Claim 4 (depends on 1)

The computing system of claim 1 wherein the dataset comprises a video stream and the computing system performs the operations in real-time as the video stream is being captured.

### Claim 5 (depends on 1)

The computing system of claim 1 wherein the at least one item of user-generated content that is associated with the closest cataloged embedding comprises a virtual layer to be rendered upon imagery of the object in an augmented reality display.

### Claim 6 (depends on 1)

The computing system of claim 1 wherein the at least one item of user-generated content that is associated with the closest cataloged embedding comprises one or more of: a textual annotation, audio annotation, or a visual annotation supplied by one of the other users.

### Claim 7 (depends on 1)

The computing system of claim 1 wherein:
the dataset comprises a plurality of images captured by the user that depict the object; and
processing, using the artificial neural network, the dataset to generate the embedding associated with the object comprises:
respectively processing, using the artificial neural network, the plurality of images to respectively generate a plurality of proto-embeddings for the object; and
aggregating the plurality of proto-embeddings to generate the embedding associated with the object.

### Claim 8 (depends on 1)

The computing system of claim 1 wherein accessing the one or more libraries of the plurality of libraries of user-generated content comprises:
receiving a user input that selects the one or more libraries of the plurality of libraries of user-generated content; and
in response to the user input, accessing the one or more libraries from a data storage location.

### Claim 9 (depends on 1)

The computing system of claim 1 wherein:
the plurality of libraries of user-generated content are respectively associated with the plurality of other users and each of the other users is able to control a set of access permissions for the respective library associated with such other user; and
accessing the one or more libraries of the plurality of libraries of user-generated content comprises accessing only libraries for which the user has been granted permission to access.

### Claim 10 (depends on 1)

The computing system of claim 1 wherein:
the one or more processors and the one or more non-transitory computer-readable media are included in one or more server computing devices that implement a user-generated content platform that serves user-generated content to a plurality of client devices that are physically distinct from the one or more server computing devices;
receiving the dataset that comprises the one or more images comprises receiving, from a user computing device associated with the user via a communications network, the dataset that comprises the one or more images that were captured by the user computing device; and
providing the at least item of user-generated content associated with the closest cataloged embedding comprises providing, to the user computing device via the communications network, the at least item of user-generated content associated with the closest cataloged embedding for presentation to the user by the user computing device.

### Claim 11 (depends on 1)

The computing system method of claim 1, wherein the one or more processors and the one or more non-transitory computer-readable media are included in a user computing device that implements an on-device user-generated content platform that serves, via an application programming interface, user-generated content to one or more client applications included on the user computing device that are distinct and separate from the on-device user-generated content platform.

### Claim 12 (depends on 1)

The computing system method of claim 1, wherein the one or more processors and the one or more non-transitory computer-readable media are included in a user computing device that implements an application that includes an on-device user-generated content platform that serves user-generated content to the application.

### Claim 13 (depends on 12)

The computing system method of claim 12, wherein accessing one or more libraries of a plurality of libraries of user-generated content further comprises downloading at least one of the one or more libraries to the user computing device.

### Claim 14 (depends on 1)

The computing system method of claim 1, wherein identifying the closest cataloged embedding comprises searching a hierarchical database, wherein the hierarchical database includes the cataloged embeddings.

### Claim 15 (depends on 1)

The computing system method of claim 1, wherein processing the dataset to generate the embedding associated with the object comprises using an object-class-specific embedding model to generate the embedding, wherein the object-class-specific embedding model is specific to a class of the object.

### Claim 16

A computer implemented method for cataloging and retrieving user-generated data associated with objects, the method comprising:
receiving, by one or more computing devices, a dataset that comprises one or more images captured by a user that depict an object;
processing, by one or more computing devices using a neural network, the dataset to generate an embedding associated with the object;
accessing, by the one or more computing devices, one or more libraries of a plurality of libraries, wherein each of the plurality of libraries comprises one or more cataloged embeddings respectively associated with one or more cataloged objects, wherein each cataloged embedding has been generated from one or more images of the associated cataloged object; wherein one or more items of content are associated with each embedding, and wherein the one or more libraries are a subset of the plurality of libraries; and
associating, by one or more computing devices, the embedding with one of the cataloged objects, wherein associating the embedding with one of the cataloged objects produces a new cataloged embedding that is accessible by the one or more computing devices.

### Claim 17 (depends on 16)

The computer implemented method of claim 16, wherein the one or more computing devices comprise a user device, and wherein at least one of the actions of receiving, processing, accessing, and associating are performed on the user device.

### Claim 18 (depends on 16)

The computer implemented method of claim 16, further comprising detecting, by one or more computing devices, an item of inappropriate content using a Spam filter.

### Claim 19 (depends on 16)

The computer implemented method of claim 16, wherein associating the embedding with one of the cataloged objects comprises searching a hierarchical database, and wherein the hierarchical database includes the cataloged embeddings.

### Claim 20

A computing system, comprising:
one or more data storage devices that store a plurality of libraries of user-generated content, wherein each of the plurality of libraries comprises one or more cataloged embeddings respectively associated with one or more cataloged objects, wherein each cataloged embedding has been generated from one or more images of the associated cataloged object captured by one or more users, wherein one or more items of user-generated content are associated with each embedding; and
one or more computing devices that implement a user-generated content platform that:
receives imagery of an object from a client;
uses an artificial neural network to generate an embedding for the object;
searches one or more of the plurality of libraries to identify a cataloged object that has a cataloged embedding that is closest to the embedding for the object; and
serves to the client at least one item of user-generated content associated with the identified cataloged object.
