# US 12,008,057 B2 · Determining a visual theme in a collection of media items > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,057 B2 - Document ID: US12008057B2 - Status: Granted - Kind: B2 - Application number: 17/509,767 - Filed: 2021-10-25 - Published: 2022-11-17 - Granted: 2024-06-11 - Earliest claimed: 2021-05-11 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Kristina Bohl; Ivan Oropeza; Lily Berg; Tracy Gu; Ethan Schreiber; Shanfeng Zhang; Howard Zhou; David Hendon; Zhen Li; Futang Peng; Teresa Ko; Jason Chang - Web page: https://patentgrep.com/patent/US12008057B2 ## CPC - G06F 16/55: Clustering; Classification - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 16/45: Clustering; Classification - G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/906: Clustering; Classification - G06F 16/9538: Presentation of query results - G06F 40/30: Semantic analysis - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A media application determines, based on pixels of images or videos from a collection of media items, clusters of media items such that the media items in each cluster have a visual similarity, wherein the collection of media items is associated with a user account. The media application selects a subset of the clusters of media from corresponding clusters of media items based on the media items in each cluster having a visual similarity within a range of threshold similarity values. The media application causes a user interface to be displayed that includes the subset of the clusters of media. ## Claims ### Claim 1 A computer-implemented method comprising: determining, based on pixels of images or videos from a collection of media items, image embeddings for clusters of media items such that the media items in each cluster have a visual similarity, wherein: each media item is associated with a location and an associated timestamp, media items captured within a predetermined time period are associated with an episode, and the collection of media items is associated with a user account; selecting a subset of the clusters of media items based on: corresponding media items in each cluster having the visual similarity within a range of threshold visual similarity values; and corresponding associated timestamps such that the corresponding media items in the subset of the clusters of media items meet a temporal diversity criteria that excludes more than a first predetermined number of the corresponding media items from the episode; responsive to a number of the corresponding media items in the subset of clusters including more than a second predetermined number of media items removing one or more of the corresponding media items based on location such that the subset of the clusters of media items meets a location diversity criteria; causing a user interface to be displayed that includes the subset of the clusters of media items; receiving aggregated feedback from users for aggregated subsets of clusters of media items; providing the aggregated feedback to a machine-learning model, wherein parameters of the machine-learning model are updated based on the aggregated feedback; and modifying the image embeddings for the clusters of media items using the parameters of the machine-learning model with the updated parameters. ### Claim 2 (depends on 1) The method of claim 1, further comprising, prior to selecting the subset of the clusters of media items, excluding from the collection of media items, media items that are associated with a category that is in a list of prohibited categories. ### Claim 3 (depends on 1) The method of claim 1, further comprising prior to determining the clusters of media items, excluding media items that correspond to a category that is in a list of prohibited categories. ### Claim 4 (depends on 1) The method of claim 1, wherein the clusters of media items are further determined based on the corresponding media items being associated with labels that have a semantic similarity. ### Claim 5 (depends on 1) The method of claim 1, further comprising: scoring each media item in the subset of the clusters of media items based on analyzing a likelihood that a user associated with the user account performs a positive action with reference to the media item; and selecting the media items in the subset of the clusters of media items based on a corresponding score meeting a threshold score. ### Claim 6 (depends on 5) The method of claim 5, further comprising: receiving feedback from the user about one or more of the media items in the subset of the clusters of media items; and modifying a corresponding score for the one or more media items in the subset of the clusters of media items based on the feedback. ### Claim 7 (depends on 6) The method of claim 6, wherein the feedback includes an explicit action as indicated by removal of the one or more media items in the subset of the clusters of media items from the user interface or an implicit action as indicated by one or more of viewing the corresponding media items in the subset of the clusters of media items or sharing the corresponding media items in the subset of the clusters of media items. ### Claim 8 (depends on 1) The method of claim 1, further comprising: causing the user interface to be updated based on modifying the image embeddings for the clusters of media items. ### Claim 9 (depends on 1) The method of claim 1, further comprising selecting a particular media item from each cluster in the subset of the clusters of media items as a cover photo for the each cluster in the subset of the clusters of media items based on the particular media item including a greatest number of objects corresponding to the visual similarity. ### Claim 10 (depends on 1) The method of claim 1, further comprising adding a title to each cluster in the subset of the clusters of media items based on a type of visual similarity and a commonly used phrase. ### Claim 11 (depends on 1) The method of claim 1, wherein the subset of the clusters of media items is displayed in the user interface at predetermined intervals. ### Claim 12 (depends on 1) The method of claim 1, further comprising providing a notification to a user associated with the user account that the subset of the clusters of media items is available, wherein the notification includes a corresponding title for each of the clusters in the subset of the clusters of media items. ### Claim 13 (depends on 1) The method of claim 1, wherein: the determining comprises generating a vector representation of each media item using a trained machine-learning model; a vector distance between vector representations of pairs of media items is indicative of the visual similarity of the media items; the vector representation is an image embedding generated by the trained machine-learning model; and the clusters are selected such that the vector distance between each pair of media items within the cluster is outside the range of threshold visual similarity values. ### Claim 14 A computer-implemented method comprising: receiving as input to a trained machine-learning model, media items from a collection of media items associated with a user account, wherein media items are each associated with an associated timestamp and the media items that are captured within a predetermined time period are associated with an episode; generating, with the trained machine-learning model, output image embeddings for clusters of media items, wherein the media items in each cluster have a visual similarity and media items that have the visual similarity are nearer to each other in vector space than media items that are dissimilar, such that partitioning the vector space generates the clusters of media items and wherein each media item is associated with a location; selecting a subset of the clusters of media items based on: corresponding media items in each cluster having a visual similarity within a range of threshold visual similarity values; and corresponding associated timestamps such that the corresponding media items in the subset of the clusters of media items meet a temporal diversity criteria that excludes more than a first predetermined number of the corresponding media items from the episode; responsive to a number of the corresponding media items in the subset of clusters including more than a second predetermined number of media items, removing one or more of the corresponding media items based on location such that the subset of the clusters of media items meets a location diversity criteria; causing a user interface to be displayed that includes the subset of the clusters of media items; receiving aggregated feedback from users for aggregated subsets of clusters of media items; providing the aggregated feedback to a machine-learning model, wherein parameters of the machine-learning model are updated based on the aggregated feedback; and modifying the image embeddings for the clusters of media items using the parameters of the machine-learning model with the updated parameters. ### Claim 15 (depends on 14) The method of claim 14, wherein functional images are removed from the collection of media items before the collection of media items is provided to the trained machine-learning model. ### Claim 16 (depends on 14) The method of claim 14, wherein the aggregated feedback from users includes a reaction to a set of media items or a modification of a title of the set of media items. ### Claim 17 A system comprising: a processor; and a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising: determining, based on pixels of images or videos from a collection of media items, image embeddings for clusters of media items such that the media items in each cluster have a visual similarity, wherein: each media item is associated with a location and an associated timestamp, media items captured within a predetermined time period are associated with an episode, and the collection of media items is associated with a user account; selecting a subset of the clusters of media items based on: corresponding media items in each cluster having the visual similarity within a range of threshold visual similarity values; and corresponding associated timestamps such that the corresponding media items in the subset of the clusters of media items meet a temporal diversity criteria that excludes more than a first predetermined number of the corresponding media items from the episode; responsive to a number of the corresponding media items in the subset of clusters including more than a second predetermined number of media items, removing one or more of the corresponding media items based on location such that the subset of the clusters of media items meets a location diversity criteria; causing a user interface to be displayed that includes the subset of the clusters of media items; receiving aggregated feedback from users for aggregated subsets of clusters of media items; providing the aggregated feedback to a machine-learning model, wherein parameters of the machine-learning model are updated based on the aggregated feedback; and modifying the image embeddings for the clusters of media items using the parameters of the machine-learning model with the updated parameters. ### Claim 18 (depends on 17) The system of claim 17, wherein: each media item has an associated timestamp; the media items captured within a predetermined time period are associated with an episode; and selecting the subset of the clusters of media items is based on corresponding associated timestamps such that the corresponding media items in the subset of the clusters of media items meet a second temporal diversity criteria that excludes more than a predetermined number of the corresponding media items from a particular episode. ### Claim 19 (depends on 17) The system of claim 17, wherein the operations further comprise: prior to selecting the subset of the clusters of media items, excluding from the collection of media items, media items that are associated with a category that is in a list of prohibited categories. ### Claim 20 (depends on 17) The system of claim 17, wherein the clusters of media items are further determined based on the corresponding media items being associated with labels that have a semantic similarity.