# US 12,118,464 B2 · Joint embedding content neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,464 B2 - Document ID: US12118464B2 - Status: Granted - Kind: B2 - Application number: 17/820,657 - Filed: 2022-08-18 - Published: 2023-03-23 - Granted: 2024-10-15 - Earliest claimed: 2018-04-26 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Lawrence Jason Muhlstein; Leonardo Ribas Machado das Neves; Yanen Li; Ning Xu - Web page: https://patentgrep.com/patent/US12118464B2 ## CPC - G06F 16/9035: Filtering based on additional data, e.g. user or group profiles - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 16/22: Indexing; Data structures therefor; Storage structures - G06N 3/045: Combinations of networks - G06N 3/0499: Feedforward networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/091: Active learning - G06N 3/098: Distributed learning, e.g. federated learning - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A neural network system can select content based on user and item content embeddings in an approach that can be updated in real time on the user device without server support. Requests for content sent to the server can include an anonymous user embedding that includes data describing the user's inputs. The content that is nearest to the user embedding in a joint embedding space can be returned as suggested content. ## Claims ### Claim 1 A method performed on a user device, the method comprising: generating, using a user embedding neural network, a user embedding with user attribute data as a first input and user interactions as a second input, the user attribute data comprising at least one of: a gender of a user, an age of the user, or a geographic location of the user; receiving an indication of a user interaction performed on a content item; generating, using the user embedding neural network, an updated user embedding based on the user interaction and the content item; identifying one or more content items using the updated user embedding and content embeddings, wherein the content embeddings and the updated user embedding share a joint embedding space, and wherein the content embeddings are generated using a content embeddings neural network, wherein the content embeddings neural network comprises two or more content embedding neural networks, the two or more content embedding neural networks corresponding to two or more content types, and wherein the user embedding neural network is trained jointly with the two or more content embedding neural networks; and storing, on the user device, the identified one or more content items. ### Claim 2 (depends on 1) The method of claim 1 wherein the user embedding neural network is a feed forward (FF) neural network. ### Claim 3 (depends on 1) The method of claim 1 further comprising: updating, using a recurrent neural network, a user interaction representation based on the user interaction, the content item, and a hidden user interaction state. ### Claim 4 (depends on 1) The method of claim 1, wherein the one or more content items are identified by determining one or more content embeddings of the content embeddings as nearest neighbors to the updated user embedding in the joint embedding space, the one or more content embeddings corresponding to the one or more content items, wherein the one or more content items are input to the content embeddings neural network with the one or more content embeddings as output. ### Claim 5 (depends on 4) The method of claim 4, wherein determining the one or more content embeddings as the nearest neighbors comprises determining one or more distances from the updated user embedding to the one or more content embeddings in the joint embedding space. ### Claim 6 (depends on 1) The method of claim 1 wherein the one or more content items are identified by: transmitting, to a network platform, a request for content items nearest to the updated user embedding; and receiving, from the network platform, the one or more content items as nearest neighbors. ### Claim 7 (depends on 1) The method of claim 1, further comprising: displaying, on the user device, the one or more content items. ### Claim 8 (depends on 1) The method of claim 1, wherein the content item and the one or more content items comprise one or more of: an electronic document, an image, a sequence of images, a slide show, a video, or an audio file. ### Claim 9 (depends on 1) The method of claim 1, wherein the content embeddings are first content embeddings, and wherein the training is based on determining distances between a series of user embeddings and second content embeddings, the second content embeddings generated as output from the content embeddings neural network with input of second content items in which a user indicated an interest. ### Claim 10 (depends on 9) The method of claim 9 wherein the series of user embeddings are generated based on updating the user embedding based on interactions of the user with the second content items. ### Claim 11 (depends on 9) The method of claim 9 wherein the user is determined to have indicated the interest in the second content items based on the user performing, with each content item of the second content items, one or more of: reading the each content item to completion, viewing the each content item for a duration of time above a threshold duration, bookmarking the each content item, or saving the each content item. ### Claim 12 (depends on 11) The method of claim 11 wherein the training is further based on determining second distances between the series of user embeddings and third content embeddings, the third content embeddings generated as second output from the content embeddings neural network with a second input of third content items in which the user indicated a lack of interest. ### Claim 13 (depends on 12) The method of claim 12 wherein the duration of time is a first duration of time, the threshold duration is a first threshold duration, and wherein the user is determined to have indicated the lack of interest in the third content items based on the user performing, with each content item of the third content items, one or more of the following group: not selecting the each content item when presented with the each content item, not reading the each content item, viewing the each content item for a second duration of time below a second threshold duration, and selecting an interface item indicating that the user is not interested in the each content item. ### Claim 14 A non-transitory machine-readable storage device embodying instructions that, when executed by one or more processors of a user device, cause the one or more processors to perform operations comprising: generating, using a user embedding neural network, a user embedding with user attribute data as a first input and user interactions as a second input, the user attribute data comprising at least one of: a gender of a user, an age of the user, or a geographic location of the user; receiving an indication of a user interaction performed on a content item; generating, using the user embedding neural network, an updated user embedding based on the user interaction and the content item; identifying one or more content items using the updated user embedding and content embeddings, wherein the content embeddings and the updated user embedding share a joint embedding space, and wherein the content embeddings are generated using a content embeddings neural network, wherein the content embeddings neural network comprises two or more content embedding neural networks, the two or more content embedding neural networks corresponding to two or more content types, and wherein the user embedding neural network is trained jointly with the two or more content embedding neural networks; and storing, on the user device, the identified one or more content items. ### Claim 15 (depends on 14) The non-transitory machine-readable storage device of claim 14 wherein the user embedding neural network is a feed forward (FF) neural network. ### Claim 16 A user device comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: generating, using a user embedding neural network, a user embedding with user attribute data as a first input and user interactions as a second input, the user attribute data comprising at least one of: a gender of a user, an age of the user, or a geographic location of the user; receiving an indication of a user interaction performed on a content item; generating, using the user embedding neural network, an updated user embedding based on the user interaction and the content item; identifying one or more content items using the updated user embedding and content embeddings, wherein the content embeddings and the updated user embedding share a joint embedding space, and wherein the content embeddings are generated using a content embeddings neural network, wherein the content embeddings neural network comprises two or more content embedding neural networks, the two or more content embedding neural networks corresponding to two or more content types, and wherein the user embedding neural network is trained jointly with the two or more content embedding neural networks; and storing, on the user device, the identified one or more content items. ### Claim 17 (depends on 16) The user device of claim 16 wherein the user embedding neural network is a feed forward (FF) neural network. ### Claim 18 (depends on 16) The user device of claim 16 wherein the operations further comprise: updating, using a recurrent neural network, a user interaction representation based on the user interaction, the content item, and a hidden user interaction state.