# US 2020/0288204 A1 · Generating and providing personalized digital content in real time based on live user context > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0288204 A1 - Document ID: US20200288204A1 - Status: Published application - Kind: A1 - Application number: 16/293,076 - Filed: 2019-03-05 - Published: 2020-09-10 - Earliest claimed: 2019-03-05 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Brian Duersch; Kristopher Paries - Web page: https://patentgrep.com/patent/US20200288204A1 ## CPC - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 18/23213: with fixed number of clusters, e.g. K-means clustering - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions} - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 40/161: {Detection; Localisation; Normalisation} - H04N 21/2187: Live feed - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/251: {Learning process for intelligent management, e.g. learning user preferences for recommending movies (details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08)} - H04N 21/4223: Cameras (H04N23/00 takes precedence) - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)} - H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user} - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure relates to generating personalized digital content in real time based on a live user context. For example, the disclosed systems can collect a stream of digital media comprising a digital video portraying a user while the user accesses one or more websites via a client device. The disclosed systems can then analyze the digital video to identify characteristics of the user portrayed in the digital video and/or to identify objects portrayed in the digital video. The disclosed systems can then utilize a context-based machine learning model to select digital content to provide to the user based on the identified characteristics and/or objects. While the user accesses the one or more websites, the disclosed systems can modify the one or more websites to include the selected subset of digital content and provide the modified one or more websites for display via the client device. ## Claims ### Claim 1 In a digital medium environment for collecting live user context data, a computer-implemented method for generating personalized digital content in real time, comprising: performing a step for training a context-based digital content machine learning model to generate digital content in response to training user contexts; collecting a stream of digital media comprising a digital video portraying a user while the user accesses one or more websites via a client device; analyzing the digital video to identify characteristics of the user portrayed in the digital video; and performing a step for utilizing the context-based digital content machine learning model to generate digital content for display while the user accesses the one or more websites via the client device based on the identified characteristics of the user. ### Claim 2 (depends on 1) The method of claim 1, wherein the context-based digital content machine learning model comprises a reinforcement learning model trained to increase a machine learning training reward in providing digital content in response to the training user contexts. ### Claim 3 (depends on 1) The method of claim 1, wherein the context-based digital content machine learning model comprises a neural network trained based on training user contexts and ground truth user results. ### Claim 4 (depends on 1) The method of claim 1, wherein the context-based digital content machine learning model comprises a scene compatibility manager that generates the digital content for display based on a scene compatibility score between a scene portrayed in the digital video and a repository of digital content. ### Claim 5 A non-transitory computer readable storage medium comprising instructions that, when executed by at least one processor, cause a computing device to: collect a stream of digital media comprising a digital video portraying a user while the user accesses one or more web sites via a client device; and generate personalized digital content to provide for display to the user within the one or more websites based on the digital video portraying the user while the user accesses the one or more websites by: analyzing the digital video utilizing a facial detection model and an object detection model to identify characteristics of the user portrayed in the digital video; utilizing a context-based digital content machine learning model to select a subset of digital content from a repository of digital content based on the identified characteristics of the user, wherein the context-based digital content machine learning model comprises a neural network trained to increase a machine learning training reward determined from a predict user response to proposed digital content based on training digital content, training user contexts, and ground truth responses; and while the user accesses the one or more websites: modifying the one or more web sites to include the subset of digital content; and providing the modified one or more websites for display via the client device. ### Claim 6 (depends on 5) The non-transitory computer readable storage medium of claim 5, wherein the facial detection model comprises an attention controlled neural network trained based on image triplets, characteristic attention projections, and a triplet-loss function. ### Claim 7 (depends on 5) The non-transitory computer readable storage medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the personalized digital content by: utilizing the object detection model to analyze the digital video to identify an object portrayed in the digital video; and utilizing the context-based digital content machine learning model to select the subset of digital content from the repository of digital content based on the identified object portrayed in the digital video and the identified characteristics of the user from the digital video portraying the user while the user accesses the one or more websites via the client device. ### Claim 8 (depends on 5) The non-transitory computer readable storage medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to train the context-based digital content machine learning model by: utilizing the context-based digital content machine learning model to generate a predicted user result based on a first training user context and a first training digital content; determining a loss by comparing the predicted user result to a ground truth using a loss function; and modifying parameters of the context-based digital content machine learning model based on the determined loss. ### Claim 9 (depends on 8) The non-transitory computer readable storage medium of claim 8, wherein the characteristics of the user comprise a gaze of the user, and further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the personalized digital content by: utilizing the facial detection model to identify a digital content element of the one or more websites associated with the gaze of the user; and while the user accesses the one or more websites, modifying the digital content element. ### Claim 10 (depends on 9) The non-transitory computer readable storage medium of claim 9, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the personalized digital content by: utilizing an audio detection model to identify a voice command from the user while the user accesses the one or more websites; and while the user accesses the one or more websites, modifying the digital content element based on the voice command. ### Claim 11 (depends on 5) The non-transitory computer readable storage medium of claim 5, wherein the stream of digital media further comprises audio content providing audio associated with the user while the user accesses the one or more websites, and further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the personalized digital content by: utilizing an audio detection model to identify additional characteristics of the user from the audio content; and utilizing the context-based digital content machine learning model to select the subset of digital content from the repository of digital content based on the additional characteristics of the user. ### Claim 12 (depends on 5) The non-transitory computer readable storage medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to: generate a digital characteristics report based on the identified characteristics of the user from the digital video portraying the user while the user accesses the one or more websites via the client device; and while the user accesses the one or more websites, modify the one or more websites to further include the digital characteristics report based on the identified characteristics of the user from the digital video portraying the user while the user accesses the one or more websites via the client device. ### Claim 13 (depends on 5) The non-transitory computer readable storage medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the computing device to: receive a manual input from the user via the client device while the user accesses the one or more web sites; and generate the personalized digital content by utilizing the context-based digital content machine learning model to select the subset of digital content from the repository of digital content based on the manual input from the user via the client device while the user accesses the one or more web sites and the identified characteristics of the user from the digital video portraying the user while the user accesses the one or more websites via the client device. ### Claim 14 A system comprising: at least one processor; and at least one non-transitory computer readable storage medium storing instructions that, when executed by the at least one processor, cause the system to: collect a stream of digital media comprising a digital video portraying a user while the user accesses one or more websites via a client device; and generate personalized digital content to provide for display to the user within the one or more websites based on the digital video portraying the user while the user accesses the one or more websites by: analyzing the digital video utilizing a facial detection model to identify facial characteristics of the user portrayed in the digital video; analyzing the digital video utilizing an object detection model comprising a neural network classifier to identify an object portrayed in the digital video; utilizing a context-based digital content machine learning model to select a subset of digital content from a repository of digital content based on the identified facial characteristics of the user and the identified object, wherein the context-based digital content machine learning model comprises a reinforcement learning model trained to select digital content based on machine learning training rewards determined from user responses resulting from providing to provided digital content; and while the user accesses the one or more websites: modifying the one or more web sites to include the subset of digital content; and providing the modified one or more websites for display via the client device. ### Claim 15 (depends on 14) The system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the system to train the context-based digital content machine learning model by: utilizing the context-based digital content machine learning model to generate proposed digital content based on a training user context; identifying a machine learning training reward associated with the proposed digital content; and modifying the context-based digital content machine learning model based on the machine learning training reward. ### Claim 16 (depends on 14) The system of claim 14, wherein the facial characteristics of the user comprise at least one of an emotion of the user, a gender of the user, an age of the user, apparel of the user, or a gaze of the user. ### Claim 17 (depends on 14) The system of claim 14, wherein the object portrayed in the digital video comprises at least one of a hand-held object held by the user, a background object, an additional person, clothing, an animal, or a picture of the object. ### Claim 18 (depends on 14) The system of claim 14, wherein the stream of digital media further comprises audio content providing audio associated with the user while the user accesses the one or more websites, and further comprising instructions that, when executed by the at least one processor, cause the system to generate the personalized digital content by utilizing the context-based digital content machine learning model to select the subset of digital content from the repository of digital content based on additional characteristics of the user identified by analyzing the audio content. ### Claim 19 (depends on 18) The system of claim 18, wherein the additional characteristics of the user comprise at least one of an emotion of the user or a tone of voice of the user. ### Claim 20 (depends on 14) The system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the system to: generate a digital characteristics report based on the identified facial characteristics of the user from the digital video portraying the user while the user accesses the one or more websites via the client device; and while the user accesses the one or more websites, modify the one or more websites to further include the digital characteristics report based on the identified facial characteristics of the user from the digital video portraying the user while the user accesses the one or more websites via the client device.