# US 10,311,619 B1 · Video stream customization using graphics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,619 B1 - Document ID: US10311619B1 - Status: Granted - Kind: B1 - Application number: 16/103,927 - Filed: 2018-08-15 - Granted: 2019-06-04 - Earliest claimed: 2017-05-16 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Crystal Rhan-Tsor Chen; Lydia Han; Aigerim Shorman; Tanner Cotant Christensen; Noah Bradley Blecher; Daniil Burdakov; Karthik Narayan; Harshdeep Singh; Pradeep George Mathias; Deepa Diwakar; Tamar Bar Lev; Christopher Jon Lang; Shupin Mao - Web page: https://patentgrep.com/patent/US10311619B1 ## CPC - G06N 20/00: Machine learning - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06F 3/147: using display panels - G06T 11/60: Creating or editing images; Combining images with text - G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels - G09G 2340/125: wherein one of the images is motion video - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract An online system customizes video conversations between users of the online system by providing graphics that are likely to interest the users. The online system may present composite views overlaying selected graphics on a video stream, which may be part of an augmented reality (AR) environment. The graphics include, for example, background graphics, masks, props, visual or particular effects, frames or borders, etc. The online system may use a machine learning model to predict whether a user is likely to select or interact with a particular graphic. The online system can also filter or rank graphics according to user affinities or user connections on the online system. Users may be encouraged to interact with the graphics that are customized to their specific interests, which can promote an engaging video conversation or AR experience. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by a server of an online system, a video stream from a client device of a user of the online system; determining user information of the user based on actions performed by the user stored in the online system; identifying a plurality of graphics provided at least in part by a plurality of users of the online system; determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user; for each graphic of the set of candidate graphics, generating a score for the graphic indicating a likelihood of the user selecting the graphic; providing, by the server to the client device, the set of candidate graphics for presentation to the user on the client device in an order based on the scores; receiving, by the server from the client device, a selection for a candidate graphic of the set of candidate graphics; and providing, by the server to the client device, a composite view generated by combining the selected candidate graphic with the video stream for presentation to the user on the client device. ### Claim 2 (depends on 1) The method of claim 1, wherein the scores are generated by a model trained based at least in part on feature vectors derived from actions performed by a population of users of the online system. ### Claim 3 (depends on 2) The method of claim 2, wherein the actions performed by the population of users of the online system include a plurality of selections of graphics of the plurality of graphics to be included in composite views presented to the population of users. ### Claim 4 (depends on 1) The method of claim 1, wherein each graphic of the set of candidate graphics is associated with one or more characteristics, and wherein filtering the plurality of graphics based at least in part on user information describing the user comprises: determining interests of the user based on the user information; and determining, for a graphic of the set of candidate graphics, that the corresponding one or more characteristics of the graphic matches at least one of the interests of the user. ### Claim 5 (depends on 1) The method of claim 1, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system. ### Claim 6 (depends on 1) The method of claim 1, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is combined with at least a portion of the visual representation of the user. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining a background portion of the video stream, wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least the background portion. ### Claim 8 A computer-implemented method comprising: receiving, by a server of an online system, an image from a client device of a user of the online system; determining user information of the user based on actions performed by the user stored in the online system; identifying a plurality of graphics provided at least in part by a plurality of users of the online system; determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user; for each graphic of the set of candidate graphics, generating a score for the graphic indicating a likelihood of the user selecting the graphic; providing, by the server to the client device, the set of candidate graphics for presentation to the user on the client device in an order based on the scores; receiving, by the server from the client device, a selection for a candidate graphic of the set of candidate graphics; and providing, by the server to the client device, a composite view generated by combining the selected candidate graphic with the image for presentation to the user on the client device. ### Claim 9 (depends on 8) The method of claim 8, wherein the scores are generated by a model trained based at least in part on feature vectors derived from actions performed by a population of users of the online system. ### Claim 10 (depends on 9) The method of claim 9, wherein the actions performed by the population of users of the online system include a plurality of selections of graphics of the plurality of graphics to be included in composite views presented to the population of users. ### Claim 11 (depends on 8) The method of claim 8, wherein each graphic of the set of candidate graphics is associated with one or more characteristics, and wherein filtering the plurality of graphics based at least in part on user information describing the user comprises: determining interests of the user based on the user information; and determining, for a graphic of the set of candidate graphics, that the corresponding one or more characteristics of the graphic matches at least one of the interests of the user. ### Claim 12 (depends on 8) The method of claim 8, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system. ### Claim 13 (depends on 8) The method of claim 8, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is combined with at least a portion of the visual representation of the user. ### Claim 14 A non-transitory computer-readable storage medium storing instructions that when executed by a processor cause the processor to perform steps including: receiving, by a server of an online system, a video stream from a client device of a user of an online system; determining user information of the user based on actions performed by the user stored in the online system; identifying a plurality of graphics provided at least in part by a plurality of users of the online system; determining a set of candidate graphics by filtering the plurality of graphics based at least in part on the user information of the user; for each graphic of the set of candidate graphics, generating a score for the graphic indicating a likelihood of the user selecting the graphic; providing, by the server to the client device, the set of candidate graphics for presentation to the user on the client device in an order based on the scores; receiving, by the server from the client device, a selection from the user for a candidate graphic of the set of candidate graphics; and providing, by the server to the client device, a composite view generated by combining the selected candidate graphic with the video stream for presentation to the user on the client device. ### Claim 15 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein the scores are generated by a model trained based at least in part on feature vectors derived from actions performed by a population of users of the online system. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the actions performed by the population of users of the online system include a plurality of selections of graphics of the plurality of graphics to be included in composite views presented to the population of users. ### Claim 17 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein each graphic of the set of candidate graphics is associated with one or more characteristics, and wherein filtering the plurality of graphics based at least in part on user information describing the user comprises: determining interests of the user based on the user information; and determining, for a graphic of the set of candidate graphics, that the corresponding one or more characteristics of the graphic matches at least one of the interests of the user. ### Claim 18 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein the plurality of graphics is provided further by at least a third party system, wherein the set of candidate graphics further includes a graphic provided by the third party system, and wherein the user information indicates that the user interacted with an object of the online system associated with the third party system. ### Claim 19 (depends on 14) The non-transitory computer readable storage medium of claim 14, wherein the composite view includes a visual representation of the user, and wherein the selected candidate graphic is combined with at least a portion of the visual representation of the user. ### Claim 20 (depends on 14) The non-transitory computer readable storage medium of claim 14, having further instructions that when executed by the processor cause the processor to: determining a background portion of the video stream, wherein the composite view is generated further by applying a visual effect of the selected candidate graphic to at least the background portion.