# US 2018/0025405 A1 · Presentation of content items synchronized with media display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025405 A1 - Document ID: US20180025405A1 - Status: Published application - Kind: A1 - Application number: 15/218,319 - Filed: 2016-07-25 - Published: 2018-01-25 - Earliest claimed: 2016-07-25 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Christopher William Jones; Mark Gerard D'Arcy; Ji Byol Lee - Web page: https://patentgrep.com/patent/US20180025405A1 ## CPC - G06Q 30/0631: {Recommending goods or services} - G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)} - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 2201/09: Recognition of logos - H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An online system displays relevant products along with a video watched by a user in a synchronized manner, which allows the user to explore relevant content (e.g., products recommended to the user) while the user is watching the video. The online system receives a video including a sequence of video frames to be displayed to the user, and the video may be associated with a specific theme, e.g., a soccer game, and one or more video frames may include points of interest (referred as “Q points”). The online system detects or receives Q points associated with corresponding video frames, where a Q point in a video frame identifies an object of interest in the video frame. The online system also correlates user profile information of the user with the detected points to select relevant products tailored to the user. ## Claims ### Claim 1 A computer-implemented method, the method comprising: receiving a video for display to a user, the video comprising a plurality of video frames; for each video frame: analyzing visual features of the video frame; and detecting one or more points of interest in the video frame based on the analysis of the visual features of the video frame, a point of interest in the video frame being associated with an object in the video frame, and the object in the video frame being related to one or more products; selecting one or more products based on the detected points of interest in the plurality of video frames; and providing for display the selected products along with the corresponding video frames being displayed to the user, wherein a selected product and a corresponding video frame showing the selected product are displayed concurrently in an user interface to the user. ### Claim 2 (depends on 1) The method of claim 1, providing for display the selected products along with the corresponding video frame comprising: for each video frame currently being displayed to the user: determining whether the video frame has a detected point interest; and selecting one or more products associated with the detected point of interest in the video frame. ### Claim 3 (depends on 2) The method of claim 2, further comprising: providing for display the video frame in a first portion of the display interface shown to the user; and providing for display one or more selected products in a scrolling carousel of products located in a second portion of the display interface, wherein at least one of the selected products is same or similar to a product shown in the video frame. ### Claim 4 (depends on 3) The method of claim 3, wherein providing for display one or more selected products in a scrolling carousel of products comprises: displaying one or more tools in the second portion of the display interface, wherein the user interacts with the displayed products through one or more tools. ### Claim 5 (depends on 4) The method of claim 4, wherein the one or more tools displayed in the second portion of the display interface comprise one of the following: a purchase tool for the user to purchase the displayed product; and a bookmark tool for the user to bookmark the displayed product. ### Claim 6 (depends on 1) The method of claim 1, wherein a point of interest in a video frame is identified by one of a timestamp associated with the video frame and frame number of the video frame. ### Claim 7 (depends on 1) The method of claim 1, wherein a selected product associated with a detected point of interest in a video frame shares one or more characteristics of the product related by identified object in the video frame. ### Claim 8 (depends on 1) The method of claim 1, further comprising: analyzing user profile information associated with the user; deriving user preference for products based on the analysis of the user profile information; and updating the selection of the products based on the derived user preference for products. ### Claim 9 (depends on 1) The method of claim 1, wherein analyzing visual features of a video frame comprising: applying a trained image feature extraction model to the video frame to extract a plurality of visual features; identifying an object in the video frame based on the extracted visual features; and identifying one or more items associated with the identified object, each identified item corresponding to a product that is likely to be of interest to the user. ### Claim 10 (depends on 1) The method of claim 1, wherein the video displayed to the user is one of a pre-recorded video and a broadcast video streamed to the user in real time. ### Claim 11 A computer-implemented method, the method comprising: detecting an opportunity to provide content to a user of an online system; providing a content item including a video for display to the user and a plurality of selected products based on detected points of interest in video frames of the video; receiving an indication from the user to play the video; playing the video and displaying each of the plurality of selected products in a synchronized way with the playing of the video, wherein a selected product and a corresponding video frame showing the selected product are displayed concurrently in an user interface to the user; receiving an interaction by the user with one of the selected products; and providing access to content describing the product to the user. ### Claim 12 (depends on 11) The method of claim 11, wherein the content describing the product includes detailed product information of the product, wherein the product information includes at least one of: price, way to purchase, style, size, color, gender, and brand name of the product. ### Claim 13 A non-transitory computer readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform the steps of: receiving a video for display to a user, the video comprising a plurality of video frames; for each video frame: analyzing visual features of the video frame; and detecting one or more points of interest in the video frame based on the analysis of the visual features of the video frame, a point of interest in the video frame being associated with an object in the video frame, and the object in the video frame being related to one or more products; selecting one or more products based on the detected points of interest in the plurality of video frames; and providing for display the selected products along with the corresponding video frames being displayed to the user, wherein a selected product and a corresponding video frame showing the selected product are displayed concurrently in an user interface to the user. ### Claim 14 (depends on 13) The non-transitory computer readable storage medium of claim 13, providing for display the selected products along with the corresponding video frame comprising: for each video frame currently being displayed to the user: determining whether the video frame has a detected point interest; and selecting one or more products associated with the detected point of interest in the video frame. ### Claim 15 (depends on 14) The non-transitory computer readable storage medium of claim 14, further comprising: providing for display the video frame in a first portion of the display interface shown to the user; and providing for display one or more selected products in a scrolling carousel of products located in a second portion of the display interface, wherein at least one of the selected products is same or similar to a product shown in the video frame. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein presenting one or more selected products in a scrolling carousel of products comprises: displaying one or more tools in the second portion of the display interface, wherein the user interacts with the displayed products through one or more tools. ### Claim 17 (depends on 16) The non-transitory computer readable storage medium of claim 16, wherein the one or more tools displayed in the second portion of the display interface comprise one of the following: a purchase tool for the user to purchase the displayed product; and a bookmark tool for the user to bookmark the displayed product. ### Claim 18 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein a point of interest in a video frame is identified by one of a timestamp associated with the video frame and frame number of the video frame. ### Claim 19 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein a selected product associated with a detected point of interest in a video frame shares one or more characteristics of the product related by identified object in the video frame. ### Claim 20 (depends on 13) The non-transitory computer readable storage medium of claim 13, further comprising: analyzing user profile information associated with the user; deriving user preference for products based on the analysis of the user profile information; and updating the selection of the products based on the derived user preference for products. ### Claim 21 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein analyzing visual features of a video frame comprising: applying a trained image feature extraction model to the video frame to extract a plurality of visual features; identifying an object in the video frame based on the extracted visual features; and identifying one or more items associated with the identified object, each identified item corresponding to a product that is likely to be of interest to the user. ### Claim 22 (depends on 13) The non-transitory computer readable storage medium of claim 13, wherein the video displayed to the user is one of a pre-recorded video and a broadcast video streamed to the user in real time.