Back to results
US 10,769,444 B2Granted
Object detection from visual search queries
AssigneeGoh Soo Siah
InventorsStephen Maurice Moore, Larry Patrick Murray, Rajalingappaa Shanmugamani
Life of this patent
Estimating the term…
Abstract
This invention includes a system and method of populating a data-base with known objects. The database can be populated with off-line data augmentation (e. g. a web crawler) or by aligning known objects and metadata clusters with defined content. A viewer can query images from live or offline media. Objects in the viewers query are linked with similar objects or recommended products in the database.
Claims11
A method of detecting an object in video and matching the object to one or more products comprising the steps of:
- a) obtaining video and automatically extracting metadata and attributes of objects in frames and/or portions of frames in the video;
- b) segmenting the video based on depicted settings and/or events by comparing contents of consecutive frames for similarities and differences, wherein the video is traversed sequentially to detect a pair of frames or sequence of frames which breach a similarity threshold, wherein a key frame is identified for each segment;
- c) compiling segments of same or similar settings and/or events, wherein each segment is tagged with a segment identifier;
- d) analysing one or more segments to detect one or more objects, wherein frames and/or portions of frames are compared with defined content in a database populated by aligning known objects and metadata clusters, wherein the metadata is linked to the frames and/or portions of frames by the segment identifier, wherein the location of the detected one or more objects are obtained in each key frame, wherein an object-wise feature vector is generated for each segment;
- e) comparing the one or more objects to products;
- f) identifying products associated with the one or more objects, wherein a convolutional neural network (CNN) is used;
- g) notifying one or more viewers of the products;
- wherein a second screen augmentation is used for live or streaming video.
5 more elements
A method of detecting one or more objects in a digital screen shot from a video and matching the one or more objects to promotional material comprising the steps of:
- a) receiving an inquiry from a viewer in the form of a digital screen shot;
- b) identifying one or more objects in the digital screen shot by comparing the digital screen shot and/or portions of the digital screen shot with defined content in a database populated by aligning known objects and metadata clusters, wherein the digital screen shot is matched with a segment identifier from the video to retrieve a list of objects linked with a segment of the video, wherein a bag of visual words approach is used and top N candidate images are filtered and added to the database;
- c) comparing the one or more objects to products;
- d) matching products associated with the one or more objects, wherein a convolutional neural network (CNN) is used, wherein spatial verification is performed on the matching products for verification;
- and
- e) contacting the viewer with promotional material related to matched products;
- wherein after receiving the inquiry, visual word clusters are assigned.
4 more elements
A system for generating relationships between objects in video to products in a database of products comprising:
- a computerized network and system to be locally or remotely exposed to a user or groups of users via a user interface application;
- a module for detecting and storing media content locally or on a server;
- a module for transmitting the media content to a processor, remote or server-based, for ingestion of metadata and/or visual features;
- a module for transmitting the media content to a processor, remote or server-based, for extraction of metadata and visual features;
- a means for receiving input from one or more users in the form of a digital image derived from the video and that includes visual features;
- a module configured to implement the following:
- identify the visual features in the digital image and correlate the visual features with objects and/or groups of related products in the database, wherein the database is populated with defined content by aligning known objects and/or groups of related products and metadata clusters, wherein a bag of visual words approach is used to identify visual features and correlate the visual features with objects and/or groups of related products in the database;
- wherein the digital image is matched with a segment identifier to retrieve a list of objects and/or groups of related products linked with a segment of the video;
- and
- analyse visual features and metadata of the digital image to correlate the visual features with objects and/or groups of related products using a convolutional neural network (CNN), wherein spatial verification is performed on the matching objects and/or groups of related products for verification, and
- a networked service that distributes information on the objects and/or groups of related products to a user and/or groups of users;
- wherein after receiving the input, visual word clusters are assigned.
9 more elements
Dependencies are inferred from the claim text.