# US 2014/0195513 A1 · System and method for using on-image gestures and multimedia content elements as search queries > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0195513 A1 - Document ID: US20140195513A1 - Status: Published application - Kind: A1 - Application number: 14/203,047 - Filed: 2014-03-10 - Published: 2014-07-10 - Earliest claimed: 2005-10-26 - Assignees: Cortica, Ltd. - Applicants: Cortica, Ltd. - Inventors: Igal RAICHELGAUZ; Karina ODINAEV; Yehoshua Y. ZEEVI - Web page: https://patentgrep.com/patent/US20140195513A1 ## CPC - G06F 16/43: Querying - G06F 16/951: Indexing; Web crawling techniques - G06N 20/00: Machine learning - H04H 20/103: {Transmitter-side switching} - H04H 20/38: Arrangements for distribution where lower stations, e.g. receivers, interact with the broadcast - H04H 60/33: Arrangements for monitoring the users' behaviour or opinions - H04H 60/37: for identifying segments of broadcast information, e.g. scenes or extracting programme ID - H04H 60/56: Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 - H04H 60/66: for using the result on distributors' side - H04N 7/17318: {Direct or substantially direct transmission and handling of requests} - H04N 21/25891: {being end-user preferences (retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)} - H04N 21/2668: Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles {(information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063)} - H04N 21/466: Learning process for intelligent management, e.g. learning user preferences for recommending movies {(services using the results of monitoring in broadcast systems H04H60/61)} - H04N 21/8106: {involving special audio data, e.g. different tracks for different languages} - H04H 2201/90: characterised by the use of signatures - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method and system for searching a plurality of information sources using a multimedia content element are provided. The method comprises receiving an input search query including the multimedia content element and at least one on-image gesture detected thereon; generating at least one signature for the multimedia content element; analyzing the at least one on-image gesture to determine at least one portion of the multimedia content element in which a user is interested; generating at least one signature for each of the at least one portion of the multimedia content element; generating a textual search query using the at least the one generated signature; searching the plurality of information sources using the generated textual search query; and causing the display of search results retrieved from the plurality of information sources on a user device. ## Claims ### Claim 1 A method for searching a plurality of information sources using a multimedia content element, comprising: receiving an input search query including the multimedia content element and at least one on-image gesture detected thereon; generating at least one signature for the multimedia content element; analyzing the at least one on-image gesture to determine at least one portion of the multimedia content element in which a user is interested; generating at least one signature for each of the at least one portion of the multimedia content element; generating a textual search query using the at least the one generated signature; searching the plurality of information sources using the generated textual search query; and causing the display of search results retrieved from the plurality of information sources on a user device. ### Claim 2 (depends on 1) The method of claim 1, further comprising: creating an overlay on the at least one multimedia content element for receiving at least one search parameter representing an intent of the user, wherein the overlay is at least a search box. ### Claim 3 (depends on 2) The method of claim 2, further comprising: receiving at least one of demographic information and location information related to a user submitting the input search query; and optimizing the generated textual search query based on at least one of the demographic information, the location information, and the at least one search parameter. ### Claim 4 (depends on 1) The method of claim 1, wherein the on-image gesture is any one of: a touch gesture, a sketch gesture, a scrolls over at least a portion of the multimedia content element, a clicks over the at least a portion of the multimedia content element, wherein the touch gesture is detected on a user device having a touch screen display. ### Claim 5 (depends on 4) The method of claim 4, wherein analyzing the at least on-image gesture further comprises: determining a type of the on-image gesture and a pattern of the on-image gesture with respect to the multimedia content element. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the textual search query further comprises: searching a query database for a textual search query that matches the at least one generated signature, wherein the query database maintains mappings of signatures to text queries, and wherein a plurality of signatures are mapped to a single textual search query. ### Claim 7 (depends on 5) The method of claim 5, further comprising: clustering the at least one generated signature; and searching the query database using the clustered signature. ### Claim 8 (depends on 1) The method of claim 1, wherein generating the textual search query further comprises: matching the at least one generated signature against a plurality of concepts, and using a metadata of at least one matching concept as the textual search query. ### Claim 9 (depends on 1) The method of claim 1, further comprising: determining a context of the multimedia content element respective of the at least one generated signature; and determining the textual search query based on the determined context. ### Claim 10 (depends on 1) The method of claim 1, wherein causing the display of search results retrieved from the plurality of information further comprises: filtering the retrieved search results using the at least one generated signature. ### Claim 11 (depends on 1) The method of claim 1, wherein the at least one multimedia content element is at least one of: an image, graphics, a video stream, a video clip, a video frame, a photograph, images of signals, and portions thereof. ### Claim 12 (depends on 1) A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1. ### Claim 13 A system for searching a plurality of information sources using a multimedia content element, comprising: a processor; and a memory coupled to the processor, the memory contains instructions that when executed by the processor cause the system to: receive an input search query including the multimedia content element and at least one on-image gesture detected thereon; generate at least one signature for the multimedia content element; analyze the at least on-image gesture to determine at least one portion of the multimedia content element in which a user is interested; generate, by a signature generator, at least one signature for each of the at least one portion of the multimedia content element; generate a textual search query using the at least one generated signature; search the plurality of information sources using the generated textual search query; and cause the display of search results retrieved from the plurality of information sources on a user device. ### Claim 14 (depends on 12) The system of claim 12, wherein the signature generator is communicatively connected to the processor. ### Claim 15 (depends on 13) The system of claim 13, wherein the system is further configured to: create an overlay on the at least one multimedia content element for receiving at least one search parameter representing an intent of the user, wherein the overlay is at least a search box. ### Claim 16 (depends on 15) The system of claim 15, wherein the system is further configured to: receive at least one of demographic information and location information related to a user submitting the input search query; and optimize the generated textual search query based on at least one of the demographic information, the location information, and the at least one search parameter. ### Claim 17 (depends on 13) The system of claim 13, wherein the on-image gesture is any one of: a touch gesture, a sketch gesture, a scrolls over at least a portion of the multimedia content element, a clicks over the at least a portion of the multimedia content element, wherein the touch gesture is detected on a user device having a touch screen display. ### Claim 18 (depends on 17) The system of claim 17, wherein the system is further configured to: determine a type of the on-image gesture and a pattern of the on-image gesture with respect to the multimedia content element. ### Claim 19 (depends on 18) The system of claim 18, wherein the system is further configured to: search a query database for a textual search query that matches the at least one generated signature, wherein the query database maintains mappings of signatures to text queries, and wherein a plurality of signatures are mapped to a single textual search query. ### Claim 20 (depends on 19) The system of claim 19, wherein the system is further configured to: cluster the at least one generated signature; and search the query database using the clustered signature. ### Claim 21 (depends on 13) The system of claim 13, wherein the system is further configured to: match the at least one generated signature against a plurality of concepts, and use a metadata of at least one matching concept as the textual search query. ### Claim 22 (depends on 13) The system of claim 13, wherein the system is further configured to: determine a context of the multimedia content element respective of the at least one generated signature; and determine the textual search query based on the determined context. ### Claim 23 (depends on 12) The system of claim 12, wherein the multimedia content element is at least one of: an image, graphics, a video stream, a video clip, a video frame, a photograph, images of signals, and portions thereof.