# US 2017/0286552 A1 · Using Gesture Selection to Obtain Contextually Relevant Information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0286552 A1 - Document ID: US20170286552A1 - Status: Published application - Kind: A1 - Application number: 15/151,581 - Filed: 2016-05-11 - Published: 2017-10-05 - Earliest claimed: 2016-03-30 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Sean Gary Nordberg; Danielle Lauren Ellbogen; Michael John Patten; Clarice Jiarui Chan - Web page: https://patentgrep.com/patent/US20170286552A1 ## CPC - G06F 16/2428: {Query predicate definition using graphical user interfaces, including menus and forms (G06F16/2423 takes precedence)} - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 16/248: Presentation of query results - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 40/205: Parsing - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Described herein are various technologies pertaining to using a gesture to obtain contextually relevant information. Content within an area selected by a gesture is extracted. The extracted content and additional contextual information are packaged, optionally disambiguated, and provided to a search engine. Result(s) provided by the search engine are provided to a user to enrich the user experience. ## Claims ### Claim 1 A system, comprising: a processor; and a memory comprising: an extraction component configured to receive a gesture selecting a portion of presented content and, based upon the received gesture, extract an object within a portion of the presented content selected by the received gesture; and an identification component configured to identify additional information to package with the extracted object to assist in obtaining contextually relevant information. ### Claim 2 (depends on 1) The system of claim 1, the memory further comprising a disambiguation component configured to parse the packaged extracted object and additional information, filter the packaged extracted object and additional information to identify a dominant entity, and provide the filtered packaged object and additional information to a search engine as a search query. ### Claim 3 (depends on 1) The system of claim 1, the memory further comprising a presentation component configured to present a search result generated by a search engine in response to the packaged extracted object and additional information. ### Claim 4 (depends on 3) The system of claim 3, wherein the search result is presented via a display. ### Claim 5 (depends on 3) The system of claim 3, wherein the search result is presented via a virtual reality visualization/manipulation system. ### Claim 6 (depends on 1) The system of claim 1, wherein the object comprises at least one of text, an image or a three-dimensional representation presented by a virtual reality visualization manipulation system. ### Claim 7 (depends on 1) The system of claim 1, wherein the gesture is received from a touch sensitive display. ### Claim 8 (depends on 1) The system of claim 1, wherein the gesture is received from a virtual reality visualization manipulation system. ### Claim 9 (depends on 1) The system of claim 1, wherein the additional information comprises at least one of a user's physical location, information surrounding the object as presented, an audio file, a page data, metadata or a page link. ### Claim 10 A method of obtaining a contextually relevant result, the method executed by at least one computing device, the method comprising: receiving a gesture selecting at least a portion of an object being presented; extracting the object based on the received gesture; generating a package comprising the extracted object; disambiguating the package to generate a search query; providing the search query to a search engine; receiving a search result from the search engine; and presenting the search result. ### Claim 11 (depends on 10) The method of claim 10, further comprising: identifying additional information to package with the extracted object, wherein the package further comprises the additional information. ### Claim 12 (depends on 11) The method of claim 11, wherein the additional information comprises at least one of a user's physical location, information surrounding the object being presented, an audio file, a page data, metadata or a page link. ### Claim 13 (depends on 10) The method of claim 10, wherein the gesture is received from a touch sensitive display. ### Claim 14 (depends on 10) The method of claim 10, wherein the gesture is received from a virtual reality visualization manipulation system. ### Claim 15 (depends on 10) The method of claim 10, wherein the object comprises at least one of text, an image or a three-dimensional representation presented by a virtual reality visualization manipulation system. ### Claim 16 (depends on 10) The method of claim 10, wherein the search result is presented by at least one of a display or a virtual reality visualization/manipulation system. ### Claim 17 A computer storage media storing computer-readable instructions that when executed cause a computing device to: receive a gesture selecting at least a portion of an object being a presented; extract the object based on the received gesture; identify additional information to package with the extracted object; generate a package comprising the extracted object and the identified additional information; disambiguate the package to generate a search query; provide the search query to a search engine; receive a search result from the search engine; and present the search result. ### Claim 18 (depends on 17) The computer storage media of claim 17, wherein the gesture is received from a virtual reality visualization manipulation system. ### Claim 19 (depends on 17) The computer storage media of claim 17, wherein the object comprises at least one of text, an image or a three-dimensional representation presented by a virtual reality visualization manipulation system. ### Claim 20 (depends on 17) The computer storage media of claim 17, wherein the additional information comprises at least one of a user's physical location, information surrounding the object being presented, an audio file, a page data, metadata or a page link.