# US 11,030,205 B2 · Contextual data transformation of image content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,030,205 B2 - Document ID: US11030205B2 - Status: Granted - Kind: B2 - Application number: 15/868,568 - Filed: 2018-01-11 - Published: 2019-06-13 - Granted: 2021-06-08 - Earliest claimed: 2017-12-13 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Yuancao Cui; Soohoon Cho; Vincent Kin-Wah Leung - Web page: https://patentgrep.com/patent/US11030205B2 ## CPC - G06F 16/248: Presentation of query results - G06F 16/532: Query formulation, e.g. graphical querying - G06F 16/951: Indexing; Web crawling techniques - G06F 16/9532: Query formulation - G06F 16/9538: Presentation of query results - G06F 16/9558: {Details of hyperlinks; Management of linked annotations} - G06F 16/538: Presentation of query results - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Non-limiting examples of the present disclosure describe processing by a contextual image analysis service that generates and surfaces contextually relevant data objects based on entities identified from image content. In one example, image content is analyzed and entity annotations are generated for the image content. The entity annotations may be converted to queries. Raw search results may be identified based on the converted queries. The raw search results may be filtered based on one or more specific content types. A contextual representation is generated based on the filtered raw search results, where the contextual representation comprises the image content and one or more contextually relevant data objects for at least one entity associated with the image content. The exemplary contextual representation of the image content and the one or more contextually relevant data objects may be surfaced through a user interface of an application/service. ## Claims ### Claim 1 A method performed by a processor of a computing system, comprising: generating, by a contextual image analysis service, an annotation for an object located within an image, wherein the annotation identifies a type of the object; prefixing, by the contextual image analysis service, content to the annotation to generate a query, wherein the query is a question; identifying, by the contextual image analysis service, a search result based upon the query, wherein the search result is an answer to the question; causing a user interface to be displayed on a display, wherein the user interface comprises the image and the query; responsive to receiving a selection of the query displayed within the user interface, updating the user interface to additionally display the search result; and causing a user interface feature to be presented on the display to allow user feedback for training the contextual image analysis service, wherein the contextual image analysis service adapts over time and learns at least one of placement, positioning, size, and amount of contextually relevant data objects to present to users, the contextually relevant data objects comprising questions and answers to the questions. ### Claim 2 (depends on 1) The method of claim 1, further comprising: prior to updating the user interface to additionally display the search result, removing metadata from the search result. ### Claim 3 (depends on 1) The method of claim 1, wherein the search result comprises a first item and a second item that are duplicative to one another, the method further comprising: prior to updating the user interface to additionally display the search result, removing the second item from the search result. ### Claim 4 (depends on 1) The method of claim 1, wherein the question is a trivia question. ### Claim 5 (depends on 1) The method of claim 1, wherein the question pertains to the object located within the image. ### Claim 6 (depends on 1) The method of claim 1, wherein the search result comprises video data. ### Claim 7 (depends on 1) The method of claim 1, wherein the search result comprises social network data. ### Claim 8 A computing system comprising: a processor; and memory, operatively connected with the processor, storing computer-executable instructions that, when executed by the processor, cause the processor to perform acts comprising: generating, by a contextual image analysis service, an annotation for an object located within an image, wherein the annotation identifies a type of the object; prefixing, by the contextual image analysis service, content to the annotation to generate a query, wherein the query is a question; identifying, by the contextual image analysis service, a search result based upon the query, wherein the search result is an answer to the question; causing a user interface to be displayed on a display, wherein the user interface comprises the image and the query; responsive to receiving a selection of the query displayed within the user interface, updating the user interface to additionally display the search result; and causing a user interface feature to be presented on the display to allow user feedback for training the contextual image analysis service, wherein the contextual image analysis service adapts over time and learns at least one of placement, positioning, size, and amount of contextually relevant data objects to present to users, the contextually relevant data objects comprising questions and answers to the questions. ### Claim 9 (depends on 8) The computing system of claim 8, the acts further comprising: prior to updating the user interface to additionally display the search result, removing metadata from the search result. ### Claim 10 (depends on 8) The computing system of claim 8, wherein the search result comprises a first item and a second item that are duplicative to one another, the acts further comprising: prior to updating the user interface to additionally display the search result, removing the second item from the search result. ### Claim 11 (depends on 8) The computing system of claim 8, wherein the question is a trivia question. ### Claim 12 (depends on 8) The computing system of claim 8, wherein the search result comprises video data. ### Claim 13 (depends on 8) The computing system of claim 8, wherein the question pertains to the object located within the image. ### Claim 14 (depends on 8) The computing system of claim 8, wherein the search result comprises social network data. ### Claim 15 A non-transitory computer-readable storage medium that stores computer-executable instructions that, when executed by a processor, cause the processor to perform acts comprising: generating, by a contextual image analysis service, an annotation for an object located within an image, wherein the annotation identifies a type of the object; prefixing, by the contextual image analysis service, content to the annotation to generate a query, wherein the query is a question; identifying, by the contextual image analysis service, a search result based upon the query, wherein the search result is an answer to the question; causing a user interface to be displayed on a display, wherein the user interface comprises the image and the query; responsive to receiving a selection of the query displayed within the user interface, updating the user interface to additionally display the search result; and causing a user interface feature to be presented on the display to allow user feedback for training the contextual image analysis service, wherein the contextual image analysis service adapts over time and learns at least one of placement, positioning, size, and amount of contextually relevant data objects to present to users, the contextually relevant data objects comprising questions and answers to the questions. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, the acts further comprising: prior to updating the user interface to additionally display the search result, removing metadata from the search result. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the search result comprises a first item and a second item that are duplicative to one another, the acts further comprising: prior to updating the user interface to additionally display the search result, removing the second item from the search result. ### Claim 18 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the question is a trivia question. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the search result comprises social network data. ### Claim 20 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the search result comprises video data.