# US 12,118,184 B2 · Efficiently augmenting images with related content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,184 B2 - Document ID: US12118184B2 - Status: Granted - Kind: B2 - Application number: 18/354,101 - Filed: 2023-07-18 - Published: 2023-11-09 - Granted: 2024-10-15 - Earliest claimed: 2021-12-28 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Charles Yang; Louis Wang; Charles J. Rosenberg - Web page: https://patentgrep.com/patent/US12118184B2 ## CPC - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 16/583: using metadata automatically derived from the content - G06F 16/951: Indexing; Web crawling techniques - G06F 40/205: Parsing - G06F 2203/04803: Split screen, i.e. subdividing the display area or the window area into separate subareas - G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text. ## Claims ### Claim 1 A method comprising: obtaining, by a computing system comprising one or more processors, an image; extracting, by the computing system, text from the image; partitioning, by the computing system, the text into a plurality of blocks; providing, by the computing system, data descriptive of a user interface that presents the plurality of blocks as respective first user-selectable targets; obtaining, by the computing system, a user selection of a first block of the plurality of blocks, wherein the first block comprises a first set of text; generating, by the computing system, a search query based on the first set of text; determining, by the computing system, a context for the image, wherein the context is descriptive of category classification for the image; processing, by the computing system, the search query and the context to determine search result content, wherein the search result content is determined based on the first set of text and the context; and providing, by the computing system, data descriptive of an updated user interface that is descriptive the search result content presented with a portion of the image that includes the first block. ### Claim 2 (depends on 1) A method as claimed in claim 1, wherein determining the context for the image comprises: processing, by the computing system, the image with one or more machine learning models to classify the image. ### Claim 3 (depends on 2) A method as claimed in claim 2, wherein an output of the one or more machine learning models comprises a predefined category. ### Claim 4 (depends on 1) A method as claimed in claim 1, wherein the context of the image is determined based on other text depicted in the image. ### Claim 5 (depends on 1) A method as claimed in claim 1, wherein the search result content is associated with a resource responsive to the search query and associated with the context. ### Claim 6 (depends on 1) A method as claimed in claim 1, wherein the category classification comprises at least one of: a menu classification, a music classification, and/or a sign classification. ### Claim 7 (depends on 1) The method as claimed in claim 1, wherein processing the search query and the context to determine the search result content comprises: modifying, by the computing system, the search query to include one or more additional terms based on the context. ### Claim 8 (depends on 1) The method of claim 1, wherein processing the search query and the context to determine the search result content comprises: identifying, by the computing system, a plurality of electronic resources based on the search query; and modifying, by the computing system, a ranking of the plurality of electronic resources based on the context. ### Claim 9 (depends on 8) A method of claim 8, wherein the plurality of electronic resources comprises one or more web pages. ### Claim 10 (depends on 8) The method of claim 8, wherein the plurality of electronic resources comprises one or more images. ### Claim 11 (depends on 8) The method of claim 8, wherein the plurality of electronic resources comprises one or more videos. ### Claim 12 (depends on 1) The method of claim 1, wherein the category classification is one of a predefined set of categories. ### Claim 13 (depends on 1) The method of claim 1, wherein determining the context for the image comprises: processing the image with one or more machine learning models to determine the category classification. ### Claim 14 A system comprising: a data processing apparatus; and a memory apparatus in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising: obtaining an image; extracting text from the image; partitioning the text into a plurality of blocks; providing data descriptive of a user interface that presents the plurality of blocks as respective first user-selectable targets; obtaining a user selection of a first block of the plurality of blocks, wherein the first block comprises a first set of text; generating a search query based on the first set of text; determining a context for the image, wherein the context is descriptive of category classification for the image; processing the search query and the context to determine search result content, wherein the search result content is determined based on the first set of text and the context; and providing data descriptive of an updated user interface that is descriptive the search result content presented with a portion of the image that includes the first block. ### Claim 15 (depends on 14) The system of claim 14, wherein partitioning the extracted text into the blocks is based at least partially on semantic analysis of the extracted text. ### Claim 16 (depends on 14) The system of claim 14, wherein the category classification comprises a menu classification. ### Claim 17 (depends on 14) The system of claim 14, wherein the category classification comprises a music classification. ### Claim 18 One or more non-transitory computer-readable storage media having instructions stored thereon, which, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising: obtaining an image; extracting text from the image; partitioning the text into a plurality of blocks; providing data descriptive of a user interface that presents the plurality of blocks as respective first user-selectable targets; obtaining a user selection of a first block of the plurality of blocks, wherein the first block comprises a first set of text; generating a search query based on the first set of text; determining a context for the image, wherein the context is descriptive of category classification for the image; processing the search query and the context to determine search result content, wherein the search result content is determined based on the first set of text and the context; and providing data descriptive of an updated user interface that is descriptive the search result content presented with a portion of the image that includes the first block. ### Claim 19 (depends on 18) The one or more non-transitory computer-readable storage media of claim 18, wherein the category classification comprises a menu classification, and wherein the first set of text is descriptive of a food item. ### Claim 20 (depends on 18) The one or more non-transitory computer-readable storage media of claim 18, wherein the category classification comprises a sign classification.