# US 12,217,014 B2 · Method, apparatus, and system for providing interpretation result using visual information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,014 B2 - Document ID: US12217014B2 - Status: Granted - Kind: B2 - Application number: 17/570,879 - Filed: 2022-01-07 - Published: 2022-07-14 - Granted: 2025-02-04 - Earliest claimed: 2021-01-08 - Assignees: Electronics and Telecommunications Research Institute - Applicants: Electronics and Telecommunications Research Institute - Inventors: Jinxia Huang; Jong Hun Shin - Web page: https://patentgrep.com/patent/US12217014B2 ## CPC - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06T 7/11: Region-based segmentation - G06V 20/63: {Scene text, e.g. street names} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06V 30/1444: {Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided is a method of providing an interpretation result using visual information, and the method includes: acquiring a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image; segmenting the acquired spatial domain image into a plurality of images; detecting text areas including text for each of the segmented images; generating text blocks, each of which is a text recognition result for each of the detected text areas, and determining the text block corresponding to the gaze position information; converting a first language included in the determined text block into a second language that is a target language; and providing the user with a conversion result of the second language. ## Claims ### Claim 1 A method of providing an interpretation result using visual information, which is performed by an apparatus for providing an interpretation result using visual information, the method comprising: acquiring a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image; segmenting the acquired spatial domain image into a plurality of images; detecting text areas including text for each of the segmented images; generating text blocks, each of which is a text recognition result for each of the detected text areas, and determining the text block corresponding to the gaze position information; converting a first language included in the determined text block into a second language that is a target language; and providing the user with a conversion result of the second language, wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes: combining text blocks consecutively located within predetermined sections adjacent to each other into one text block; and assigning a unique number to the text block or the combined text block, and wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes recognizing the text block corresponding to the unique number assigned in previous visual information as the same text block for a certain period of time. ### Claim 2 (depends on 1) The method of claim 1, further comprising arranging the spatial domain image and the gaze position information corresponding to the spatial domain area based on time. ### Claim 3 The method of chair 1, wherein the combining of the text blocks consecutively located within the predetermined sections adjacent to each other into one text block includes separating the text blocks having at least one of a different size and a different font and located in the same or adjacent text areas. ### Claim 4 (depends on 1) The method of claim 1, wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes selecting the text block closest to the gaze position information among the generated text blocks and determining the text block. ### Claim 5 (depends on 1) The method of claim 1, wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes, in response to point information of the user different from the gaze position information being present, selecting the text block closest to the point information of the user and determining the text block. ### Claim 6 (depends on 5) The method of claim 5, wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes, in response to the point information of the user being detected in a plurality of pieces thereof, providing feedback information to use only one piece of point information among the plurality of pieces of point information. ### Claim 7 (depends on 5) The method of claim 5, wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes, when the point information of the user continuously moves for a certain period of time within the same text block, selecting a text range pointed at by the point information as one independent user-specified text block and determining the text range. ### Claim 8 (depends on 1) The method of claim 1, wherein the generating of the text blocks, each of which is the text recognition result for each of the detected text areas, and the determining of the text block corresponding to the gaze position information includes, when the gaze position information sporadically shakes without being fixed for a certain period of time, providing a message guiding that the point information of the user is to be used. ### Claim 9 (depends on 1) The method of claim 1, wherein the segmenting of the acquired spatial domain image into the plurality of images includes: segmenting the acquired spatial domain image into the plurality of images on the basis of a pre-trained neural network algorithm; determining, for each of the segmented images, an individual object corresponding to one of previously-specified classes; and generating relative position information of the individual object based on the text block that is the text recognition result for the determined individual object. ### Claim 10 (depends on 9) The method of claim 9, wherein the converting of the first language included in the determined text block into the second language that is the target language includes converting the relative position information of the individual object corresponding to the determined text block into the second language, and wherein the providing of the converted second language to the user includes providing the user with the second language for the determined text block and the position information. ### Claim 11 An apparatus for providing an interpretation result using visual information, the apparatus comprising: a visual information receiver configured to acquire a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image; a visual information converter configured to segment the acquired spatial domain image into a plurality of images, detect text areas including text for each of the segmented images, determine, among text blocks, each of which is a text recognition result for each of the detected text areas, a text block corresponding to the gaze position information, and convert a first language included in the determined text block into a second language that is a target language; and an interpretation result provider configured to provide the user with a conversion result of the second language, wherein the visual information converter is configured to combine text blocks consecutively located within predetermined sections adjacent to each other into one text block and assign a unique number to the text block or the combined text block, and wherein the visual information converter is configured to recognize the text block corresponding to the unique number assigned in previous visual information as the same text block for a certain period of time. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein the visual information converter is configured to select the text block closest to the gaze position information among the generated text blocks and determine the text block. ### Claim 13 (depends on 11) The apparatus of claim 11, wherein the visual information converter is configured to, in response to point information of the user different from the gaze position information being present, select the text block closest to the point information of the user and determining the text block. ### Claim 14 (depends on 11) The apparatus of claim 11, wherein the visual information converter is configured to segment the acquired spatial domain image into the plurality of images on the basis of a pre-trained neural network algorithm and determine, for each of the segmented images, an individual object corresponding to one of previously-specified classes, and wherein the interpretation result provider is configured to generate relative position information of the individual object based on the text block that is the text recognition result for the determined individual object. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the visual information converter is configured to convert the relative position information of the individual object corresponding to the determined text block into the second language, and wherein the interpretation result provider is configured to provide the user with the second language for the determined text block and the position information. ### Claim 16 A system for providing an interpretation result using visual information, the system comprising: a communicator configured to acquire a spatial domain image including line-of-sight information of a user and gaze position information in the spatial domain image from an external device; a memory in which a program for recognizing text on the basis of the spatial domain image and the gaze position information and provide an interpretation result for the recognized text is stored; a processor configured to execute the program stored in the memory to: segment the acquired spatial domain image into a plurality of images; detect text areas including text for each of the segmented images; determine, among text blocks, each of which is a text recognition result for each of the detected text areas, a text block corresponding to the gaze position information; and convert a first language included in the determined text block into a second language that is a target language; and a device configured to provide the user with a conversion result of the second language, wherein the processor is configured to combine text blocks consecutively located within predetermined sections adjacent to each other into one text block and assign a unique number to the text block or the combined text block, and wherein the processor is configured to recognize the text block corresponding to the unique number assigned in previous visual information as the same text block for a certain period of time.