Method and electronic device for recognizing text in image

An electronic device for recognizing a text in an image, the electronic device comprising:

• a display;
• a memory storing one or more instructions;
• and
• at least one processor configured to execute the one or more instructions stored in the memory to:
• detect spatial positions of pieces of text included in the text in the image,
• generate cropped images by cropping an area for each of the pieces of text in the image based on the detected spatial positions,
• recognize characters of the pieces of text for each of the cropped images,
• generate a sentence by inputting the characters and respective spatial positions of each piece of text into a multimodal language model, and
• control the display to display the sentence,
• wherein the multimodal language model is an artificial intelligence (AI) model for inferring an original sentence of the text,
• wherein the multimodal language model includes a plurality of sub-networks having different layers appropriate for processing different input modality data, the sub-networks including a first sub-network configured to calculate first input modality data comprising positions of text and a second sub-network configured to calculate second input modality data comprising characters of text, and
• wherein the multimodal language model applies a first weight to the first sub-network and a second weight to the second sub-network, and obtains the sentence by applying the first weight to the positions of the text and the second weight to the characters of the text.

One or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of an electronic device individually or collectively, cause the electronic device to perform operations for recognizing a text in an image, the operations comprising:

• detecting spatial positions of pieces of text included in the text in the image;
• generating cropped images by cropping an area for each of the pieces of text in the image based on the detected spatial positions;
• recognizing characters of the pieces of text for each of the cropped images;
• generating a sentence by inputting the characters and respective spatial positions of each piece of text into a multimodal language model;
• and
• displaying the sentence,
• wherein the multimodal language model is an artificial intelligence (AI) model for inferring an original sentence of the text,
• wherein the multimodal language model includes a plurality of sub-networks having different layers appropriate for processing different input modality data, the sub-networks including a first sub-network configured to calculate first input modality data comprising positions of text and a second sub-network configured to calculate second input modality data comprising characters of text, and
• wherein the multimodal language model applies a first weight to the first sub-network and a second weight to the second sub-network, and obtains the sentence by applying the first weight to the positions of the text and the second weight to the characters of the text.