# US 2022/0036112 A1 · Word recognition method, apparatus and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036112 A1 - Document ID: US20220036112A1 - Status: Published application - Kind: A1 - Application number: 17/263,418 - Filed: 2020-03-31 - Published: 2022-02-03 - Earliest claimed: 2020-03-31 - Assignees: Boe Technology Group Co., Ltd. - Applicants: Boe Technology Group Co., Ltd. - Inventors: Guangwei HUANG; Yue LI - Web page: https://patentgrep.com/patent/US20220036112A1 ## CPC - G06V 30/19173: {Classification techniques} - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 30/153: {using recognition of characters or words} - G06T 2207/20084: Artificial neural networks [ANN] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present invention provides a word recognition method. The method includes: acquiring an image of a word to be recognized; recognizing edges of each character of the word to be recognized from the image of the word to be recognized; determining a geometric position of the word to be recognized; stretching the geometric position of the word to be recognized to a horizontal position; and recognizing the word to be recognized in the horizontal position. ## Claims ### Claim 1 A word recognition method, comprising: acquiring an image of a word to be recognized; recognizing edges of each character of the word to be recognized from the image of the word to be recognized; determining a geometric position of the word to be recognized; stretching the geometric position of the word to be recognized to a horizontal position; and recognizing the word to be recognized in the horizontal position. ### Claim 2 (depends on 1) The method according to claim 1, wherein recognizing the edges of each character of the word to be recognized from the image of the word to be recognized includes: determining the edges of each character of the word to be recognized by using a maximally stable extremal regions (MSER) algorithm. ### Claim 3 (depends on 1) The method according to claim 1, wherein determining the geometric position of the word to be recognized includes: determining an initial bounding box of the character according to the edges of the character; determining a position of a main axis by using a central point of the initial bounding box as a rotatory central point; continuously rotating and translating the main axis to determine boundaries; comparing areas of regions enclosed by the boundaries to obtain a minimum bounding box; establishing a coordinate system with any vertex of the image of the word to be recognized as an origin, a horizontal direction as an x axis, and a vertical direction as a y-axis; determining coordinates of four vertexes of the minimum bounding box in the coordinate system; determining edges in height and edges in width of the minimum bounding box according to the coordinates of the four vertexes, and using an angle between one of the edges in width and the x axis as an orientation angle of the character; and determining the geometric position of the word to be recognized according to orientation angles of all characters. ### Claim 4 (depends on 3) The method according to claim 3, wherein stretching the geometric position of the word to be recognized to the horizontal position includes: with a central point of the minimum bounding box as a central point of the character, connecting central points of all characters in order from left to right to obtain a central line of text; sampling a plurality of central points at equal intervals on the central line of the text; selecting a point on each of two edges in height in an orientation of a character corresponding to each sampled central point as a base point; and calculating a thin-plate spline (TPS) interpolation transformation matrix based on base points; and performing bilinear interpolation sampling to stretch the geometric position of the word to the horizontal position. ### Claim 5 (depends on 1) The method according to claim 1, wherein recognizing the word to be recognized in the horizontal position includes recognizing the word to be recognized in the horizontal position by a convolutional recurrent neural network (CRNN) algorithm.) ### Claim 6 (depends on 5) The method according to claim 5, wherein before recognizing the word to be recognized in the horizontal position by the CRNN algorithm, the method further includes: scaling the image of the word to be recognized in the horizontal position to a preset height and a preset width. ### Claim 7 (depends on 6) The method according to claim 6, wherein the CRNN algorithm includes: a convolutional neural network layer, the convolutional neural network layer including first and second convolution layers, first to fifth convolution blocks, first to fourth maxpooling layers, and a customized network layer, wherein characteristic vector sequences of the word to be recognized in the scaled image are extracted through the first and second convolution layers and the first to fifth convolution blocks, the characteristic vector sequences being used for recognizing the word. ### Claim 8 (depends on 7) The method according to claim 7, wherein the CRNN algorithm further includes: a recurrent neural network layer, the recurrent neural network layer including two bidirectional long short-term memory (LSTM) layers that are used for predicting characteristic vector sequences of the word to be recognized so as to obtain prediction results. ### Claim 9 (depends on 8) The method according to claim 8, wherein the CRNN algorithm further includes: a transcription layer for decoding the prediction results that are generated by the recurrent neural network layer into characters, and removing spacing characters and repetitive characters so as to obtain recognition results of the word to be recognized. ### Claim 10 (depends on 1) The method according to claim 1, wherein acquiring the image of the word to be recognized includes: setting a fixed-size rectangular region as a virtual pen tip according to a size of the word in the image with a position of a pen tip as a central point of a bottom edge; calculating an overlapping area between the fixed-size rectangular region and each of text boxes that are detected; finding a text box with an overlapping area that occupies a largest proportion of the fixed-size rectangular region, and using a word in the text box as the word to be recognized to obtain the image of the word to be recognized. ### Claim 11 An electronic apparatus, comprising a memory, a processor, and a computer program that is stored on the memory and executable on the processor, wherein when the processor executes the program, a word recognition method is implemented, including: acquiring an image of a word to be recognized; recognizing edges of each character of the word to be recognized from the image of the word to be recognized; determining a geometric position of the word to be recognized; stretching the geometric position of the word to be recognized to a horizontal position; and recognizing the word to be recognized in the horizontal position. ### Claim 12 (depends on 11) The electronic apparatus according to claim 11, wherein recognizing the edges of each character of the word to be recognized from the image of the word to be recognized includes: determining the edges of each character of the word to be recognized by using a maximally stable extremal regions (MSER) algorithm. ### Claim 13 (depends on 11) The electronic apparatus according to claim 11, wherein determining the geometric position of the word to be recognized includes: determining an initial bounding box of the character according to the edges of the character; determining a position of a main axis by using a central point of the initial bounding box as a rotatory central point; continuously rotating and translating the main axis to determine boundaries; comparing areas of regions enclosed by the boundaries to obtain a minimum bounding box; establishing a coordinate system with any vertex of the image of the word to be recognized as an origin, a horizontal direction as an x axis, and a vertical direction as a y-axis; determining coordinates of four vertexes of the minimum bounding box in the coordinate system; determining edges in height and edges in width of the minimum bounding box according to the coordinates of the four vertexes, and using an angle between one of the edges in width and the x axis as an orientation angle of the character; and determining the geometric position of the word to be recognized according to orientation angles of all characters. ### Claim 14 (depends on 13) The electronic apparatus according to claim 13, characterized in that stretching the geometric position of the word to be recognized to the horizontal position includes: with a central point of the minimum bounding box as a central point of the character, connecting central points of all characters in order from left to right to obtain a central line of text; sampling a plurality of central points at equal intervals on the central line of the text; selecting a point on each of two edges in height in an orientation of a character corresponding to each sampled central point as a base point; calculating a thin-plate spline (TPS) interpolation transformation matrix based on base points; and performing bilinear interpolation sampling to stretch the geometric position of the word to the horizontal position. ### Claim 15 (depends on 13) The electronic apparatus according to claim 13, wherein recognizing the word to be recognized in the horizontal position includes: recognizing the word to be recognized in the horizontal position by a convolutional recurrent neural network (CRNN) algorithm. ### Claim 16 (depends on 15) The electronic apparatus according to claim 15, wherein the program, when being excited by the processor, further performs: before recognizing the word to be recognized in the horizontal position by the CRNN algorithm, scaling the image of the word to be recognized in the horizontal position to a preset height and a preset width. ### Claim 17 (depends on 16) The electronic apparatus according to claim 16, wherein the CRNN algorithm includes: a convolutional neural network layer, the convolutional neural network layer including first and second convolution layers, first to fifth convolution blocks, first to fourth maxpooling layers, and a customized network layer, wherein characteristic vector sequences of the word to be recognized in the scaled image are extracted through the first and second convolution layers and the first to fifth convolution blocks, the characteristic vector sequences being used for recognizing the word. ### Claim 18 (depends on 17) The electronic apparatus according to claim 17, wherein the CRNN algorithm further includes: a recurrent neural network layer, the recurrent neural network layer including two bidirectional long short-term memory (LSTM) layers that are used for predicting characteristic vector sequences of the word to be recognized so as to obtain prediction results; and/or, a transcription layer for decoding the prediction results that are generated by the recurrent neural network layer into characters, and removing spacing characters and repetitive characters so as to obtain recognition results of the word to be recognized. ### Claim 19 (canceled) ### Claim 20 (depends on 11) The electronic apparatus according to claim 11, wherein acquiring the image of the word to be recognized includes: setting a fixed-size rectangular region as a virtual pen tip according to a size of the word in the image with a position of a pen tip as a central point of a bottom edge; calculating an overlapping area between the fixed-size rectangular region and each of text boxes that are detected; finding a text box with an overlapping area that occupies a largest proportion of the fixed-size rectangular region; and using a word in the text box as the word to be recognized to obtain the image of the word to be recognized. ### Claim 21 (depends on 1) A non-volatile computer-readable storage medium having stored a computer program thereon, wherein when the program is executed by a processor, the word recognition method according to claim 1 is implemented.