# US 2020/0394266 A1 · Image-to-text recognition for a sequence of images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394266 A1
- Document ID: US20200394266A1
- Status: Published application
- Kind: A1
- Application number: 16/441,298
- Filed: 2019-06-14
- Published: 2020-12-17
- Earliest claimed: 2019-06-14
- Assignees: KYOCERA Document Solutions Inc.
- Applicants: KYOCERA Document Solutions, Inc.
- Inventors: Allan Sahagun
- Web page: https://patentgrep.com/patent/US20200394266A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/279: Recognition of textual entities
- G06V 30/153: {using recognition of characters or words}
- G06V 30/242: Division of the character sequences into groups prior to recognition; Selection of dictionaries
- G06V 30/10: Character recognition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods relate generally to image-to-text recognition. In an example thereof, a device captures an image and generates image data and associated meta data. A parser parses the image data and the meta data into sub-image data and sub-meta data, respectively. A recognizer renders the sub-image data into strings for one or more identified objects in the sub-image data. A mapper-controller maps the sub-image data in association with the sub-meta data to generate a data structure having positional information with respect to the one or more identified objects. A dictionary store is accessed by a natural language processing engine responsive to the strings. The natural language processing engine converts the strings into a natural language expression for the strings and the positional information obtained from the data structure.

## Claims

### Claim 1

A system, comprising:
a device configured to capture frames for an image and generate image data therefor;
a parser to parse the image data from each of the frames into corresponding sub-images;
a recognizer to render one or more identified objects in the sub-images to one or more strings;
the recognizer configured to generate meta data for the strings;
a mapper to map the meta data in association with the one or more identified objects to generate tracking information;
a first comparator to compare the tracking information between the frames to obtain positional information;
a string store configured to group the strings of each of the frames for a frame-to-frame comparison of groups of the strings;
a second comparator configured for a frame-to-frame comparison of groups of the strings to obtain difference information;
a third comparator to compare the strings with one or more language libraries to identify potential words and rank the potential words using the difference information for each of the potential words having multiple recognitions; and
a natural language processing engine having access to the one or more language libraries to convert one or more of the potential words into a natural language expression using the positional information.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the positional information is configured to provide an indication for movement of the one or more identified objects for differences between the sub-images.

### Claim 3

A system, comprising:
a device configured to capture frames of an image and generate image data and associated meta data;
a parser to parse the image data and the meta data into sub-image data and sub-meta data, respectively;
a recognizer to render the sub-image data into strings for one or more identified objects in the sub-image data;
a mapper-controller configured to map the sub-image data in association with the sub-meta data to generate a data structure having positional information with respect to the one or more identified objects;
a dictionary store accessible by a natural language processing engine responsive to the strings; and
the natural language processing engine to convert the strings into a natural language expression for the strings and the positional information obtained from the data structure.

### Claim 4 (depends on 3)

The system according to claim 3, wherein:
the device is a multi-function printer and includes a scanner; and
the one or more identified objects include one or more symbols, words, characters, ideograms, icons, emojis, or logograms.

### Claim 5 (depends on 3)

The system according to claim 3, wherein:
the device is a mobile device and includes a camera; and
the one or more identified objects include one or more symbols, words, characters, ideograms, icons, emojis, or logograms.

### Claim 6 (depends on 5)

The system according to claim 5, wherein:
the parser is configured to parse the image data into frames with each frame thereof having a sub-image of an associated one of sub-images corresponding to the frames;
the recognizer includes at least one recognition encoder configured to encode the one or more identified objects in the sub-images into the strings; and
the mapper-controller configured to map a sequence of the sub-images in association with the sub-meta data to provide the positional information in the data structure.

### Claim 7 (depends on 6)

The system according to claim 6, wherein the mobile device includes one or more accelerometers to sense one or more of movement, direction, speed, or tilt of the mobile device in association with the capture of the frames of the image to provide at least a portion of the meta data.

### Claim 8 (depends on 7)

The system according to claim 7, wherein:
the mobile device includes a global positioning system to obtain geolocation information; and
the meta data includes the geolocation information with respect to a location of the mobile device during the capture of the frames.

### Claim 9 (depends on 6)

The system according to claim 6, wherein the recognizer includes a character recognition encoder and a word recognition encoder.

### Claim 10 (depends on 9)

The system according to claim 9, wherein:
the recognizer further includes an image preprocessor configured to preprocess the sub-images prior to encoding into the strings; and
the recognizer is configured to provide a ranking for each string of the strings having multiple recognitions.

### Claim 11 (depends on 9)

The system according to claim 9, wherein the image is for a digital panorama photo, a digital still photo, a video clip, or a graphical interchange formatted clip.

### Claim 12 (depends on 9)

The system according to claim 9, wherein memory of the natural language processing engine includes a table having the strings, the frames, and the positional information in association with one another.

### Claim 13

A method, comprising:
capturing with a device frames of an image and generating image data and associated meta data;
parsing with a parser the image data and the meta data into sub-image data and sub-meta data, respectively;
rendering with a recognizer the sub-image data into strings for one or more identified objects in the sub-image data;
mapping with a mapper-controller the sub-image data in association with the sub-meta data to generate a data structure having positional information with respect to the one or more identified objects;
accessing a dictionary store by a natural language processing engine responsive to the strings; and
converting by the natural language processing engine the strings into a natural language expression for the strings and the positional information obtained from the data structure.

### Claim 14 (depends on 13)

The method according to claim 13, wherein:
the device is a multi-function printer and includes a scanner; and
the one or more identified objects include one or more symbols, words, characters, ideograms, icons, emojis, or logograms.

### Claim 15 (depends on 13)

The method according to claim 13, wherein:
the device is a mobile device and includes a camera; and
the one or more identified objects include one or more symbols, words, characters, ideograms, icons, emojis, or logograms.

### Claim 16 (depends on 15)

The method according to claim 15, further comprising:
parsing with the parser the image data with each frame thereof having a sub-image of an associated one of sub-images corresponding to the frames;
encoding the one or more identified objects in the sub-images into the strings by at least one recognition encoder of the recognizer; and
mapping by the mapper-controller a sequence of the sub-images in association with the sub-meta data to provide the positional information in the data structure.

### Claim 17 (depends on 16)

The method according to claim 16, further comprising sensing with one or more accelerometers of the mobile device one or more of movement, direction, speed, or tilt of the mobile device in association with the capturing of the image to provide the meta data.

### Claim 18 (depends on 17)

The system according to claim 17, further comprising:
obtaining geolocation information with a global positioning system of the mobile device; and
wherein the meta data includes the geolocation information with respect to a location of the mobile device during the capturing of the image.

### Claim 19 (depends on 16)

The system according to claim 16, wherein the encoding includes using at least one of a character recognition encoder or a word recognition encoder of the recognizer.

### Claim 20 (depends on 19)

The system according to claim 19, wherein:
preprocessing the sub-images with an image preprocessor of the recognizer prior to the encoding into the strings;
ranking by the recognizer each string of the strings having multiple recognitions; and
wherein the image is for a digital panorama photo, a digital still photo, a video clip, or a graphical interchange formatted clip.
