# US 2022/0019837 A1 · Viewfinder assistant for visually impaired

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0019837 A1
- Document ID: US20220019837A1
- Status: Published application
- Kind: A1
- Application number: 16/928,455
- Filed: 2020-07-14
- Published: 2022-01-20
- Earliest claimed: 2020-07-14
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Richard J. Tomsett; Corey Sonier; William Kirby Wright, III
- Web page: https://patentgrep.com/patent/US20220019837A1

## CPC

- G06V 10/40: Extraction of image or video features
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G06V 30/1475: {Inclination or skew detection or correction of characters or of image to be recognised}
- G10L 13/00: Speech synthesis; Text to speech systems
- G09B 21/001: {Teaching or communicating with blind persons (G09B21/02 - G09B21/06 take precedence)}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)

## Abstract

In an approach for guiding a visually impaired user to position a mobile device appropriately in relation to a screen so that dynamic information on the screen can be reliably extracted and conveyed to the visually impaired user, a processor receives an image captured by a camera of a mobile device. A processor performs object recognition on the image to identify a digital screen and a location of the digital screen in the image. A processor retrieves a template of the digital screen. A processor performs angle-sensitive optical character recognition (OCR) on the location of the digital screen in the image. Responsive to a processor determining text on the digital screen can be extracted, a processor conveys the text to a user. Responsive to a processor determining text on the digital screen cannot be extracted, a processor guides the user to re-orient the mobile device to capture a better image.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, by one or more processors, an image captured by a camera of a user mobile device;
performing, by the one or more processors, object recognition on the image to identify a digital screen and a location of the digital screen in the image, wherein the digital screen is identified to be of a known type, brand, or model of digital screen;
retrieving, by the one or more processors, a template of the digital screen based on the known type, brand, or model of digital screen;
performing, by the one or more processors, angle-sensitive optical character recognition (OCR) on the location of the digital screen in the image; and
determining, by the one or more processors, whether, within the image, text on the digital screen can be extracted.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
responsive to determining the text can be extracted, audibly conveying, by the one or more processors, the text to a user of the user mobile device using text-to-speech.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1,
responsive to determining the text cannot be extracted, guiding, by the one or more processors, a user of the user mobile device to re-orient at least one of a position and a rotation of the user mobile device to capture another image.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein performing angle-sensitive OCR on the location of the digital screen in the image comprises:
detecting, by the one or more processors, rectangular regions in the image that contain text;
calculating, by the one or more processors, angles of the rectangular regions relative to a horizontal axis; and
converting, by the one or more processors, text detected in these rectangular regions from image data to text data.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
wherein the template indicates a set of locations of where text would be located on the digital screen; and
comparing, by the one or more processors, locations of the rectangular regions in the image to the set of locations of where text would be located on the digital screen based on the template.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein determining whether the text on the digital screen can be extracted is based on comparing the locations of the rectangular regions in the image to the set of locations of where text would be located on the digital screen based on the template.

### Claim 7

(canceled)

### Claim 8

A computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions comprising:
program instructions to receive an image captured by a camera of a user mobile device;
program instructions to perform object recognition on the image to identify a digital screen and a location of the digital screen in the image, wherein the digital screen is identified to be of a known type, brand, or model of digital screen;
program instructions to retrieve a template of the digital screen based on the known type, brand, or model of digital screen;
program instructions to perform angle-sensitive optical character recognition (OCR) on the location of the digital screen in the image; and
program instructions to determine whether, within the image, text on the digital screen can be extracted.

### Claim 9 (depends on 8)

The computer program product of claim 8, further comprising:
responsive to determining the text can be extracted, program instructions to audibly convey the text to a user of the user mobile device using text-to-speech.

### Claim 10 (depends on 8)

The computer program product of claim 8, further comprising:
responsive to determining the text cannot be extracted, program instructions to guide a user of the user mobile device to re-orient at least one of a position and a rotation of the user mobile device to capture another image.

### Claim 11 (depends on 8)

The computer program product of claim 8, wherein the program instructions to perform angle-sensitive OCR on the location of the digital screen in the image comprise:
program instructions to detect rectangular regions in the image that contain text;
program instructions to calculate angles of the rectangular regions relative to a horizontal axis; and
program instructions to convert text detected in these rectangular regions from image data to text data.

### Claim 12 (depends on 11)

The computer program product of claim 11, further comprising:
wherein the template indicates a set of locations of where text would be located on the digital screen; and
program instructions to compare locations of the rectangular regions in the image to the set of locations of where text would be located on the digital screen based on the template.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the program instructions to determine whether the text on the digital screen can be extracted is based on comparing the locations of the rectangular regions in the image to the set of locations of where text would be located on the digital screen based on the template.

### Claim 14

(canceled)

### Claim 15

A computer system comprising:
one or more computer processors;
one or more computer readable storage media;
program instructions stored on the computer readable storage media for execution by at least one of the one or more processors, the program instructions comprising:
program instructions to receive an image captured by a camera of a user mobile device;
program instructions to perform object recognition on the image to identify a digital screen and a location of the digital screen in the image, wherein the digital screen is identified to be of a known type, brand, or model of digital screen;
program instructions to retrieve a template of the digital screen based on the known type, brand, or model of digital screen;
program instructions to perform angle-sensitive optical character recognition (OCR) on the location of the digital screen in the image; and
program instructions to determine whether, within the image, text on the digital screen can be extracted.

### Claim 16 (depends on 15)

The computer system of claim 15, further comprising:
responsive to determining the text can be extracted, program instructions to audibly convey the text to a user of the user mobile device using text-to-speech.

### Claim 17 (depends on 15)

The computer system of claim 15, further comprising:
responsive to determining the text cannot be extracted, program instructions to guide a user of the user mobile device to re-orient at least one of a position and a rotation of the user mobile device to capture another image.

### Claim 18 (depends on 15)

The computer system of claim 15, wherein the program instructions to perform angle-sensitive OCR on the location of the digital screen in the image comprise:
program instructions to detect rectangular regions in the image that contain text;
program instructions to calculate angles of the rectangular regions relative to a horizontal axis; and
program instructions to convert text detected in these rectangular regions from image data to text data.

### Claim 19 (depends on 18)

The computer system of claim 18, further comprising:
wherein the template indicates a set of locations of where text would be located on the digital screen; and
program instructions to compare locations of the rectangular regions in the image to the set of locations of where text would be located on the digital screen based on the template.

### Claim 20 (depends on 19)

The computer system of claim 19, wherein the program instructions to determine whether the text on the digital screen can be extracted is based on comparing the locations of the rectangular regions in the image to the set of locations of where text would be located on the digital screen based on the template.
