# US 2017/0286383 A1 · Augmented imaging assistance for visual impairment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286383 A1
- Document ID: US20170286383A1
- Status: Published application
- Kind: A1
- Application number: 15/242,940
- Filed: 2016-08-22
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Anirudh Koul; Ao Li; Elias Haroun; Irene Wen Ling Chen; Shweta Sharma; Christiano Bianchet; Saqib Shaikh; Stéphane Morichère-Matte; Biing Tsyr Lai; Nathan Pak Kei Lam; Wendy Lu
- Web page: https://patentgrep.com/patent/US20170286383A1

## CPC

- A61F 9/08: Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/20: in augmented reality scenes
- G09B 21/00: Teaching, or communicating with, the blind, deaf or mute (audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B)
- G09B 21/008: {using visual presentation of the information for the partially sighted}
- H04M 1/2474: {Telephone terminals specially adapted for disabled people (user interfaces for cordless or mobile telephones specially adapted for disabled users H04M1/72475; network based special services or facilities for hearing-impaired persons H04M3/42391; devices for conversing with the deaf-blind G09B21/04)}
- H04M 1/2476: {for a visually impaired user (illuminating or arrangements for improving visibility of characters on dials H04M1/22)}
- G06V 30/10: Character recognition
- H04M 2203/359: Augmented reality
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

Systems, apparatuses, services, platforms, and methods are discussed herein that provide assistance for user interface devices. In one example, an assistance application is provided comprising an imaging system configured to capture an image of a scene, an interface system configured to provide data associated with the image to a distributed assistance service that responsively processes the data to recognize properties of the scene and establish feedback for a user based at least on the properties of the scene, and a user interface configured to provide the feedback to the user.

## Claims

### Claim 1

An assistance application provided for a user interface device, comprising:
an imaging system configured capture an image of a scene;
an assistance interface configured to provide data associated with the image to a distributed assistance service that responsively processes the data to recognize properties of the scene and establish feedback for a user based at least on the properties of the scene;
a user interface configured to provide the feedback to the user.

### Claim 2 (depends on 1)

The assistance application of claim 1, comprising:
the assistance interface configured to indicate to the distributed assistance service a scene recognition request for the data associated with the image, and responsively receive at least partial recognition information for at least one element in the scene.

### Claim 3 (depends on 2)

The assistance application of claim 2, wherein the partial recognition information comprises graphical annotations related to descriptions of objects in the scene, and comprising:
the assistance interface configured to merge the graphical annotations with the scene;
the user interface configured to present the graphical annotations overlaid with the scene to the user.

### Claim 4 (depends on 1)

The assistance application of claim 1, comprising:
the assistance interface configured to receive repositioning instructions determined by the distributed assistance service to increase a recognition level of at least one element in the scene;
the user interface configured to present the repositioning instructions to the user.

### Claim 5 (depends on 4)

The assistance application of claim 4, wherein the repositioning instructions comprise directional notifications which prompt the user to move an imaging sensor of the imaging system to increase the recognition level of the at least one element in the scene.

### Claim 6 (depends on 4)

The assistance application of claim 4, comprising:
the user interface configured to indicate to the user an alert to capture an image based on a state of the repositioning instructions.

### Claim 7 (depends on 1)

The assistance application of claim 1, comprising:
the assistance interface configured to indicate to the distributed assistance service a scene recognition request for the data associated with the image, and responsively receive a description of the scene; and
the user interface configured to present the description of the scene to the user.

### Claim 8 (depends on 7)

The assistance application of claim 7, comprising:
the user interface configured to receive one or more queries from the user related to the description of the scene;
the assistance interface configured to indicate to the distributed assistance service further scene recognition requests related to the one or more queries related to the description of the scene and responsively receive one or more further descriptions of the scene; and
the user interface configured to present the one or more further descriptions of the scene to the user.

### Claim 9 (depends on 1)

The assistance application of claim 1, comprising:
the assistance interface configured to indicate a document recognition request with the data associated with the image to the distributed assistance service, wherein the distributed assistance service responsively recognizes one or more textual formatting properties of a document captured in the image;
the assistance interface configured to receive document description information determined based at least on the one or more textual formatting properties of a document captured in the image; and
the user interface configured to present the document description information to the user.

### Claim 10

An apparatus comprising:
one or more computer readable storage media;
program instructions stored on the one or more computer readable storage media that, when executed by a processing system, direct the processing system to at least:
receive an image of a scene captured by an imaging element;
provide data associated with the image to a remote assistance interface that responsively selects one or more distributed recognition services to recognize properties of the scene and establish feedback for a user based at least on the properties of the scene;
provide the feedback to the user via a user interface.

### Claim 11 (depends on 10)

The apparatus of claim 10, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
indicate to the remote assistance interface a scene recognition request for the data associated with the image, and responsively receive at least partial recognition information for at least one element in the scene.

### Claim 12 (depends on 11)

The apparatus of claim 11, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
receive a query from the user related to the at least one element in the scene;
indicate the query to the remote assistance interface that responsively selects among the one or more distributed recognition services to provide further recognition information;
present the further recognition information to the user.

### Claim 13 (depends on 10)

The apparatus of claim 10, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
receive repositioning instructions determined by the one or more distributed recognition services to increase a recognition level of at least one element in the scene;
present the repositioning instructions to the user.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the repositioning instructions comprise directional notifications which prompt the user to move the imaging element to increase the recognition level of the at least one element in the scene.

### Claim 15 (depends on 13)

The apparatus of claim 13, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
indicate to the user an alert to capture an image based on a state of the repositioning instructions.

### Claim 16 (depends on 10)

The apparatus of claim 10, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
indicate to the remote assistance interface a scene recognition request for the data associated with the image, and responsively receive a description of the scene; and
present the description of the scene to the user.

### Claim 17 (depends on 16)

The apparatus of claim 16, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
receive one or more queries from the user related to the description of the scene;
indicate to the remote assistance interface further scene recognition requests for the one or more queries related to the description of the scene and responsively receive one or more further descriptions of the scene; and
present the one or more further descriptions of the scene to the user.

### Claim 18 (depends on 10)

The apparatus of claim 10, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
indicate a document recognition request with the data associated with the image to the remote assistance interface, wherein the remote assistance interface responsively selects at least a document recognition service among the one or more distributed recognition services to recognize one or more textual formatting properties of a document captured in the image;
receive document description information determined based at least on the one or more textual formatting properties of a document captured in the image; and
present the document description information to the user.

### Claim 19 (depends on 18)

The apparatus of claim 18, comprising further program instructions, when executed by the processing system, direct the processing system to at least:
based on the document description information, perform at least one search query using descriptors in the document description information to retrieve further descriptors for the document; and
present the further descriptors to the user.

### Claim 20

A user interface device, comprising:
an imaging apparatus configured capture one or more images of a scene;
an assistance application configured to provide data associated with the one or more images to an assistance computing interface that responsively selects one or more distributed recognition services to recognize properties of the scene to establish graphical annotations related to the scene based at least on the properties of the scene; and
a network interface configured to communicate with the assistance computing interface.
