# US 9,436,887 B2 · Apparatus and method for automatic action selection based on image context

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,436,887 B2
- Document ID: US9436887B2
- Status: Granted
- Kind: B2
- Application number: 14/137,373
- Filed: 2013-12-20
- Published: 2014-09-18
- Granted: 2016-09-06
- Earliest claimed: 2013-03-15
- Assignees: OrCam Technologies Ltd.
- Applicants: Yonatan Wexler; Erez Na'Aman; Amnon Shashua
- Inventors: Yonatan Wexler; Erez Na'Aman; Amnon Shashua
- Web page: https://patentgrep.com/patent/US9436887B2

## CPC

- G09B 21/006: {using audible presentation of the information}
- A61F 9/08: Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06F 40/279: Recognition of textual entities
- G06V 10/17: {using hand-held instruments}
- G06V 10/235: {based on user input or interaction}
- G06V 10/243: {by compensating for image skew or non-uniform image deformations}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 20/20: in augmented reality scenes
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G06V 30/1478: {of characters or characters lines}
- G06V 30/40: Document-oriented image-based pattern recognition
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06V 30/418: Document matching, e.g. of document images
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06V 40/172: {Classification, e.g. identification}
- G08B 3/10: using electric transmission; using electromagnetic transmission
- G08B 6/00: Tactile signalling systems, e.g. tactile personal calling systems
- G09B 21/00: Teaching, or communicating with, the blind, deaf or mute (audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B)
- G09B 21/001: {Teaching or communicating with blind persons (G09B21/02 - G09B21/06 take precedence)}
- G09B 21/003: {using tactile presentation of the information, e.g. Braille displays}
- G10L 13/00: Speech synthesis; Text to speech systems
- H04N 23/50: Constructional details
- H04N 23/51: Housings
- H04N 23/61: based on recognised objects
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G01B 11/24: for measuring contours or curvatures
- G02C 11/10: {Electronic devices other than hearing aids}
- G06V 10/16: {using multiple overlapping images; Image stitching}
- G06V 30/144: using a slot moved over the image; using discrete sensing elements at predetermined points; using automatic curve following means
- G06V 30/226: of cursive writing
- G06V 30/43: {Editing text-bitmaps, e.g. alignment, spacing; Semantic analysis of bitmaps of text without OCR}
- H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G02C: SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES (trial frames for testing the eyes A61B3/04; goggles or eyeshields not having the same features as spectacles A61F9/00)

## Abstract

Devices and a method are provided for providing context-related feedback to a user. In one implementation, the method comprises capturing real time image data from an environment of the user. The method further comprises identifying in the image data a hand-related trigger. Multiple context-based alternative actions are associated with the hand-related trigger. Further, the method comprises identifying in the image data an object associated with the hand-related trigger. The object is further associated with a particular context. Also, the method comprises selecting one of the multiple alternative actions based on the particular context. The method further comprises outputting the context-related feedback based on a result of the executed alternative action.

## Claims

### Claim 1

An apparatus for providing context-related feedback to a user, the apparatus comprising:
an image sensor configured to capture real time image data from an environment of the user;
at least one processor device configured to:
identify in the image data a hand-related trigger, wherein multiple context-based alternative actions are associated with the hand-related trigger;
identify in the image data an object based on the hand-related trigger, wherein the object is associated with a particular context;
select one of the multiple alternative actions associated with the hand-related trigger based on the particular context associated with the identified object,
execute the selected alternative action; and
output the context-related feedback based on a result of the executed alternative action.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the hand-related trigger includes an identification in the image data of at least a portion of the user's hand.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to identify a plurality of hand-related triggers, and each hand-related trigger is associated with different multiple context-based alternative actions.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the hand-related trigger includes an erratic movement of the object.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the hand-related trigger includes an atypical movement of the object.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to identify in the image data a persistence of the hand-related trigger; select a differing alternative action from the multiple alternative actions; and execute the selected differing alternative action after outputting the context-related feedback.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the image sensor s configured to be movable with a head of the user.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine the particular context using an identity of the object.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine the particular context using a background of the object.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine the particular context using a location of the hand-related trigger in at least one frame of the image.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine the particular context using information indicative of a location of the user.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine the particular context using information indicative of a time in which the hand-related trigger is identified.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine the particular context using information from at least one previous result of an action executed by the at least one processor.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the at least one processor device is further configured to determine movement of a head of the user and to use the head movement determination to identify the object.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the alternative action is selected from a group comprising: announcing an identity of an inanimate object, announcing an identity of an individual, scene identification, summing money, monitoring a status of a traffic light, saving an individual's name, audibly reading text, audibly reading text summary, monitoring an objected expected to change, identifying a bus number, identifying currency, identifying a credit card, and identifying a pill.

### Claim 16

An apparatus for providing context-related feedback to a user, the apparatus comprising:
an image sensor configured to be worn by the user and to capture real time image data from an environment of the user;
at least one processor device configured to:
identify a trigger in the image data, wherein multiple context-based alternative actions are associated with the trigger;
identify an object in the image data based on the trigger;
identify a background of the object in the image data;
determine a particular context using information about at least one of the identified object and the identified background;
select one of the multiple alternative actions associated with the trigger based on the particular context associated with the identified object;
execute the selected alternative action; and
output the context-related feedback based on a result of the executed alternative action.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the trigger in the image data includes a hand-related trigger.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the trigger in the image data includes an appearance of a new object within a field of view of the image sensor, and wherein the at least one processor device is further configured to:
determine whether the user has focused on the new object.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the at least one processor device is further configured to identify a plurality of triggers, and each trigger is associated with a differing alternative action.

### Claim 20 (depends on 16)

The apparatus of claim 16, wherein the at least one processor device is further configured to determine the particular context using information indicative of a location of the user.

### Claim 21 (depends on 16)

The apparatus of claim 16, wherein the at least one processor device is further configured to determine the particular context using information retrieved from audio processing.

### Claim 22 (depends on 16)

The apparatus of claim 16, wherein the at least one processor device is further configured to determine movement of a head of the user and to use the head movement determination to determine the particular context.

### Claim 23

A method for providing context-related feedback to a user, the method comprising:
capturing real time image data from an environment of the user;
identifying in the image data a hand-related trigger, wherein multiple context-based alternative actions are associated with the hand-related trigger;
identifying in the image data an object based on the hand-related trigger, wherein the object is further-associated with a particular context;
selecting one of the multiple alternative actions associated with the hand-related trigger based on the particular context associated with the identified object; and
outputting the context-related feedback based on a result of the executed alternative action.

### Claim 24 (depends on 23)

A software product stored on a non-transitory computer readable medium and comprising data and computer implementable instructions for carrying out the method of claim 23.