# US 2017/0289341 A1 · Intuitive computing methods and systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0289341 A1
- Document ID: US20170289341A1
- Status: Published application
- Kind: A1
- Application number: 15/620,380
- Filed: 2017-06-12
- Published: 2017-10-05
- Earliest claimed: 2009-10-28
- Assignees: Digimarc Corporation
- Applicants: Digimarc Corporation
- Inventors: Tony F. Rodriguez; Geoffrey B. Rhoads; Bruce L. Davis; Gilbert B. Shaw
- Web page: https://patentgrep.com/patent/US20170289341A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 9/50: Allocation of resources, e.g. of the central processing unit [CPU]
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 17/00: Speaker identification or verification techniques
- H04M 1/72448: with means for adapting the functionality of the device according to specific conditions
- H04M 1/72469: for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
- H04W 72/02: Selection of wireless resources by user or terminal
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, the phone is guided in various of its intuitive computing operations by user-spoken clues. A discovery session may be launched by the user speaking a cueing expression, which serves to switch the device from a lower activity state to a heightened alert state. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

## Claims

### Claim at position 1

1-82. (canceled)

### Claim 83

A method employing a device equipped with a processor, a display, a camera and a microphone, the camera capturing imagery depicting plural items in a user's physical environment, the method comprising the acts:
capturing first speech of the user, with the device microphone;
the device processor detecting that the captured first speech includes a cueing expression, and in response to detection of the cueing expression, the device switching from a lower activity state to a heightened alert state, in the heightened alert state the device performing functionality including:
capturing second user speech with the device microphone;
sending data corresponding to the second user speech to a recognition module, and receiving recognized second speech data in return, the recognized second user speech indicating one of said plural items depicted in the captured imagery as of particular user interest;
based on one or more descriptors included in the recognized second speech data, determining a first of said plural depicted items as being of likely user interest;
presenting a marking on the device display, at a location indicating said first item;
capturing third user speech with the device microphone, the captured third user speech being different than the second user speech;
sending data corresponding to the third user speech to the recognition module, and receiving recognized third speech data in return, the recognized third speech data again indicating one of said plural items as of particular user interest;
based on one or more descriptors included in the recognized third speech data, determining that a second, different one of said plural depicted items is of greater interest to the user than the first item;
moving said marking on the device display to a location indicating said second item; and
taking an action based on the second item, said action including presenting information related to the second item to the user;
wherein the device is not on heightened alert all the time, but is cued into activation from a lower activity state by the cueing expression, thereby bounding the device's processing efforts, and the descriptors in the recognized second and third speech data iteratively guide the device in identifying which of the plural items in the user's physical environment is of user interest, thereby further bounding the device's processing efforts.

### Claim 84 (depends on 83)

The method of claim 83 that includes sending data corresponding to the second item to a recognition engine, receiving second item recognition data in return, and taking an action based on the second item recognition data.

### Claim 85 (depends on 84)

The method of claim 84 in which the second item is a face of a person, and the method includes sending data corresponding to said face to a facial recognition engine, receiving information about said person in return, and presenting the received information to the user.

### Claim 86 (depends on 83)

The method of claim 83 that further includes receiving a user confirmation that the second item is of interest, prior to presenting information related to the second item to the user.

### Claim 87 (depends on 83)

The method of claim 83 that further includes presenting information related to the second item to the user on the device display.

### Claim 88 (depends on 83)

The method of claim 83 in which said device is battery-powered, wherein the method includes the device processor detecting the cueing expression using battery power.

### Claim 89

A method employing a device equipped with a processor, a camera and a microphone, the camera capturing imagery depicting plural items in a user's physical environment, the method comprising the acts:
capturing first speech of the user, with the device microphone;
detecting, with said device processor, that the captured first speech includes a cueing expression;
in response to detection of the cueing expression, switching the device from a lower activity state to a heightened alert state, in the heightened alert state the device performing functionality including:
capturing second speech of the user;
sending data from the device, said data including data corresponding to the second user speech and data corresponding to the captured imagery, and receiving data, including (a) recognized second speech data and (b) recognition-processed data about a subject depicted in the imagery, in return; and
taking an action based on said received data, including presenting information based on the recognition-processed data to the user;
wherein the device is not on heightened alert all the time, but is cued into activation from a lower activity state by the cueing expression, thereby bounding the device's processing efforts, and wherein, in its heightened alert state, the device cooperates with a remote computer system to recognition-process imagery captured by the device camera.

### Claim 90 (depends on 89)

The method of claim 89 in which said imagery depicts plural items in a user's physical environment, and said functionality further includes:
based on one or more descriptors included in the recognized second speech data, determining a first of said plural depicted items as being of likely user interest;
presenting a marking on a display of the device, at a location indicating said first item;
capturing third user speech with the device microphone;
sending data corresponding to the third user speech from the device, and receiving recognized third speech data in return;
based on one or more descriptors included in the recognized third speech data, determining that a second, different one of said plural depicted items is of greater interest to the user than the first item;
moving said marking on the device display to a location indicating said second item;
receiving a user confirmation that the second item is of interest; and
sending data corresponding to the second item from the device for recognition processing;
wherein descriptors in the recognized second and third speech data iteratively guide the device in identifying which of the plural items in the user's physical environment is of user interest, thereby further bounding the device's processing efforts.

### Claim 91 (depends on 89)

The method of claim 89 in which the cueing expression consists of two words.

### Claim 92 (depends on 89)

The method of claim 89 in which said device is battery-powered, wherein the method includes the device processor detecting the cueing expression using battery power.

### Claim 93

A tangible computer readable medium containing instructions to configure a device, equipped with a display, a camera and a microphone, to perform acts including:
capturing first speech of the user;
detecting that the captured first speech includes a cueing expression, and in response to detection of the cueing expression, switching the device from a lower activity state to a heightened alert state, in the heightened alert state the instructions configuring the device to perform functions including:
capturing second user speech;
sending data corresponding to the second user speech to a recognition module, and receiving recognized second speech data in return;
based on one or more descriptors included in the recognized second speech data, determining a first of said plural depicted items as being of likely user interest;
presenting a marking on the display, at a location indicating said first item;
capturing third user speech, the captured third user speech being different than the second user speech;
sending data corresponding to the third user speech to the recognition module, and receiving recognized third speech data in return;
based on one or more descriptors included in the recognized third speech data, determining that a second, different one of said plural depicted items is of greater interest to the user than the first item;
moving said marking on the display to a location indicating said second item; and
taking an action based on the second item, said action including presenting information related to the second item to the user;
wherein the device is not on heightened alert all the time, but is cued by said instructions into activation from a lower activity state by the cueing expression, thereby bounding the device's processing efforts, and the instructions enable descriptors in the recognized second and third speech data to iteratively guide the device in identifying which of the plural items in the user's physical environment is of user interest, thereby further bounding the device's processing efforts.
