# US 2017/0257474 A1 · Intuitive computing methods and systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0257474 A1
- Document ID: US20170257474A1
- Status: Published application
- Kind: A1
- Application number: 15/467,819
- Filed: 2017-03-23
- Published: 2017-09-07
- Earliest claimed: 2009-10-28
- Assignees: Digimarc Corporation
- Applicants: Digimarc Corporation
- Inventors: Geoffrey B. Rhoads; Tony F. Rodriguez; Gilbert B. Shaw; Bruce L. Davis; William Y. Conwell
- Web page: https://patentgrep.com/patent/US20170257474A1

## CPC

- H04W 4/02: Services making use of location information
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G01C 21/36: Input/output arrangements for on-board computers
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 18/24: Classification techniques
- G06Q 10/10: Office automation; Time management
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09G 5/00: Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T)
- H04M 1/724: User interfaces specially adapted for cordless or mobile telephones
- H04M 1/72403: with means for local support of applications that increase the functionality
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- H04N 23/80: Camera processing pipelines; Components thereof
- H04W 4/027: {using movement velocity, acceleration information}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06V 20/20: in augmented reality scenes
- H04W 4/029: Location-based management or tracking services
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

## Claims

### Claim at position 1

1-5. (canceled)

### Claim 6

A method of operating a camera-equipped portable device that processes image data, the device being conveyed by a user, the method including the following acts:
performing an initial set of plural different image processing operations; and
without express user command, invoking additional image processing operations as circumstances warrant;
wherein the device acts autonomously to fulfill inferred or anticipated user desires.

### Claim at position 3

7-10. (canceled)

### Claim 11 (depends on 6)

The method of claim 6 wherein the invoking act includes invoking additional image processing operations based on a circumstance including at least one of:
(a) location;
(b) time of day;
(c) proximity to one or more people;
(d) an output based on the initial set of image processing operations; or
(e) a statistical model of user behavior.

### Claim 12 (depends on 6)

The method of claim 6 wherein the device acts autonomously to determine the value of a group of coins imaged by the device's camera.

### Claim 13 (depends on 6)

The method of claim 6 that includes selecting a first set of additional image processing operations to be performed, from a larger second set of possible image processing operations, based on data indicating one or more of:
(a) device resource usage;
(b) resource demands associated with different of the possible operations; and
(c) correspondence between different of the possible operations.

### Claim 14 (depends on 6)

The method of claim 6 that includes discerning one or more visual features within a scene represented by the image data, and storing data related to each such feature in association with a corresponding identifier, wherein the identifier is based on at least two of the following:
(a) a session ID;
(b) an explicit object ID; and
(c) data derived from the feature, or derived from a related circumstance.

### Claim at position 8

15-20. (canceled)

### Claim 21

An image processing method comprising the acts:
capturing imagery with a camera portion of a mobile system, the system also including a processor controlled by configuration instructions stored in a memory;
selecting a first recognition process to apply to the captured imagery, from among several recognition processes that the system is equipped to apply, said selection of the first recognition process being based on context information rather than on express user instruction;
launching the selected first recognition process, employing said processor, to discern information about a subject depicted in the captured imagery;
receiving first data, after said selected first recognition process has been launched, said first data comprising (a) user input data indicating express or implied user encouragement or discouragement of said first recognition process by the user, and/or (b) a detection state metric, representing a quantified likelihood that a recognition goal sought by the first recognition process will be reached; and
allocating increased or decreased resources to said first recognition process based on said received first data, said allocation of increased resources being apart from increased resource usage due to any increasing base complexity of successive stages of the selected first recognition process.

### Claim 22 (depends on 21)

The method of claim 21 in which the received first data comprises user input data indicating user encouragement of said process, and the method includes allocating increased resources to said first recognition process based on said indication of encouragement, wherein the mobile system seems to respond intuitively to a user interest in said subject.

### Claim 23 (depends on 21)

The method of claim 21 in which the received first data indicates comprises user input data indicating user discouragement of said process, and the method includes allocating reduced resources to said first recognition process based on said indication of discouragement, wherein the mobile system seems to respond intuitively to a user dis-interest in said subject.

### Claim 24 (depends on 21)

The method of claim 21 in which the received first data comprises a detection state metric representing a quantified likelihood that the recognition goal sought by the first recognition process will be reached, and the method includes allocating increased resources to the first recognition process as a consequence of said detection state metric.

### Claim 25 (depends on 21)

The method of claim 21 in which the received first data comprises a detection state metric representing a quantified likelihood that the recognition goal sought by the first recognition process will be reached, and the method includes allocating decreased resources to the first recognition process as a consequence of said detection state metric.

### Claim 26 (depends on 21)

The method of claim 21 that includes:
receiving user data indicating express or implied user encouragement or discouragement of the first recognition process; and
receiving a detection state metric representing a quantified likelihood that the recognition goal sought by the first recognition process will be reached;
wherein the method includes allocating resources to said first recognition process based on both (1) said received user data; and (2) said received detection state metric.

### Claim 27 (depends on 21)

The method of claim 21 in which the context information on which selection of the first recognition process is based, comprises image information.

### Claim 28 (depends on 27)

The method of claim 27 that includes selecting said first recognition process, from the several recognition processes the system is equipped to apply, based on results of baseline image processing operations applied to the captured imagery.

### Claim 29 (depends on 27)

The method of claim 27 that includes selecting said first recognition process, from the several recognition processes the system is equipped to apply, also based on context data distinct from image data.

### Claim 30 (depends on 27)

The method of claim 27 in which the context information on which selection of the first recognition process is based, also comprises location information.

### Claim 31 (depends on 21)

The method of claim 21 in which the context information on which selection of the first recognition process is based, comprises location information.

### Claim 32 (depends on 21)

The method of claim 21 that includes selecting said first recognition process, from the several recognition processes the system is equipped to apply, based on context data distinct from image data.

### Claim 33 (depends on 21)

The method of claim 21 that includes:
selecting both first and second recognition processes to apply to the captured imagery, from among said several recognition processes the system is equipped to apply, based on context information rather than on express user instruction;
launching both the first and second recognition processes;
receiving second data, after said second recognition process has been launched, said second data comprising (a) user input data indicating express or implied user encouragement or discouragement of said second recognition process by the user, and/or (b) a detection state metric, representing a quantified likelihood that a recognition goal sought by the second recognition process will be reached; and
based on the received first and second data, allocating increased resources to the first recognition process, and allocating reduced resources to the second recognition process.

### Claim 34 (depends on 21)

The method of claim 21 in which allocating increased resources to said first recognition process comprises making GPU resources available instead of CPU resources, allocating more scratchpad memory for calculations, allowing access to faster-responding memory, allowing increased power consumption, or allowing access to a faster network connection.

### Claim 35

A non-transitory computer readable storage medium containing software instructions to configure a mobile system, equipped with a processor and a camera, to perform acts including:
selecting a first recognition process to apply to imagery captured by the camera, from among several recognition processes that the system is equipped to apply, said selection of the first recognition process being based on context information rather than on express user instruction;
launching the selected first recognition process, to discern information about a subject depicted in the captured imagery;
receiving first data, after said selected first recognition process has been launched, said first data comprising (a) user input data indicating express or implied user encouragement or discouragement of said first recognition process by the user, and/or (b) a detection state metric, representing a quantified likelihood that a recognition goal sought by the first recognition process will be reached; and
allocating increased or decreased resources to said first recognition process based on said received first data, said allocation of increased resources being apart from increased resource usage due to any increasing base complexity of successive stages of the selected first recognition process.
