# US 2019/0320056 A1 · Intuitive computing methods and systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0320056 A1
- Document ID: US20190320056A1
- Status: Published application
- Kind: A1
- Application number: 16/390,798
- Filed: 2019-04-22
- Published: 2019-10-17
- Earliest claimed: 2009-10-28
- Assignees: Digimarc Corporation
- Applicants: Digimarc Corporation
- Inventors: Geoffrey B. Rhoads; Tony F. Rodriguez; Gilbert B. Shaw; Bruce L. Davis; William Y. Conwell
- Web page: https://patentgrep.com/patent/US20190320056A1

## CPC

- H04W 4/02: Services making use of location information
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G01C 21/36: Input/output arrangements for on-board computers
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 18/24: Classification techniques
- G06Q 10/10: Office automation; Time management
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09G 5/00: Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T)
- H04M 1/724: User interfaces specially adapted for cordless or mobile telephones
- H04M 1/72403: with means for local support of applications that increase the functionality
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- H04N 23/80: Camera processing pipelines; Components thereof
- H04W 4/027: {using movement velocity, acceleration information}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06V 20/20: in augmented reality scenes
- H04W 4/029: Location-based management or tracking services
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user's environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone's camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user's apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

## Claims

### Claim at position 1

1-14. (canceled)

### Claim 15

A method comprising the acts:
receiving imagery depicting a scene; and
presenting said imagery depicting the scene to a user on a display, overlaid with indicia corresponding to supplemental data that relate to one or more features in the scene;
wherein the method further includes narrowing a universe of possible indicia for display, down to a smaller number of indicia, said narrowing being a function of first and second criteria, where:
the first criterion comprises one or more factors selected from the list: actions previously taken by the user expressing interest or disinterest in presented indicia; actions previously taken by one or more third parties expressing interest or disinterest in presented indicia; time of day; topical preference data for the user; areas of the display in which indicia would respectively be overlaid; and payments offered by one or more commercial parties for presentation of their indicia; and
the second criterion comprises a setting of a user-settable verbosity control;
wherein the user can vary how much information, identified using the first criterion, is overlaid on the displayed scene, by varying the setting of said verbosity control.

### Claim 16 (depends on 15)

The method of claim 15 wherein said indicia comprise baubles.

### Claim 17 (depends on 15)

The method of claim 15 in which some of said indicia are commercial in nature, and some are non-commercial.

### Claim 18 (depends on 17)

The method of claim 17 that comprises limiting a number of said commercial indicia, in accordance with the user-settable verbosity control.

### Claim 19 (depends on 17)

The method of claim 17 in which said limiting comprises establishing a maximum ratio of commercial indicia to non-commercial indicia, in accordance with the user-settable verbosity control.

### Claim 20 (depends on 15)

The method of claim 15 wherein at least one of said indicia presented on the display is a result of an auction process, said auction process having been based at least in part on information about a context of the user.

### Claim 21

In an augmented reality method practiced using a user's portable system, a method of preventing a display of the portable system from being overrun by augmentations vying for the user's attention, the method comprising the acts:
capturing imagery depicting a scene from the user's physical environment, with a camera apparatus of the portable system;
recognizing multiple subjects depicted in the captured imagery;
identifying supplemental data that relate to plural of the subjects recognized in the depicted scene, including first supplemental data that relates to a first of said subjects;
combining the captured imagery with indicia corresponding to said identified supplemental data; and
presenting the combined imagery to the user on the display of said portable system;
wherein the method further includes a software-configured hardware processor narrowing a universe of possible indicia for display down to a smaller number of indicia, said narrowing being a function of first and second different criteria, where:
the first criterion comprises one or more factors selected from the list: actions previously taken by the user expressing interest or disinterest in presented indicia; actions previously taken by one or more third parties expressing interest or disinterest in presented indicia; time of day; topical preference data for the user; areas of the display in which indicia would respectively be overlaid; and payments offered by one or more commercial parties for presentation of their indicia; and
the second criterion comprises data created for the purpose of display control, namely data from a user-settable verbosity control;
wherein the user can vary how much information identified using the first criterion is overlaid on the displayed scene by varying the setting of said verbosity control.

### Claim 22 (depends on 21)

The method of claim 21 wherein said indicia comprise baubles.

### Claim 23 (depends on 21)

The method of claim 21 in which some of said indicia are commercial in nature, and some are non-commercial.

### Claim 24 (depends on 23)

The method of claim 23 that comprises limiting a number of said commercial indicia, in accordance with a user-settable control.

### Claim 25 (depends on 23)

The method of claim 23 in which said limiting comprises establishing a maximum ratio of commercial indicia to non-commercial indicia, in accordance with a user-settable control.

### Claim 26 (depends on 21)

The method of claim 21 wherein at least one of said indicia presented on the display is a result of an auction process, said auction process having been based at least in part on information about a context of the user.

### Claim 27 (depends on 21)

The method of claim 21 wherein at least one of said indicia is selected for presentation on the display based on data from a sensor in a portable device conveyed by the user, and also based on stored user history data.

### Claim 28 (depends on 21)

The method of claim 21 in which said stored user history data comprises data about previous interactions by the user with indicia, wherein indicia that are evident favorites of the user are more likely presented.

### Claim 29 (depends on 21)

The method of claim 21 wherein the appearance of one of said indicia evolves over time, as additional information is discerned about the subject to which it corresponds.

### Claim 30 (depends on 21)

The method of claim 21 wherein the first criterion comprises actions previously taken by the user expressing interest in certain of the indicia.

### Claim 31 (depends on 21)

The method of claim 21 wherein the first criterion comprises actions previously taken by the user expressing disinterest in certain of the indicia.

### Claim 32 (depends on 21)

The method of claim 21 wherein the first criterion comprises actions previously taken by a third party expressing interest in certain of the indicia.

### Claim 33 (depends on 21)

The method of claim 21 wherein the first criterion comprises actions previously taken by a third party expressing disinterest in certain of the indicia.

### Claim 34

A portable system comprising:
a battery;
a camera portion;
a display;
a memory;
a processor powered by the battery and responsive to instructions stored in the memory;
the memory containing instructions that configure the system to perform acts including:
capturing imagery depicting a scene from the user's physical environment, with said camera portion of the system;
recognizing multiple subjects depicted in the captured imagery;
identifying supplemental data that relate to plural of the subjects recognized in the depicted scene, including first supplemental data that relates to a first of said subjects;
combining the captured imagery with indicia corresponding to said identified supplemental data;
said instructions further configuring the system to perform acts including:
narrowing a universe of possible indicia for display down to a smaller number of indicia, said narrowing being a function of first and second different criteria, where:
the first criterion comprises one or more factors selected from the list: actions previously taken by the user expressing interest or disinterest in presented indicia; actions previously taken by one or more third parties expressing interest or disinterest in presented indicia; time of day; topical preference data for the user; areas of the display in which indicia would respectively be overlaid; and payments offered by one or more commercial parties for presentation of their indicia; and
the second criterion comprises data created for the purpose of display control, namely data from a user-settable verbosity control; and
presenting the combined imagery to the user on the display of said portable system;
wherein the user can vary how much information identified using the first criterion is overlaid on the displayed scene by varying the setting of said verbosity control.
