# US 2014/0152855 A1 · Image Capture and Identification System and Process

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0152855 A1
- Document ID: US20140152855A1
- Status: Published application
- Kind: A1
- Application number: 14/173,195
- Filed: 2014-02-05
- Published: 2014-06-05
- Earliest claimed: 2000-11-06
- Assignees: Nant Holdings IP, LLC
- Applicants: Nant Holdings IP, LLC
- Inventors: Wayne C. Boncyk, V; Ronald H. Cohen
- Web page: https://patentgrep.com/patent/US20140152855A1

## CPC

- G06Q 30/0277: {Online advertisement}
- A61F 9/08: Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
- A63F 13/00: Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F 13/20: Input arrangements for video game devices
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/335: using Internet
- A63F 13/35: Details of game servers
- A63F 13/45: Controlling the progress of the video game
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- A63F 13/70: Game security or game management aspects
- A63F 13/792: for payment purposes, e.g. monthly subscriptions
- A63F 13/92: Video game devices specially adapted to be hand-held while playing
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 16/24: Querying
- G06F 16/29: Geographical information databases
- G06F 16/50: of still image data
- G06F 16/51: Indexing; Data structures therefor; Storage structures
- G06F 16/583: using metadata automatically derived from the content
- G06F 16/5838: {using colour}
- G06F 16/5846: {using extracted text}
- G06F 16/5854: {using shape and object relationship}
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 16/94: {Hypermedia (Hyperlinking G06F40/134)}
- G06F 16/9554: {by using bar codes}
- G06F 16/9558: {Details of hyperlinks; Management of linked annotations}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/23: Clustering techniques
- G06F 40/134: Hyperlinking
- G06Q 10/02: Reservations, e.g. for tickets, services or events
- G06Q 20/102: {Bill distribution or payments}
- G06Q 20/14: specially adapted for billing systems
- G06Q 20/202: {Interconnection or interaction of plural electronic cash registers [ECR] or to host computer, e.g. network details, transfer of information from host to ECR or from ECR to ECR}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 20/24: Credit schemes, i.e. "pay after"
- G06Q 20/327: {Short range or proximity payments by means of M-devices}
- G06Q 20/3567: {Software being in the reader}
- G06Q 20/40145: {Biometric identity checks}
- G06Q 30/0217: involving input on products or services in exchange for incentives or rewards
- G06Q 30/0241: Advertisements
- G06Q 30/0253: {During e-commerce, i.e. online transactions}
- G06Q 30/0257: {User requested}
- G06Q 30/0267: {Wireless devices}
- G06Q 30/0268: {at point-of-sale [POS]}
- G06Q 30/0269: {based on user profile or attribute}
- G06Q 30/04: Billing or invoicing
- G06Q 30/0601: Electronic shopping [e-shopping]
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0635: {replenishment orders; recurring orders}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06Q 40/00: Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q 90/00: Systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not involving significant data processing
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06T 7/13: Edge detection
- G06T 7/136: involving thresholding
- G06T 7/194: involving foreground-background segmentation
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/33: using feature-based methods
- G06T 7/337: {involving reference images or patches}
- G06T 7/73: using feature-based methods
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/56: relating to colour
- G06V 10/7515: {Shifting the patterns to accommodate for positional errors}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/20: in augmented reality scenes
- G06V 30/142: using hand-held instruments; Constructional details of the instruments
- G06V 30/224: of printed characters having additional code marks or containing code marks
- G07F 17/3206: {Player sensing means, e.g. presence detection, biometrics}
- G07F 17/3241: {Security aspects of a gaming system, e.g. detecting cheating, device integrity, surveillance (computer security G06F21/00)}
- G07F 17/3244: {Payment aspects of a gaming system, e.g. payment schemes, setting payout ratio, bonus or consolation prizes}
- G09B 21/006: {using audible presentation of the information}
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04N 1/00244: {with a server, e.g. an internet server (fax-servers or the like for store and forward H04N1/324)}
- H04N 5/91: Television signal processing therefor
- H04N 7/183: {for receiving images from a single remote source}
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- H04N 21/23109: {by placing content in organized collections, e.g. EPG data repository (details of retrieval of video data and associated meta data in video databases G06F16/739)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/254: Management at additional data server, e.g. shopping server, rights management server {(arrangements for maintenance or administration in data networks H04L41/00; network services using third party service providers H04L67/53)}
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4722: for requesting additional data associated with the content
- H04N 21/4781: {Games}
- H04N 21/47815: {Electronic shopping (payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12)}
- H04N 21/4782: Web browsing {, e.g. WebTV}
- H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)}
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04N 21/8126: {involving additional data, e.g. news, sports, stocks, weather forecasts}
- H04N 21/8173: {End-user applications, e.g. Web browser, game}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image}
- H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet
- G06F 2218/12: Classification; Matching
- H04N 2201/3253: Position information, e.g. geographical position at time of capture, GPS data
- H04N 2201/3254: Orientation, e.g. landscape or portrait; Location or order of the image data, e.g. in memory
- H04N 2201/3274: Storage or retrieval of prestored additional information
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07F: COIN-FREED OR LIKE APPARATUS (coin sorting G07D3/00; coin testing G07D5/00; {handling coins or paper currencies apart from payment activated apparatus G07D; payment architectures, schemes or protocols G06Q20/00})
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A digital image of the object is captured and the object is recognized from plurality of objects in a database. An information address corresponding to the object is then used to access information and initiate communication pertinent to the object.

## Claims

### Claim at position 1

1-28. (canceled)

### Claim 29

A mobile device comprising:
an image sensor configured to capture image data of a physical environment that includes at least a first object; and
an image processing platform coupled with the image sensor and configured to:
obtain the image data from the image sensor;
recognize a first target object based at least in part on a visual appearance derived from the image data;
retrieve information related to the first target object, including an information address;
determine at least one of a position and an orientation of the first object with respect to a second object in the physical environment based on the visual appearance; and
enable an interaction with a computer-system based software application via the information address as a function of the information, and as a function of at least one of the position and the orientation.

### Claim 30 (depends on 29)

The mobile device of claim 29, wherein the image processing platform is configured to determine at least one of the position and the orientation of the first object by using a database having different views of the first target object from different orientations to determine a relative position and orientation.

### Claim 31 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-system based software application comprises playing a sound.

### Claim 32 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-system based software application comprises displaying at least one of a text, a graphic, an animation, a video, an advertisement, and an indicator light.

### Claim 33 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-based software application comprises initiating a transaction.

### Claim 34 (depends on 33)

The mobile device of claim 33, wherein the transaction comprises allowing a user to purchase the first object via an online transaction.

### Claim 35 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-based software application comprises an interaction with an account.

### Claim 36 (depends on 35)

The mobile device of claim 35, wherein the account comprises an account linked to the mobile device.

### Claim 37 (depends on 35)

The mobile device of claim 35, wherein the account comprises an account linked to a user.

### Claim 38 (depends on 35)

The mobile device of claim 35, wherein the account comprises an account linked to a terminal.

### Claim 39 (depends on 35)

The mobile device of claim 35, wherein the account comprises an account linked to a credit card.

### Claim 40 (depends on 35)

The mobile device of claim 35, wherein the account comprises a bank account.

### Claim 41 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-based software application comprises streaming audio samples.

### Claim 42 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-based software application comprises interacting with an enterprise that manufactured the first object.

### Claim 43 (depends on 29)

The mobile device of claim 29, wherein the interaction with the computer-based software application comprises interacting with the first object.

### Claim 44 (depends on 29)

The mobile device of claim 29, wherein the second object comprises the mobile device.

### Claim 45 (depends on 29)

The mobile device of claim 29, where in the image data comprises video data.

### Claim 46 (depends on 29)

The mobile device of claim 29, wherein the first object comprises a tag.

### Claim 47 (depends on 29)

The mobile device of claim 29, wherein the first object lacks a tag.

### Claim 48 (depends on 29)

The mobile device of claim 29, wherein the mobile device further comprises a location tracking sensor configured to track a current location of the mobile device.

### Claim 49 (depends on 48)

The mobile device of claim 48, wherein the location tracking sensor comprises a GPS transceiver.

### Claim 50 (depends on 48)

The mobile device of claim 48, wherein the image processing platform is further configured to enable the interaction with the computer-system based software application as a function of the information, as a function of at least one of the position and the orientation, and as a function of the current location of the mobile device.

### Claim 51 (depends on 29)

The mobile device of claim 29, wherein the mobile device further comprises a security apparatus.

### Claim 52 (depends on 29)

The mobile device of claim 29, wherein the image sensor comprises a security camera.

### Claim 53 (depends on 29)

The mobile device of claim 29, wherein the image data comprises a representation of on screen content.

### Claim 54 (depends on 29)

The mobile device of claim 29, wherein the first object comprises a game object.

### Claim 55 (depends on 29)

The mobile device of claim 29, further comprising an audio module configured to present audio information to a visually impaired user based on the image data.

### Claim 56 (depends on 29)

The mobile device of claim 29, wherein the interaction comprises registering a vote.

### Claim 57

A mobile device comprising:
a location tracking sensor configured to track a current location of the mobile device;
an image sensor configured to capture image data of a physical environment that includes at least a first object; and
an image processing platform coupled with the image sensor and configured to:
obtain the image data from the image sensor;
recognize a first target object based at least in part on a visual appearance derived from the image data;
retrieve information related to the first target object, including an information address;
determine at least one of a position and an orientation of the first object with respect to a second object in the physical environment based on the visual appearance; and
enable an interaction with a computer-system based software application via the information address as a function of (i) the information, (ii) at least one of the position and the orientation, and (iii) the current location of the mobile device.

### Claim 58

A mobile device comprising:
an image sensor configured to capture image data of a physical environment that includes at least a first object; and
an image processing platform coupled with the image sensor and configured to:
obtain the image data from the image sensor;
recognize a first target object based at least in part on a visual appearance derived from the image data;
retrieve information related to the first target object, including an information address;
determine at least one of a position and an orientation of the first object with respect to a second object in the physical environment based on the visual appearance; and
enable an interaction with a computer-system based software application via the information address as a function of (i) the information, (ii) a time, and (iii) at least one of the position and the orientation.