# US 2012/0084276 A1 · Providing associations between objects and individuals associated with relevant media items

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0084276 A1
- Document ID: US20120084276A1
- Status: Published application
- Kind: A1
- Application number: 12/895,673
- Filed: 2010-09-30
- Published: 2012-04-05
- Earliest claimed: 2010-09-30
- Assignees: Microsoft Corporation
- Inventors: Scott Martin Heimendinger
- Web page: https://patentgrep.com/patent/US20120084276A1

## CPC

- G06F 16/434: {using image data, e.g. images, photos, pictures taken by a user}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments provide identification and/or reporting features as part of managing aspects of a product and/or service, but are not so limited. In an embodiment, a computer-implemented method uses one or more search references to identify media items that correspond with the one or more search references and one or more individuals associated with each media item. The method of one embodiment provides a searching and identification service that uses one or more images as one or more search key inputs to identify relevant images of a media collection, including associating information of any depicted individuals with search key inputs. In one embodiment, a computing system operates to provide an association, reporting, and/or display service as part of managing an asset by using one or more search references to search through an image collection as part of determining how a product or service is being used by characterizing an association between each individual associated with the image and the one or more search references. Other embodiments are included and available.

## Claims

### Claim 1

A method comprising:
using a search key as a searching reference as part of a search operation;
searching a repository of information that includes one or more data representations including video and image data and metadata using the search key to identify one or more relevant media items, each relevant media item including a portion corresponding to the search key;
capturing information associated with each relevant media item including data and metadata associated with one or more individuals associated with each relevant media item;
associating each relevant media item and captured information with the search key; and,
storing each association in storage.

### Claim 2 (depends on 1)

The method of claim 1, further comprising associating multiple search keys including image, graphical, and textual references with one or more of a relevant video, image, and a portion of a data stream.

### Claim 3 (depends on 1)

The method of claim 1, further comprising identifying the one or more individuals associated with each relevant media item depicting all or some portion of an object associated with the search key and bounding each individual and object for further analysis.

### Claim 4 (depends on 3)

The method of claim 3, further comprising collecting profile information associated with the one or more individuals, including tagged and identified individuals for relevance evaluations.

### Claim 5 (depends on 1)

The method of claim 1, further comprising identifying and storing information of each relevant media item including profile information of one or more of an authoring user, an uploading user, a recognized individual, and a tagged individual using image matching and other processing operations.

### Claim 6 (depends on 5)

The method of claim 5, further comprising identifying and storing information of each relevant media item including profile information of one or more tagged individuals identified as appearing in the relevant media item, an individual appearing in a different media item relative to the relevant media item, and an individual at a same geographic location at the time the media item was recorded.

### Claim 7 (depends on 1)

The method of claim 1, further comprising identifying importance of a user association with a product or service based in part on a boundary point evaluation between one or more bounded individuals and a bounded object associated with the search key captured in the relevant media.

### Claim 8 (depends on 7)

The method of claim 7, further comprising identifying importance of a user association with a product brand based in part on a physical proximity defined in part by distance between a first boundary point associated with a tagged individual and a second boundary point associated with a bounded object.

### Claim 9 (depends on 1)

The method of claim 1, further comprising associating a matched image with the search key and storing the captured information in one or more of local and remote storage, including defining mappings between the captured information and the matched image.

### Claim 10 (depends on 1)

The method of claim 1, further comprising using the search key that includes one or more reference images comprising a brand identifier, name, trade name, logo, slogan, and a design scheme, the search operation providing information that can be used in part as part of a good-will assessment in conjunction with association of the search key and an associated user.

### Claim 11 (depends on 1)

The method of claim 1, further comprising using a database to store information when reporting information of relevant media items that correspond with a search image, a search sound, and search phrase, the database including records that include one of a media file and a reference to the media file, and retrievable or referenced user profile information of found or tagged users.

### Claim 12 (depends on 1)

The method of claim 1, further comprising using a search key that includes one or more of a search image and a search phrase, wherein the search phrase is provided using an optical character recognition (OCR) process associated with an input search image.

### Claim 13

A system comprising:
a storage medium that includes a media collection;
processing resources; and
a computer-implemented search component that uses a number of search references to:
locate relevant media items included with the media collection using one or more character recognition and pattern recognition algorithms;
identify relevant portions of each relevant media item based in part on each search reference;
determine profile information for each individual associated with each relevant media item;
determine relevance associations between each individual and an object associated with each search reference for each relevant media item; and
store the relevance associations for each relevant media item.

### Claim 14 (depends on 13)

The system of claim 13, wherein the search component uses the number of search references to associate multiple search image references, including graphical and textual representations, included in an image or image set.

### Claim 15 (depends on 13)

The system of claim 13, wherein the search component collects one or more of demographic, geographic, and temporal information associated with each relevant media item.

### Claim 16 (depends on 13)

The system of claim 13, wherein the search component uses OCR and image matching features to identify aspects of a search key contained in a relevant image including product names, brands, logos, and emblems as part of identifying a particular asset.

### Claim 17

A computer readable medium including encoded instructions used in part to provide searching services comprising:
using a search key that includes all or some portion of a reference image to identify relevant data representations that include one or more associated individuals and an object associated with the search key;
retrieving profile information of each associated individual of each relevant data representation identified using the search key;
determining an association between each associated individual and the search key using bounded identifying portions with each relevant data representation; and
storing each association for each relevant data representation.

### Claim 18 (depends on 17)

The computer readable storage medium of claim 17 including encoded instructions to identify physical relationships between tagged individuals and an object associated with the search key.

### Claim 19 (depends on 17)

The computer readable storage medium of claim 17 including encoded instructions to identify behavioral aspects of depicted individuals and an object associated with the search key.

### Claim 20 (depends on 17)

The computer readable storage medium of claim 17 including encoded instructions to delineate identification boundaries for each identified individual and an object associated with the search key, and store one of demographic, geographic, and temporal information of a relevant image of a hosted image collection.
