# US 2012/0251011 A1 · Event Determination From Photos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0251011 A1
- Document ID: US20120251011A1
- Status: Published application
- Kind: A1
- Application number: 13/079,592
- Filed: 2011-04-04
- Published: 2012-10-04
- Earliest claimed: 2011-04-04
- Assignees: Microsoft Corporation
- Inventors: Mingyan Gao; Xian-Sheng Hua
- Web page: https://patentgrep.com/patent/US20120251011A1

## CPC

- G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06V 20/20: in augmented reality scenes
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Events may be determined based on an image and context data associated with the image. An event type associated with the image may be determined based on a concept of the image. A list of events may be retrieved from an event database based on the context data. The retrieved list of events may then be ranked based on the determined event type and the context data. Through this event determination, a user may obtain information of one or more events happening at a specific location simply by capturing an image of that specific location, thereby saving the user from searching and browsing the Internet or brochure to locate the information of the one or more events at the specific location.

## Claims

### Claim 1

A computer-implemented method comprising:
under control of one or more processors configured with executable instructions:
receiving an image;
extracting image features from the image;
determining a concept associated with image based at least on the extracted image features, the concept including an action, an object and/or a background scene shown in the image; and
determining an event type of an event shown in the image based at least on the determined concept.

### Claim 2 (depends on 1)

The computer-implemented method as recited in claim 1, further comprising:
receiving context data associated with the image, the context data including spatial data and/or temporal data associated with the image; and
retrieving a first list of events from an event database based at least on the context data.

### Claim 3 (depends on 2)

The computer-implemented method as recited in claim 2, further comprising obtaining a second list of events from the first list of events based on the determined event type.

### Claim 4 (depends on 1)

The computer-implemented method as recited in claim 1, wherein the image comprises a scene captured using the client device, a scene captured by an application of the client device, a scene captured by the client device being pointed at the scene without snapping an image of the scene.

### Claim 5

A computer-implemented method comprising:
under control of one or more processors configured with executable instructions:
receiving an image;
obtaining context data associated with the image;
comparing the context data to an event database to retrieve a first list of events;
analyzing the image to determine an event type of an event shown in the image; and
comparing event types of the events in the first list with the determined event type to obtain a second list of events.

### Claim 6 (depends on 5)

The computer-implemented method as recited in claim 5, wherein obtaining context data associated with the image includes receiving the context data along with the image in a query, or generating the context data based at least on content of the image.

### Claim 7 (depends on 5)

The computer-implemented method as recited in claim 5, wherein the context data includes spatial data and/or temporal data associated with an event shown in the image.

### Claim 8 (depends on 7)

The computer-implemented method as recited in claim 7, wherein the spatial data associated with the event shown in the image includes a location at which the image was captured, and the temporal data associated with the event shown in the image includes a time at which the image was captured.

### Claim 9 (depends on 7)

The computer-implemented method as recited in claim 7, further comprising ranking a first event in the second list at a position higher than a second event in the second list in response to determining that the location associated with the event shown in the image is closer to a location associated with the first event than a location associated with the second event.

### Claim 10 (depends on 7)

The computer-implemented method as recited in claim 7, further comprising ranking a first event in the second list at a position higher than a second event in the second list in response to determining that the time associated with the event shown in the image is closer to a time associated with the first event than a time associated with the second event.

### Claim 11 (depends on 5)

The computer-implemented method as recited in claim 5, further comprising ranking a first event in the second list at a position higher than a second event in the second list in response to determining that an event type of the first event matches the determined event type and an event type of the second event does not matches the determined event type.

### Claim 12 (depends on 5)

The computer-implemented method as recited in claim 5, further comprising returning the second list of events to a client from which the image is received.

### Claim 13 (depends on 5)

The computer-implemented method as recited in claim 5, wherein the event database includes planned events and emergent events, the planned events including events that are pre-scheduled to occur and the emergent events including events which occurrences are impromptu.

### Claim 14 (depends on 5)

The computer-implemented method as recited in claim 5, further comprising:
determining a point of focus in the image; and
estimating a location of the point of focus based on the context data and the image.

### Claim 15 (depends on 14)

The computer-implemented method as recited in claim 14, wherein comparing the context data to the event database includes comparing the location of the point of focus to the event database to retrieve the first list of events, the events in the first list being occurred within a predetermined distance from the location of the point of focus.

### Claim 16

One or more computer-readable media configured with computer-executable instructions that, when executed by one or more processors, configure the one or more processors to perform acts comprising:
receiving an image and context data associated with the image, the context data including spatial information and temporal information associated with the image;
analyzing content of the image to detect an event type shown in the image;
retrieving a first list of events from an event database based on the spatial information and the temporal information associated with the image; and
ranking the first list of events to obtain a ranked list of events based on the detected event type of the image, the spatial information and the temporal information.

### Claim 17 (depends on 16)

The one or more computer-readable media as recited in claim 16, wherein the spatial information associated with the image includes a location at which the image was captured, and the temporal information associated with the image includes a date and/or time at which the image was captured.

### Claim 18 (depends on 17)

The one or more computer-readable media as recited in claim 17, wherein retrieving the first list of events from an event database includes retrieving one or more events that occur within a predetermined distance from the location at which the image is captured and within a predetermined time range before and after the time at which the image is captured.

### Claim 19 (depends on 16)

The one or more computer-readable media as recited in claim 16, wherein the acts further comprise:
determining a point of focus in the image; and
estimating a location of the point of focus based on the context data and the image.

### Claim 20 (depends on 19)

The one or more computer-readable media as recited in claim 19, wherein retrieving the first list of events from an event database is further based on comparing the location of the point of focus to the event database to retrieve the first list of events, the events in the first list being occurred within a predetermined distance from the location of the point of focus.
