Method and apparatus for classifying and querying temporal and spatial information in video

Abstract

A method and apparatus for classifying or annotating image and video content by assigning each class of event, action, region or object a unique symbol and then building symbol strings to represent sequences in space or time. The symbol strings can be decoded using a library of annotated PT descriptors to automatically label the image and video content. Furthermore, the PT descriptors can be used for searching by sketch or searching by example where the search query input is converted to symbol strings, which are efficiently compared based on the presence and relative counts of PT descriptors mapped into a PT matrices. The Precedence Template (PT) descriptor can be used for classifying and querying video based on the spatial and temporal orderings of regions, objects, actions or events. Applied to video, the PT descriptors provide a way to compare the temporal order of events, actions, or objects such as those represented in a scene transition graph, key-frame list, or event string. Applied to images, the PT descriptors provide a way to compare the spatial arrangement of image regions or objects. By capturing the spatial and temporal relationships statistically, the PT descriptors provide a robust way to measure similarity in the presence of insertions, deletions, substitutions, replications and relocations of events, actions, regions or objects.

Dependencies are inferred from the claim text.