# US 2013/0156348 A1 · Device, system, and method of visual inference by collaborative composition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0156348 A1
- Document ID: US20130156348A1
- Status: Published application
- Kind: A1
- Application number: 13/712,982
- Filed: 2012-12-13
- Published: 2013-06-20
- Earliest claimed: 2011-12-15
- Assignees: Yeda Research and Development Co. Ltd.
- Inventors: Michal IRANI; Alon FAKTOR
- Web page: https://patentgrep.com/patent/US20130156348A1

## CPC

- G06F 16/55: Clustering; Classification
- G06F 16/53: Querying
- G06F 16/5854: {using shape and object relationship}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/23: Clustering techniques
- G06V 10/464: {using a plurality of salient features, e.g. bag-of-words [BoW] representations}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Device, system, and method of visual inference by collaborative composition. A method of searching for similar target content-items for a plurality of source content-items, the method implementable on a computing device, comprises: for each of said plurality of source content-items, searching for similar target content-items in a dataset, while taking into account collaborative feedback exchanged among at least some of currently-searched said plurality of source content-items.

## Claims

### Claim 1

A method of searching for similar target content-items for a plurality of source content-items, the method implementable on a computing device, the method comprising:
for each of said plurality of source content-items, searching for similar target content-items in a dataset, while taking into account collaborative feedback exchanged among at least some of currently-searched said plurality of source content-items.

### Claim 2 (depends on 1)

The method of claim 1, wherein said plurality of source content-items are contained in said dataset of target content-items.

### Claim 3 (depends on 1)

The method of claim 1, wherein said searching comprises:
measuring similarity between each one of said source content-items and a small subset of target content-items in said dataset, wherein said small subset is unique to each one of said source content-items and is determined by collaborative feedback exchanged among said plurality of source content-items.

### Claim 4 (depends on 1)

The method of claim 1, wherein said searching comprises computing similarity-by-composition, wherein said computing similarity-by-composition comprises measuring similarity of detected shared regions among said source content-items and said target content-items.

### Claim 5 (depends on 4)

The method of claim 4, wherein detecting said shared regions comprises:
for source descriptors of a source content-item, randomly sampling target descriptors among target content-items; and
propagating good matches of source-target descriptors to neighboring descriptors in said source content-item and in one or more matching target content-items.

### Claim 6 (depends on 5)

The method of claim 5, wherein said randomly sampling comprises at least one of: uniform random sampling, and
randomly sampling by utilizing a modifiable sampling distribution.

### Claim 7 (depends on 6)

The method of claim 6, comprising:
dynamically modifying said modifiable sampling distribution between two consecutive search iterations, based on said collaborative feedback.

### Claim 8 (depends on 6)

The method of claim 6, wherein said modifiable sampling distribution is specific to said source content-item.

### Claim 9 (depends on 4)

The method of claim 4, wherein said detected shared regions are at least one of:
large shared regions, and
statistically significant regions.

### Claim 10 (depends on 4)

The method of claim 4, wherein said measuring similarity of detected shared regions comprises computing a likelihood ratio according to an equation:
Likelihood


Ratio


(
R
)
=
p

(
R
|
Ref
)
p

(
R
|
H
0
)
wherein R is a region in one of said source content-items,
wherein Ref is one of said target content-items which shares a similar region,
wherein p (R|Ref) measures a degree of similarity between said region R in said source content-item and said similar region in said target content-item,
wherein p (R|H0) measures a likelihood of said region R to occur at random.

### Claim 11 (depends on 10)

The method of claim 10, wherein said degree of similarity is a function of at least one of the following: sum-of-differences, sum-of-squared differences, sum-of-absolute-differences, an Lp norm measure, mutual information, correlation, normalized correlation, a robust measure, a histogram comparison measure, a motion consistency measure, and a probability measure.

### Claim 12 (depends on 10)

The method of claim 10, wherein p(R|X)=Πip(di|X),
wherein said di are descriptors composing said region R.

### Claim 13 (depends on 12)

The method of claim 12, wherein said descriptors are at least one of:
an image descriptor, a Scale-Invariant Feature Transform (SIFT) descriptor, a Speeded Up Robust Features (SURF) descriptor, a Histogram Oriented Gradients (HOG) descriptor, a shape-context descriptor, a self-similarity descriptor, a local-binary pattern descriptor, a color descriptor, a greyscale descriptor, a normalized color descriptor, a normalized greyscale descriptor, a gradient based descriptor, a normalized gradient based descriptor, a video descriptor, a HOG-3D descriptor, a Histogram of Optical Flow (HOF) descriptor, a a flow field based descriptor, a normal flow based descriptor, a motion based descriptor, a temporal derivative based descriptor, a normalized temporal derivative based descriptor, an audio descriptor, a spectogram based descriptor, a Cepstrum based descriptors, a Mel-Frequency Cepstral Coefficient (MFCC), a Short Time Fourier Transform (STFT) descriptor, a Constant Q-Transform (CQT) descriptor, a descriptor in the spectral domain of an audio signal.

### Claim 14 (depends on 10)

The method of claim 10, wherein said p(R|H0) is estimated by utilizing an error of descriptors composing said region R with respect to a quantized codebook of descriptors.

### Claim 15 (depends on 14)

The method of claim 14, wherein said error of at least one of said descriptors is an error computed with respect to at least one nearest neighbor of said at least one of said descriptors in said quantized codebook.

### Claim 16 (depends on 14)

The method of claim 14, comprising:
generating said quantized codebook from a collection of descriptors by at least one of:
k-means clustering of said collection of descriptors,
hierarchal clustering of said collection of descriptors,
random sampling from said collection of descriptors.

### Claim 17 (depends on 1)

The method of claim 1, wherein said content-items comprise at least one of: images, video sequences, and audio items.

### Claim 18 (depends on 3)

The method of claim 3, wherein said small subset comprises also a randomly sampled small subset generated by randomly sampling based on a sampling distribution, wherein said sampling distribution comprises at least one of:
a uniform distribution,
a non-uniform distribution, and
a dynamically modifiable sampling distribution based on said collaborative feedback.

### Claim 19 (depends on 18)

The method of claim 18, comprising:
updating said modifiable sampling distribution, for a particular source content-item, by utilizing a function that comprises at least one of the following:
a first sampling distribution component corresponding to a subset in which said particular source content-item found other similar content items;
a second sampling distribution component corresponding to a subset in which another, similar, content-item found similar content-items; and
a third sampling distribution component corresponding to uniform sampling of said dataset.

### Claim 20 (depends on 3)

The method of claim 3, wherein said small subset comprises one or more target content-items that are suggested for searching based on collaborative feedback from one or more source content-items.

### Claim 21 (depends on 1)

The method of claim 1, wherein the collaborative feedback comprises suggestions obtained from source content-items indicating to each other where to sample in said dataset in a subsequent search iteration.

### Claim 22 (depends on 1)

The method of claim 1, wherein the collaborative feedback comprises at least a sparse set of meaningful affinities among source content-items.

### Claim 23 (depends on 1)

The method of claim 1, comprising:
based on affinity output of the searching, performing at least one of: content-item clustering, content-item classification, content-item retrieval, detecting informative content-items, localization of information within content-items, label transfer, annotation, and generating links between content-items.

### Claim 24

A computing device for searching for similar target content-items for a plurality of source content-items, the computing device comprising:
a searcher module to search, for each of said plurality of source content-items, for similar target content-items in a dataset, while taking into account collaborative feedback exchanged among at least some of currently-searched said plurality of source content-items.

### Claim 25 (depends on 24)

The computing device of claim 24, wherein said plurality of source content-items are contained in said dataset of target content-items.

### Claim 26 (depends on 24)

The computing device of claim 24, comprising:
an affinity calculator to measure similarity between each one of said source content-items and a small subset of target content-items in said dataset, wherein said small subset is unique to each one of said source content-items and is determined by collaborative feedback exchanged among said plurality of source content-items.

### Claim 27 (depends on 24)

The computing device of claim 24, wherein said affinity calculator computes similarity-by-composition by measuring similarity of detected shared regions among said source content-items and said target content-items.

### Claim 28 (depends on 27)

The computing device of claim 27, comprising:
a sampling module to randomly sample, for source descriptors of a source content-item, target descriptors among target content-items; and
a propagation module to propagate good matches of source-target descriptors to neighboring descriptors in said source content-item and in one or more matching target content-items.