# US 9,436,890 B2 · Method of generating feature vector, generating histogram, and learning classifier for recognition of behavior

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,436,890 B2
- Document ID: US9436890B2
- Status: Granted
- Kind: B2
- Application number: 14/562,948
- Filed: 2014-12-08
- Published: 2015-07-23
- Granted: 2016-09-06
- Earliest claimed: 2014-01-23
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Won Jun Kim; Byung In Yoo; Jae Joon Han
- Web page: https://patentgrep.com/patent/US9436890B2

## CPC

- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Example embodiments disclose a method of generating a feature vector, a method of generating a histogram, a learning unit classifier, a recognition apparatus, and a detection apparatus, in which a feature point is detected from an input image based on a dominant direction analysis of a gradient distribution, and a feature vector corresponding to the detected feature point is generated.

## Claims

### Claim 1

A method of generating a feature vector, the method comprising:
detecting a feature point from an input image based on a dominant direction of a gradient distribution in the input image, the detecting including,
detecting a pixel corresponding to a window having a contrast of coherence in a dominant direction; and
generating a feature vector corresponding to the feature point.

### Claim 2 (depends on 1)

The method of claim 1, wherein the
contrast of coherence is an indication of differences between at least one eigenvalue associated with the detected pixel and eigenvalues of other pixels in the window.

### Claim 3 (depends on 1)

The method of claim 1, wherein the generating of the feature vector comprises:
accumulating a strength of coherence in a dominant direction within a local area corresponding to the feature point to generate the feature vector.

### Claim 4 (depends on 1)

The method of claim 1, wherein the detecting of the feature point comprises:
calculating a gradient for a plurality of pixels comprised in the input image;
calculating a structure tensor for the plurality of pixels based on the gradient;
calculating a maximum eigenvalue for the plurality of pixels by performing an Eigen analysis on the structure tensor; and
determining the feature point through a contrast amongst maximum eigenvalues.

### Claim 5 (depends on 4)

The method of claim 4, wherein the calculating of the structure tensor for the plurality of pixels comprises:
when the input image is a video image, calculating a structure tensor of a single pixel based on a matrix
[
∑
B
⁢
G
x
2
∑
B
⁢
G
x
⁢
G
y
∑
B
⁢
G
x
⁢
G
t
∑
B
⁢
G
x
⁢
G
y
∑
B
⁢
G
y
2
∑
B
⁢
G
y
⁢
G
t
∑
B
⁢
G
x
⁢
G
t
∑
B
⁢
G
y
⁢
G
t
∑
B
⁢
G
t
2
]
,
wherein Gx denotes a gradient in an x axis direction, Gy denotes a gradient in a y axis direction, Gt denotes a gradient in a time axis direction, and B denotes a predetermined size of a block comprising the single pixel.

### Claim 6 (depends on 4)

The method of claim 4, wherein the calculating of the structure tensor for the plurality of pixels comprises:
when the input image is a still image, calculating a structure tensor of a single pixel based on a matrix
[
∑
B
⁢
G
x
2
∑
B
⁢
G
x
⁢
G
y
∑
B
⁢
G
x
⁢
G
y
∑
B
⁢
G
y
2
]
,
wherein Gx denotes a gradient in an x axis direction, Gy denotes gradient in a y axis direction, and B denotes a predetermined size of a block comprising the single pixel.

### Claim 7 (depends on 4)

The method of claim 4, wherein the determining of the feature point comprises:
calculating a sum of differences between a strength of coherence of a single pixel and strengths of coherence of neighboring pixels within a window corresponding to the single pixel; and
determining the single pixel to be the feature point when the sum is greater than a threshold value.

### Claim 8 (depends on 7)

The method of claim 7, wherein the strength of coherence comprises at least one of:
a maximum eigenvalue of the single pixel; and
a difference between the maximum eigenvalue and another eigenvalue of the single pixel.

### Claim 9 (depends on 1)

The method of claim 1, wherein the generating of the feature vector comprises:
with respect to a local pixels comprised in a local region,
mapping the dominant direction of a local pixel to predetermined bins; and
accumulating a dominant direction energy of the local pixel in the mapped bin.

### Claim 10 (depends on 9)

The method of claim 9, wherein the dominant direction energy is a strength of coherence associated with a gradient of the local pixel, and the dominant direction corresponds to a maximum eigenvalue associated with the gradient of the local pixel.

### Claim 11 (depends on 9)

The method of claim 9, wherein the predetermined bins quantize a time and a space of the input image to a predetermined number when the input image is a video image.

### Claim 12 (depends on 9)

The method of claim 9, wherein the predetermined bins quantize the space of the input image to a predetermined number when the input image is a still image.

### Claim 13 (depends on 1)

The method of claim 1, wherein the gradient comprises:
when the input image is a video image, a gradient in an x axis direction in a frame comprised in the input image, a gradient in a y axis direction in the frame comprised in the input image, and a gradient in a time axis direction between frames comprised in the input image.

### Claim 14 (depends on 1)

The method of claim 1, wherein the gradient comprises:
when the input image is a still image, a gradient in an x axis direction in a frame comprised in the input image and a gradient in a y axis direction in the frame comprised in the input image.

### Claim 15 (depends on 1)

The method of claim 1, further comprising:
generating of the feature vector corresponding to the input image based on a frequency characteristic extracted from the input image.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
dividing the input image into combinations of a plurality of global regions using a spatial pyramid; and
extracting frequency characteristics of the plurality of global regions comprised in the combinations of the plurality of global regions, and generating feature vectors corresponding to the plurality of global regions.

### Claim 17 (depends on 1)

A non-transitory computer-readable medium comprising a program for instructing a computer to perform the method of claim 1.

### Claim 18

A method of image processing, the method comprising:
detecting a plurality of feature points comprised in an input image based on a dominant direction of a gradient distribution in the input image;
generating a plurality of feature vectors corresponding to the plurality of feature points, the generating including
accumulating a strength of coherence in the dominant direction within a local region corresponding to the plurality of feature points in order to generate the plurality of feature vectors; and
mapping the plurality of feature vectors to codewords in a codebook, to generate a histogram corresponding to the input image based on the mapped feature vectors.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
analyzing the input image based on a learned parameter and the histogram.

### Claim 20 (depends on 19)

The method of claim 19, wherein the learned parameter is generated by pre-learning a plurality of training images based on the dominant direction analysis of the gradient distribution.

### Claim 21 (depends on 19)

The method of claim 19, wherein the analyzing of the input image comprises:
recognizing content of the input image.

### Claim 22 (depends on 21)

The method of claim 21, wherein the content of the input image comprises at least one of:
a behavior of a performer comprised in the input image; and
an object comprised in the input image.

### Claim 23 (depends on 18)

The method of claim 18, wherein the generating of the histogram comprises:
normalizing values corresponding to the codewords, and
generating a normalized histogram.

### Claim 24 (depends on 18)

The method of claim 18, further comprising:
detecting a set of feature points from a plurality of training images based on the dominant direction analysis;
generating feature vectors corresponding to a portion of feature points selected arbitrarily from among the set of feature points;
clustering the feature vectors corresponding to the portion of feature points; and
generating the codebook.

### Claim 25 (depends on 18)

The method of claim 18, wherein the detecting of the plurality of feature points comprises:
detecting pixels corresponding to windows having a contrast of coherence in a dominant direction to be the plurality of feature points.

### Claim 26 (depends on 18)

The method of claim 18, wherein the
strength of coherence is based on a difference between eigenvalues associated with the feature point.

### Claim 27 (depends on 18)

The method of claim 18, further comprising:
generating a global feature vector corresponding to the input image based on a frequency characteristic extracted from the input image; and
combining the histogram and the global feature vector.

### Claim 28 (depends on 18)

The method of claim 18, further comprising:
unsupervised learning of the histogram; and
generating a mid-level feature vector.

### Claim 29

An apparatus for learning a classifier, the apparatus comprising:
a detector configured to detect a plurality of feature points comprised in a plurality of training images based on a dominant direction analysis of a gradient distribution, wherein the detector is configured to detect pixels corresponding to windows having a contrast of coherence in a dominant direction to be the plurality of feature points;
a generator configured to generate a plurality of feature vectors corresponding to the plurality of feature points;
a mapper configured to map the plurality of feature vectors to codewords comprised in a given codebook, and generate a histogram corresponding to the plurality of training images; and
a learning unit configured to learn a classifier based on the histogram.

### Claim 30 (depends on 29)

The apparatus of claim 29, wherein the learning unit is configured to receive the histogram and a label of the plurality of training images to learn the classifier.

### Claim 31 (depends on 29)

The apparatus of claim 29, further comprising:
a clustering unit configured to cluster the plurality of feature vectors, and generate the codebook.

### Claim 32 (depends on 29)

The apparatus of claim 29, wherein the contrast of coherence is an indication of a difference between at least one eigenvalue associated with the detected pixel and eigenvalues of other pixels in the window.

### Claim 33 (depends on 29)

The apparatus of claim 29, wherein the generator is configured to accumulate a strength of coherence.

### Claim 34 (depends on 29)

The apparatus of claim 29, wherein the mapper is configured to map a set of the plurality of feature vectors generated from a corresponding training image from among the plurality of training images, and generates a histogram corresponding to the corresponding training image.

### Claim 35 (depends on 29)

The apparatus of claim 29, wherein the mapper is configured to
normalize values corresponding to the codewords, and
generate a normalized histogram.

### Claim 36

A recognition apparatus, the apparatus comprising:
a detector configured to detect a plurality of feature points in an input image based on a dominant direction analysis of a gradient distribution;
a generator configured to generate a plurality of feature vectors corresponding to the plurality of feature points, wherein the generator is configured to accumulate a strength of coherence in a dominant direction of a local region corresponding to the plurality of feature points in order to generate the plurality of feature vectors;
a mapper configured to map the plurality of feature vectors in a given codebook, and generate a histogram corresponding to the input image; and
a recognizer configured to recognize content of the input image based on a learned parameter and the histogram.

### Claim 37 (depends on 36)

The apparatus of claim 36, wherein the content of the input image comprises
at least one of a behavior of a performer comprised in the input image; and
an object comprised in the input image.

### Claim 38 (depends on 36)

The apparatus of claim 36, wherein the learned parameter is generated by pre-learning a plurality of training images based on the dominant direction analysis of the gradient distribution.

### Claim 39 (depends on 36)

The apparatus of claim 36, wherein the detector is configured to detect pixels corresponding to windows having a contrast of coherence in a dominant direction to be the plurality of feature points.

### Claim 40 (depends on 36)

The apparatus of claim 36, wherein the strength of coherence is based on a difference between eigenvalues associated with the feature point.

### Claim 41

A detection apparatus, the apparatus comprising:
a feature point detector configured to detect a plurality of feature points comprised in at least a portion of an input image based on a dominant direction analysis of a gradient distribution, wherein the feature point detector is configured to detect pixels corresponding to window having a contrast of coherence in a dominant direction to be the plurality of feature points;
a generator configured to generate a plurality of feature vectors corresponding to the plurality of feature points;
a mapper configured to map the plurality of feature vectors to codewords comprised in a given codebook, and generate a histogram corresponding to the at least one portion; and
a region detector configured to compare a reference histogram to the histogram, and detect a region corresponding to the reference histogram.

### Claim 42 (depends on 41)

The apparatus of claim 41, wherein the feature point detector is configured to detect a plurality of reference feature points comprised in a reference image based on the dominant direction analysis of the gradient distribution,
the generator is configured to generate a plurality of reference feature vectors corresponding to the plurality of reference feature points, and
the mapper is configured to map the plurality of reference feature vectors to the codewords, and generates the reference histogram corresponding to the reference image.