Confidence weighted classifier combination for multi-modal identification
AssigneeHonda Motor Co., Ltd.
InventorsYuri Ivanov, Thomas R. Serre
Life of this patent
Estimating the term…
Abstract
Techniques are disclosed for multi-modal identification that utilize a classifier combination framework. One embodiment of the present invention provides a multi-modal identification system that includes a collection of classifiers that classify feature streams derived from audio and/or video sources. A classifier combination scheme is used to combine the classifier outputs having varying degrees of confidence, but in a robust way by using a confidence-based weighting scheme that operates on a “per-class” basis, rather than (or in addition to) the traditional “per-classifier” basis. The system can be distributed across several machines running independent feature classifiers on the subscription basis.
Claims18
A multi-class classifier combination system comprising:
- at least one processor;
- a plurality of multi-class classifiers, each classifier executing on one of the at least one processor, each classifier for classifying a target feature captured in at least one feature stream;
- and
- a combination module for combining classifier outputs into a joint prediction, wherein each multi-class classifier prediction is weighted in accordance with a per-class weighting scheme prior to combining, wherein the per-class weighting scheme carried out by the combination module is in accordance with
- P
- s
- ≈
- ∑
- λ
-
- w
- λ
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PS is the joint prediction, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, wλ is a per-classifier weight, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
A method for multi-class classifier combination using predictions of a plurality of multi-class classifiers, the method comprising using a processor to perform the steps of:
- weighting each multi-class classifier prediction in accordance with a per-class weighting scheme;
- and
- combining the weighted predictions from two or more multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
- P
- s
- ≈
- ∑
- λ
-
- w
- λ
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PS is the joint prediction, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, wλ is a per-classifier weight, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
A multi-class classifier combination system comprising:
- a means for classifying a first target feature captured in at least one feature stream using a first multi-class classifier;
- a means for classifying a second target feature captured in at least one feature stream using a second multi-class classifier;
- a means for weighting each multi-class classifier prediction in accordance with a per-class weighting scheme;
- a means for combining the weighted predictions from the first and second multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
- P
- s
- ≈
- ∑
- λ
-
- w
- λ
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PS is the joint prediction, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, w80 is a per-classifier weight, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier;
- and
- a processor for executing the means for combining the weighted predictions from the first and second multi-class classifiers into a joint prediction.
A computer-readable medium encoded with instructions, that when executed by a processor, cause the processor to carry out a multi-class classifier combination process using predictions of a plurality of multi-class classifiers, the process comprising:
- weighting each multi-class classifier prediction in accordance with a per-class weighting scheme;
- and
- combining the weighted predictions from two or more multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
- P
- s
- ≈
- ∑
- λ
-
- w
- λ
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PS is the joint prediction, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, w80 is a per-classifier weight, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
A multi-class classifier combination system comprising:
- at least one processor;
- a plurality of multi-class classifiers, each classifier executing on one of the at least one processor, each classifier for classifying a target feature captured in at least one feature stream;
- and
- a combination module for combining classifier outputs into a joint prediction, wherein each multi-class classifier prediction is weighted in accordance with a per-class weighting scheme prior to combining, wherein the per-class weighting scheme carried out by the combination module is in accordance with
- P
- P
-
- (
- ω
- ❘
- x
- )
- =
- 1
- Z
-
- ∏
- λ
-
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PP is the joint prediction using a product combination rule, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
A method for multi-class classifier combination using predictions of a plurality of multi-class classifiers, the method comprising using a processor to perform the steps of:
- weighting each multi-class classifier prediction in accordance with a per-class weighting scheme;
- and
- combining the weighted predictions from two or more multi-class classifiers into a joint prediction, wherein the per-class weighting scheme is in accordance with
- P
- P
-
- (
- ω
- ❘
- x
- )
- =
- 1
- Z
-
- ∏
- λ
-
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PP is the joint prediction using a product combination rule, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
A computer-readable medium encoded with instructions, that when executed by a processor, cause the processor to carry out a multi-class classifier combination process using predictions of a plurality of multi-class classifiers, the process comprising:
- weighting each multi-class classifier prediction in accordance with a per-class weighting scheme;
- and
- combining the weighted predictions from two or more multi-class classifiers into a joint prediction wherein the per-class weighting scheme is in accordance with
- P
- P
-
- (
- ω
- ❘
- x
- )
- =
- 1
- Z
-
- ∏
- λ
-
- [
- ∑
- ω
- ~
-
- P
- λ
-
- (
- ω
- ❘
- ω
- ~
- )
-
- P
- λ
-
- (
- ω
- ~
- ❘
- x
- )
- ]
- ,
- where PP is the joint prediction using a product combination rule, x is a set of features in a given scene, Pλ({tilde over (ω)}|x) is the prediction of the individual classifier, Z is a normalizing constant, and confidence measure, Pλ(ω|{tilde over (ω)},x), is approximated by its projection, Pλ(ω|{tilde over (ω)}), wherein ω is a true class label and {tilde over (ω)} is an output of an individual multi-class classifier.
Dependencies are inferred from the claim text.