# US 2024/0203131 A1 · Verifying object classification using component-based descriptors and temporal similarity measures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203131 A1
- Document ID: US20240203131A1
- Status: Published application
- Kind: A1
- Application number: 18/578,591
- Filed: 2022-07-08
- Published: 2024-06-20
- Earliest claimed: 2021-07-12
- Assignees: HRL Laboratories, LLC
- Applicants: HRL Laboratories, LLC
- Inventors: Amir RAHIMI; Hyukseong Kwon; Amit AGARWAL; Rajan BHATTACHARYYA
- Web page: https://patentgrep.com/patent/US20240203131A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06V 20/17: taken from planes or by drones
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus for verifying object classification includes comparing detected object-components with trained component-based descriptors and similarity measures to generate object classification verification data. The object classification verification data represents object classification confidence or misclassification errors to weight uncertainty for perception-based decision making. A sequence of similarity measures associated with a current frame and prior frames within a reference observation time constraint are compared to a reference temporal similarity measure boundary to generate the object classification verification data.

## Claims

### Claim 1

An autonomous vehicle comprising:
a sensor for providing perception data that captures images of a detected object during a sequence of frames;
a speed and steering control system; and
an autonomous vehicle control system including memory and an autonomous vehicle controller connected to the memory;
the memory including a reference object class that represents the detected object, the reference object class having reference component-descriptors, a reference observation time constraint, and reference temporal similarity measure boundary;
the autonomous vehicle controller including:
an object detector responsive to each captured image for identifying an object localization of the detected object and generating an object classification associated with the object localization at each frame;
a component-based similarity measure generator configured to generate a sequence of similarity measures associated with the sequence of frames, the component-based similarity measure generator being responsive to the object classification and the object localization at each frame for (i) generating object component-descriptors and (ii) comparing the object component-descriptors with the reference component-descriptors to generate each similarity measure in the sequence of similarity measures;
an object classification verifier that compares the sequence of similarity measures generated within the reference observation time constraint to the reference temporal similarity measure boundary for generating object classification verification data associated with the object classification;
an autonomous decision-making system responsive to the object classification verification data for generating a decision-making command,
wherein the speed and control system is responsive to the decision-making command to autonomously maneuver the autonomous vehicle.

### Claim 2 (depends on 1)

An autonomous vehicle according to claim 1, wherein the reference component-descriptors (i) define components of an object classification label associated with the generated object classification and (ii) include a reference embedding space mapping protocol for generating object component-descriptors to identify component characteristics from the object localization associated with the generated object classification.

### Claim 3 (depends on 1)

An autonomous vehicle according to claim 1, wherein:
the reference component-descriptors include (i) a histogram of reference component-descriptors that represent reference component cluster centroids in an embedded space associated with the reference object class and (ii) an embedding space mapping protocol associated with generating the histogram of reference component-descriptors; and
the object component-descriptors include a histogram of object component-descriptors that represent object component locations in the embedded space associated with the reference object class.

### Claim 4 (depends on 3)

An autonomous vehicle according to claim 3, wherein:
the histogram of reference component-descriptors is a histogram of N reference component-descriptors Y1 through YN that represent N reference component cluster centroids in the embedded space associated with the reference object class;
the histogram of object component-descriptors is a histogram of N object component-descriptors X1 through XN that represent N object component locations in the embedded space associated with the reference object class; and
the embedding space mapping protocol generates the histogram of N object component-descriptors X1 through XN from the object localization at each frame.

### Claim 5 (depends on 1)

An autonomous vehicle according to claim 1, wherein:
the memory includes a set of reference object classes, each reference object class in the set of reference object classes having an associated set of reference component-descriptors, a reference observation time constraint, and a reference temporal similarity measure boundary; and
the component-based similarity generator is responsive to an object classification label associated with the generated object classification for selecting a reference object class from the set of reference object classes.

### Claim 6 (depends on 1)

An autonomous vehicle according to claim 1, wherein the object detector generates a classification confidence value associated with the object classification, and transmits the object classification to the component-based similarity measure generator when the classification confidence value satisfies a classification confidence threshold.

### Claim 7 (depends on 2)

An autonomous vehicle according to claim 2, wherein the component-based similarity measure generator includes an object component-descriptor detector responsive to the object classification for selecting the reference embedding space mapping protocol to generate the histogram of object component-descriptors from the object localization at each frame.

### Claim 8 (depends on 2)

An autonomous vehicle according to claim 2, wherein the component-based similarity measure generator includes a component-descriptor comparator that generates the similarity measure as a cumulative difference between the object component-descriptors and the reference component-descriptors.

### Claim 9 (depends on 4)

An autonomous vehicle according to claim 4, wherein:
the component-based similarity measure generator includes a component-descriptor comparator that generates the similarity measure as a cumulative difference function between the object component-descriptors X1 through XN and the reference component-descriptors Y1 through YN, and the cumulative difference function is:
similarity
⁢
measure
=
∑
i
=
1
N
❘
"\[LeftBracketingBar]"
X
i
-
Y
i
❘
"\[RightBracketingBar]"

### Claim 10 (depends on 1)

An autonomous vehicle according to claim 1, wherein:
the sequence of frames has a current frame and prior frames;
the reference observation time constraint an observation start time tstart and an observation end time tend;
the object classification verification data represents a validation measurement for the object classification at the current frame, the validation measurement is a comparison of (i) the sequence of similarity measures associated with the detected object at the current frame and the prior frames within the observation start time tstart and the observation end time tend and (ii) the reference temporal similarity measure boundary associated with the reference object class.

### Claim 11 (depends on 1)

An autonomous vehicle according to claim 1, wherein the validation measurement for the object classification is a verified classification when the sequence of similarity measures associated with the detected object at the current frame and the prior frames during the reference observation time constraint is within the reference temporal similarity measure boundary associated with the reference object class.

### Claim 12 (depends on 10)

An autonomous vehicle according to claim 10, wherein:
the validation measurement for the object classification at the current frame is determined from combined similarity measures and probabilistic signal temporal logic constraints based on (i) the sequence of similarity measures during the current frame and the prior frames within reference observation time constraint; and (ii) the reference temporal similarity boundary associated with the reference object class;
the validation measurement represents a verified classification at the current frame when the sequence of similarity measures associated with the detected object during the reference observation time constraint is within the reference temporal similarity measure boundary; and
the validation measurement represents a misclassification at the current frame when the sequence of similarity measures associated with the detected object during the reference observation time constraint are not within the reference temporal similarity measure boundary.

### Claim 13 (depends on 12)

An autonomous vehicle according to claim 12, wherein the combined similarity measures with probabilistic signal temporal logic constraints are generated as follows:
∀
z
,
Pr
⁢
(
SM
⁢
(
z
,
t
start
:
t
end
)
≤
SM_
⁢
boundary
→
the
⁢
reference
⁢
object
⁢
class
)
where:
Pr (⋅) is a predicate;
SM(z, tstart, tend) is the observation z of the sequence of similarity measures SM during a sequence of frames including the current frame and the prior frames within the reference observation time constraint associated with the reference object class 1101;
SM_boundary represents performance characteristics from reference similarity measure sequences within the reference observation time constraint for the selected reference object class, where the performance characteristics reflect verified object detection characteristics for instances of similarity measure sequencies within time constraint associated with reference object class; and
the symbol “≤” refers to SM (z, tstart, tend) being within SM_boundary for determining the validation measurement associated with object classification at the current frame.

### Claim 14

A method for verifying object classification in a perception system, the method comprising the steps of:
storing a reference object class including reference component-descriptors, a reference observation time constraint, and a reference temporal similarity measure boundary;
receiving perception data from a sensor that captures images of a detected object during a sequence of frames;
identifying an object localization of the detected object in response to each captured image and generating an object classification associated with the object localization at each frame;
generating a sequence of similarity measures associated with the sequence of frames, wherein the step of generating the sequence of similarity measures is responsive to the object classification and the object localization for (i) generating object component-descriptors and (ii) comparing the object component-descriptors with the reference component-descriptors to generate each similarity measure in the sequence of similarity measures;
comparing the sequence of similarity measures generated within the reference observation time constraint to the reference temporal similarity measure boundary for generating object classification verification data associated with the object classification;
generating a decision-making command in response to the object classification verification data; and
controlling the perception system in response to the decision-making command.

### Claim 15 (depends on 14)

A method of verifying object classification in a perception system according to claim 14, wherein the reference component-descriptors (i) define components of an object classification label associated with the generated object classification and (ii) include a reference embedding space mapping protocol for generating object component-descriptors to identify component characteristics from the object localization associated with the generated object classification.

### Claim 16 (depends on 14)

A method of verifying object classification in a perception system according to claim 14, wherein:
the reference component-descriptors include (i) a histogram of reference component-descriptors that represent reference component cluster centroids in an embedded space associated with the reference object class and (ii) an embedding space mapping protocol associated with generating the histogram of reference component-descriptors; and
the object component-descriptors include a histogram of object component-descriptors that represent object component locations in the embedded space associated with the reference object class.

### Claim 17 (depends on 16)

A method of verifying object classification in a perception system according to claim 16, wherein:
the histogram of reference component-descriptors is a histogram of N reference component-descriptors Y1 through YN that represent N reference component cluster centroids in the embedded space associated with the reference object class;
the histogram of object component-descriptors is a histogram of N object component-descriptors X1 through XN that represent N object component locations in the embedded space associated with the reference object class; and
the embedding space mapping protocol generates the histogram of N object component-descriptors X1 through XN from the object localization at each frame.

### Claim 18 (depends on 14)

A method of verifying object classification in a perception system according to claim 14, wherein:
the memory includes a set of reference object classes, each reference object class in the set of reference object classes having an associated set of reference component-descriptors, a reference observation time constraint, and a reference temporal similarity measure boundary; and
the component-based similarity generator is responsive to an object classification label associated with the generated object classification for selecting a reference object class from the set of reference object classes.

### Claim 19 (depends on 14)

A method of verifying object classification in a perception system according to claim 14, wherein the method further comprising:
generating a classification confidence value associated with the object classification; and
generating the sequence of similarity measures when the classification confidence value satisfies a classification confidence threshold.

### Claim 20 (depends on 15)

A method of verifying object classification in a perception system according to claim 15, wherein the method further comprising:
selecting the reference embedding space mapping protocol in response to the object classification to generate the histogram of object component-descriptors from the object localization at each frame.

### Claim 21 (depends on 15)

A method of verifying object classification in a perception system according to claim 15, wherein the method further comprising:
generating the similarity measure as a cumulative difference between the object component-descriptors and the reference component-descriptors.

### Claim 22 (depends on 17)

A method of verifying object classification in a perception system according to claim 17, wherein the method further comprising:
generating the similarity measure as a cumulative difference function between the object component-descriptors X1 through XN and the reference component-descriptors Y1 through YN, and the cumulative difference function is:
similarity
⁢
measure
=
∑
i
=
1
N
❘
"\[LeftBracketingBar]"
X
i
-
Y
i
❘
"\[RightBracketingBar]"

### Claim 23 (depends on 14)

A method of verifying object classification in a perception system according to claim 14, wherein:
the sequence of frames has a current frame and prior frames;
the reference observation time constraint an observation start time tstart and an observation end time tend;
the object classification verification data represents a validation measurement for the object classification at the current frame, the validation measurement is a comparison of (i) the sequence of similarity measures associated with the detected object at the current frame and the prior frames within the observation start time tstart and the observation end time tend and (ii) the reference temporal similarity measure boundary associated with the reference object class.

### Claim 24 (depends on 14)

A method of verifying object classification in a perception system according to claim 14, wherein the validation measurement for the object classification is a verified classification when the sequence of similarity measures associated with the detected object at the current frame and the prior frames during the reference observation time constraint is within the reference temporal similarity measure boundary associated with the reference object class.

### Claim 25 (depends on 23)

A method of verifying object classification in a perception system according to claim 23, wherein:
the validation measurement for the object classification at the current frame is determined from combined similarity measures and probabilistic signal temporal logic constraints based on (i) the sequence of similarity measures during the current frame and the prior frames within reference observation time constraint; and (ii) the reference temporal similarity boundary associated with the reference object class;
the validation measurement represents a verified classification at the current frame when the sequence of similarity measures associated with the detected object during the reference observation time constraint is within the reference temporal similarity measure boundary; and
the validation measurement represents a misclassification at the current frame when the sequence of similarity measures associated with the detected object during the reference observation time constraint are not within the reference temporal similarity measure boundary.

### Claim 26 (depends on 25)

A method of verifying object classification in a perception system according to claim 25, wherein the combined similarity measures with probabilistic signal temporal logic constraints are generated as follows:
∀
z
,
Pr
⁢
(
SM
⁢
(
z
,
t
start
:
t
end
)
≤
SM_
⁢
boundary
→
the
⁢
reference
⁢
object
⁢
class
)
where:
Pr (⋅) is a predicate;
SM(z, tstart, tend) is the observation z of the sequence of similarity measures SM during a sequence of frames including the current frame and the prior frames within the reference observation time constraint associated with the reference object class 1101;
SM_boundary represents performance characteristics from reference similarity measure sequences within the reference observation time constraint for the selected reference object class where the performance characteristics reflect verified object detection characteristics for instances of similarity measure sequencies within time constraint associated with reference object class; and
the symbol “≤” refers to SM (z, tstart, tend) being within SM_boundary for determining the validation measurement associated with object classification at the current frame.

### Claim 27 (depends on 14)

The method of verifying object classification in a perception system according to claim 14, wherein the perception system is embedded in an autonomous vehicle that includes a (i) sensor and (ii) a speed and steering control system, and the step of controlling the perception system includes controlling the speed and control system in response to the decision-making command for autonomously maneuvering the autonomous vehicle.

### Claim 28 (depends on 14)

The method of verifying object classification in a perception system according to claim 14, wherein the perception system is embedded in an autonomous aviation security system that includes a surveillance system, and the step of controlling the perception system includes controlling the surveillance system in response to the decision-making command for autonomously controlling the aviation security system.

### Claim 29

A method of training component-based descriptors and temporal similarity measures to verify object classification in an object detector, the method comprising the steps of:
generating reference component-descriptors for a selected object class by performing the steps of: (i) selecting a first set of images having an object associated with the selected object class from a benchmark dataset; (ii) mapping component patches from detected objects in the first set of images to an embedding space with a selected number of N clusters having maximum inter-cluster variations and discriminative visual appearance; and (iii) generating a set of N reference component-descriptors in response to the N clusters;
determining a similarity measure threshold in response to the reference component-descriptors by performing the steps of: (i) selecting a second set of images having an object associated with the selected object class from the benchmark dataset;
(ii) mapping component patches from detected objects to an embedding space defining a set of N training component-descriptors for each detected object; (iii) generating similarity measures between each set of N training component-descriptors and the reference set of n component-descriptors; and (iv) determining a reference similarity measure threshold for identifying similarity measures associated with expected true positive detections and similarity measures associated with expected false positive detections; and
validating an observation time constraint for a reference similarity measure boundary by performing the steps of: (i) selecting a set of video sequences with images having an object associated with the selected object class from the benchmark dataset; (ii) selecting a time constraint having an observation start time tstart and an observation end time tend; (iii) mapping component patches from detected objects to an embedding space defining a set of N validation component-descriptors for each detected object during the observation start time tstart and the observation end time tend; (iv) determining a temporal similarity measure boundary based on performance characteristics from similarity measures sequences within the time constraint; and (v) comparing the sequences of generated similarity measures to ground truth data for the selected object class to determine accuracy of time constraint,
wherein:
if the accuracy of time constraint satisfies a validating threshold target, then store training parameters in memory for the selected object class; and
if the accuracy of time constraint does not satisfy the validating threshold target, then adjust the time constraint and repeat steps for validating the observation time constraint.

### Claim 30 (depends on 29)

The method of training component-based parameters according to claim 29, wherein the step of generating reference component-descriptors includes:
a. for each image in the first set of images, (i) detecting the object associated with the selected object class; (ii) extracting patches from the detected object; and (iii) mapping extracted patches to an embedding space with a clustering criterion and a selected number of N clusters that maximizes inter-cluster variations and discriminative visual appearance of reference component descriptor clusters in embedded space, the cluster criterion is defined by an embedding space protocol associated with reference component descriptor clusters; and
b. generating a reference set of N component-descriptors Y1 to YN in response to the N clusters having component cluster centroids for detected objects having the selected object class.

### Claim 31 (depends on 29)

The method of training component-based parameters according to claim 29, the step of determining a similarity measure threshold includes:
a. for each image in the second set of images, (i) detecting the object associated with the selected object class; (ii) extracting patches from the detected object; (iii) using the embedding space protocol to determine a set of N training component descriptors X1 to XN from the extracted patches; (iv) generating a similarity measure between the training component descriptors X1 to XN.n and the reference component descriptors Y1 to YN; and (v) compiling a distribution of similarity measures representing a likelihood of possible similarity measures for the detected object;
b. selecting a reference similarity measure threshold for a threshold boundary defining the similarity measures associated with expected true positive detections and the similarity measures associated with expected false positive detections.

### Claim 32 (depends on 29)

The method of training component-based parameters according to claim 29, the step of validating the time constraint includes:
a. selecting the time constraint;
b. for each video sequence during the time constraint, (i) detecting the object associated with the selected object class; (ii) extracting patches from the detected object; (iii) using the embedding mapping space protocol to determine another set of N validation component descriptors X1 to XN from the extracted patches; (iv) generating a similarity measure between the training component descriptors X1 to XN.n and the reference component descriptors Y1 to YN; and (v) compiling a distribution of similarity measures representing a likelihood of possible similarity measures for each detected object during the time constraint;
c. determining a temporal similarity measure boundary based on performance characteristics from similarity measures sequences within the time constraint; e; and
d. comparing the sequences of generated similarity measures to ground truth data for the selected object class to determine accuracy of the time constraint t_s to t_e,
wherein:
if the accuracy of time constraint satisfies a validating threshold target, then store training parameters in memory for the selected object class; and
if the accuracy of time constraint does not satisfy the validating threshold target, then adjust the time constraint and repeat steps for validating the observation time constraint
