US 12,592,065 B2Granted

Systems and methods for object detection in extreme low-light conditions

AssigneePurdue Research Foundation

InventorsStanley H. Chan, Chengxi Li, Abhiram Gnanasambandam, Xiangyu Qu

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods for detecting objects in photon-limited environments is disclosed for use in, for example, security, defense, life science, autonomous vehicles, and various consumer and medical applications. At least one embodiment integrates a non-local feature aggregation method and a knowledge distillation method with state-of-the-art detector networks. The two methods offer better feature representations for photon limited images. In comparison with baseline systems, detectors according to embodiments of the present disclosure demonstrate superior performance in synthetic and real environments. When embodiments are applied to the latest photon counting devices, object detection can be achieved at a photon level of 1 photon per pixel or lower, significantly surpassing the capabilities of existing CMOS image sensors and algorithms.

Claims20

Dependent claims

A system, comprising:

one or more processors to:
(1) train a student network using a teacher network, wherein
(1a) the teacher network defines an architecture with weights, wherein the teacher network receives clean image information,
generates results based on the clean image information, and
provides the results to a comparison module;
(1b) the student network defines an architecture with weights, wherein the student network architecture is the same as the teacher network architecture, and wherein the student network
receives noisy low light image information,
generates results based on the noisy low light image information, and
provides the results to the comparison module;
(1c) after 1a and 1b, the comparison module
receives the results from the teacher network;
receives the results from the student network;
determines the differences between the results of the teacher network and the results of the student network;
generates information for updating the student network, wherein the information for updating the student network is configured to result in
large changes in the student network when the differences between the results of the teacher network and the results of the student network are large, and
small changes in the student network when the differences between the results of the teacher network and the results of the student network are small;
provides the information for updating the student network to the student network;
and
(1d) after 1c, the student network
receives the information for updating the student network from the comparison module, and
modifies the weights in the student network architecture;
and
when a termination criterion is met, the one or more processors begin stage 2;
and
when the termination criterion is not met, the one or more processors repeat stage 1 with the weighs in the student network being the modified weights;
(2) analyze a plurality of low light image frames with an image extracting module, wherein an image extracting module
receives one image frame of the plurality of low light image frames;
generates a feature map of the one image frame and the image frames adjacent to one image frame, each feature map including a plurality of feature vectors,
calculates the inner product of each feature vector in the one image frame;
calculates the inner product of each feature vector in the frames adjacent to the one image frame;
calculates the differences between the inner products of the feature vectors in the one image frame and the feature vectors in the frames adjacent to the one image frame,
selects one or more of the feature vectors with inner products that have the smallest differences between the one image frame and the adjacent image frames,
generates an aggregated non-local feature vector by averaging the inner products of the selected one or more feature vectors,
providing the aggregated non-local feature vectors to a user display.

31 more elements

A non-transitory computer-readable medium storing instructions, the instructions, comprising one or more instructions that, when executed by one or more processors, cause the one or more processors to:

(1) train a student network using a teacher network, wherein
(1a) the teacher network defines an architecture with weights, wherein the teacher network
receives clean image information,
generates results based on the clean image information, and
provides the results to a comparison module;
(1b) the student network defines an architecture with weights, wherein the student network architecture is the same as the teacher network architecture, and wherein the student network
receives noisy low light image information,
generates results based on the noisy low light image information, and
provides the results to the comparison module;
(1c) after 1a and 1b, the comparison module
receives the results from the teacher network,
receives the results from the student network, and
determines the differences between the results of the teacher network and the results of the student network;
when a termination criterion is met, the one or more processors begin stage 2;
and
when the termination criterion is not met,
the comparison module
generates information for updating the student network, and
provides the information for updating the student network to the student network;
and
the student network
receives the information for updating the student network from the comparison module, and
modifies the weights in the student network architecture;
and
the one or more processors repeat stage 1 with the weighs in the student network being the modified weights;
(2) analyze a plurality of low light image frames with an image extracting module, wherein an image extracting module
receives one image frame of the plurality of low light image frames;
generates a feature map of the one image frame and the image frames adjacent to one image frame, each feature map including a plurality of feature vectors,
calculates the inner product of each feature vector in the one image frame;
calculates the inner product of each feature vector in the frames adjacent to the one image frame;
calculates the differences between the inner products of the feature vectors in the one image frame and the feature vectors in the frames adjacent to the one image frame,
selects one or more of the feature vectors with inner products that have the smallest differences between the one image frame and the adjacent image frames,
generates an aggregated non-local feature vector by averaging the inner products of the selected one or more feature vectors,
providing the aggregated non-local feature vectors to a user display.

31 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/045,716Filed 11 Oct 2022 · us provisional application
Priority: 10 Oct 2021
Published: US 2023/0298330 A121 Sep 2023
Granted: 31 Mar 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06V 10/7784
Active pattern-learning, e.g. online learning of image or video features: based on feedback from supervisors
G06V 10/7792
Based on feedback from supervisors: the supervisor being an automated module, e.g. "intelligent oracle"
G06V 10/7715
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…
G06V 10/774
Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or…:…