US 11,768,919 B2Granted

Image processing system

AssigneeFotoNation Limited

InventorsCian Ryan, Richard Blythman, Joseph Lemley, Paul Kielty

Life of this patent

Estimating the term…

Loading patent…

Abstract

Disclosed is a multi-modal convolutional neural network (CNN) for fusing image information from a frame based camera, such as, a near infra-red (NIR) camera and an event camera for analysing facial characteristics in order to produce classifications such as head pose or eye gaze. The neural network processes image frames acquired from each camera through a plurality of convolutional layers to provide a respective set of one or more intermediate images. The network fuses at least one corresponding pair of intermediate images generated from each of image frames through an array of fusing cells. Each fusing cell is connected to at least a respective element of each intermediate image and is trained to weight each element from each intermediate image to provide the fused output. The neural network further comprises at least one task network configured to generate one or more task outputs for the region of interest.

Claims17

Dependent claims

An image processing system comprising:

a frame based camera configured to periodically provide an image frame covering a field of view of the camera;
an event camera having a substantially common field of view and configured to provide event information in response to an event indicating a detected change in light intensity at an x,y location within the field of view of the event camera exceeding a threshold;
a detector for identifying a region of interest within the common field of view;
an accumulator for accumulating event information from a plurality of events occurring during successive event cycles within said region of interest, each event indicating an x,y location within said region of interest, a polarity for a change of detected light intensity incident at said x,y location and an event cycle at which said event occurred, and responsive to an event criterion for the region of interest being met, for generating an image frame for the region of interest from event information accumulated from within the region of interest;
a neural network configured to receive an image frame from the frame based camera and an image frame from the accumulator for the region of interest, the neural network being configured to process each image frame through a plurality of convolutional layers to provide a respective set of one or more intermediate images, the neural network being further configured to fuse at least one corresponding pair of intermediate images generated from each of said image frames through an array of fusing cells that are arranged in a linear and sequential manner, each fusing cell being connected to at least a respective element of each intermediate image and being trained to weight each element from each intermediate image to provide said fused output, said neural network further comprising at least one task network configured to receive said fused output from a final pair of intermediate images and to generate one or more task outputs for the region of interest.

2 more elements

An image processing method operable in a system comprising:

a frame based camera configured to periodically provide an image frame covering a field of view of the camera;
and an event camera having a substantially common field of view and configured to provide event information in response to an event indicating a detected change in light intensity at an x,y location within the field of view of the event camera exceeding a threshold, the method comprising;
identifying a region of interest with the common field of view;
accumulating event information from a plurality of events occurring during successive event cycles within said region of interest, each event indicating an x,y location within said region of interest, a polarity for a change of detected light intensity incident at said x,y location and an even cycle at which said event occurred, and responsive to an event criterion for the region of interest being met, generating an image frame for the region of interest from event information accumulated from within the region of interest;
receiving an image frame from the frame based camera and an image frame from the accumulator for the region of interest;
processing each image frame through a plurality of convolutional layers to provide a respective set of one or more intermediate images;
fusing at least one corresponding pair of intermediate images generated from each of said image frames through an array of fusing cells that are arranged in a linear and sequential manner, each fusing cell being connected to at least a respective element of each intermediate image and being trained to weight each element from each intermediate image to provide said fused output;
receiving said fused output from a final pair of intermediate images;
and generating one or more task outputs for the region of interest.

6 more elements

Dependencies are inferred from the claim text.