Back to results
US 11,706,546 B2Granted
Image sensor with integrated single object class detection deep neural network (DNN)
AssigneeSony Semiconductor Solutions Corporation
InventorKevin Chan
Life of this patent
Estimating the term…
Abstract
An image sensor, electronic device and method thereof that performs on-sensor single object class detection using an on-sensor single object class detection deep neural network (DNN), such as a face detection DNN. The single object class detection DNN includes a pixel array layer configured to capture an image and transfer image data of the captured image, and a logic and single object class detection deep neural network (DNN) layer that receives the image data directly from the pixel array layer and outputs the image data with the single object class detection data to a communication bus of an electronic device.
Claims18
A stacked image sensor comprising:
- a pixel array layer configured to capture an image and transfer image data of the captured image;
- and
- a logic and single object class detection deep neural network (DNN) layer, the logic and single object class detection DNN layer including a single object class detection DNN, wherein
- the logic and the single object class detection DNN layer is configured to:
- receive the image data directly from the pixel array layer;
- perform preprocessing of the image data to generate preprocessed image data;
- perform single class object detection with the single object class detection DNN using at least a portion of the preprocessed image data;
- perform postprocessing of single object class detection data output by the single object class detection DNN;
- and
- output the image data in combination with the postprocessed single object class detection data to a communication bus of an electronic device, and
- wherein the logic and single object class detection DNN layer is configured to implement a feature extractor module, a box predictor module and a class predictor module, each of which comprise a plurality of convolutional blocks and inverted residual blocks.
8 more elements
A stacked image sensor comprising:
- a pixel array layer configured to capture an image and transfer image data of the captured image;
- and
- a logic and single object class detection deep neural network (DNN) layer, the logic and single object class detection DNN layer including a single object class detection DNN, wherein
- the logic and the single object class detection DNN layer is configured to:
- receive the image data directly from the pixel array layer;
- perform preprocessing of the image data to generate preprocessed image data;
- perform single class object detection with the single object class detection DNN using at least a portion of the preprocessed image data;
- perform postprocessing of single object class detection data output by the single object class detection DNN;
- and
- output the image data in combination with the postprocessed single object class detection data to a communication bus of an electronic device,
- wherein the postprocessing of the single object class detection data output by the single object class detection DNN includes using first order linear approximations in place of logarithm and exponential functions.
8 more elements
A stacked image sensor comprising:
- a pixel array layer configured to capture an image and transfer image data of the captured image;
- and
- a logic and single object class detection deep neural network (DNN) layer, the logic and single object class detection DNN layer including a single object class detection DNN, wherein
- the logic and the single object class detection DNN layer is configured to:
- receive the image data directly from the pixel array layer;
- perform preprocessing of the image data to generate preprocessed image data;
- perform single class object detection with the single object class detection DNN using at least a portion of the preprocessed image data;
- perform postprocessing of single object class detection data output by the single object class detection DNN;
- and
- output the image data in combination with the postprocessed single object class detection data to a communication bus of an electronic device, and
- wherein the logic and single object class detection DNN layer is further configured to:
- capture low resolution motion detection data;
- determine whether a motion has been detected;
- and
- based on the motion being detected, switch between a low-resolution capture mode and a high-resolution capture mode of the stacked image sensor based on a presence or an absence of a detected object of the single object class.
12 more elements
A stacked image sensor comprising:
- a pixel array layer configured to capture an image and transfer image data of the captured image;
- and
- a logic and single object class detection deep neural network (DNN) layer, the logic and single object class detection DNN layer including a single object class detection DNN, wherein
- the logic and the single object class detection DNN layer is configured to:
- receive the image data directly from the pixel array layer;
- perform preprocessing of the image data to generate preprocessed image data;
- perform single class object detection with the single object class detection DNN using at least a portion of the preprocessed image data;
- perform postprocessing of single object class detection data output by the single object class detection DNN;
- and
- output the image data in combination with the postprocessed single object class detection data to a communication bus of an electronic device, and
- wherein the logic and single object class detection DNN layer is further configured to:
- switch between a low-power motion detection mode, a low-resolution capture mode, and a high-resolution capture mode of an image sensor based on a change in received illuminance and a presence or an absence of a detected object of the single object class.
9 more elements
A method comprising:
- storing a single object class deep neural network (DNN) in a logic and single object class detection DNN layer of a stacked image sensor;
- capturing, by a pixel array layer of the stacked image sensor, an image;
- transferring, by the pixel array layer, image data of the captured image to the logic and single object class detection deep neural network (DNN) layer of the stacked image sensor;
- performing, by the logic and single object class detection DNN, preprocessing of the image data to generate preprocessed image data;
- performing, by the logic and single object class detection DNN layer, single object class detection using at least a portion of the preprocessed image data as input;
- performing, by the logic and single object class detection DNN, postprocessing of single object class detection data output by the single object class detection DNN;
- and
- outputting, by the logic and single object class detection DNN, the captured image in combination with the postprocessed single object class detection data to a communication bus of an electronic device,
- wherein the single object class detection DNN is configured such that an output size for each convolutional block and each inverted residual block of the single object class detection DNN is no larger than available working memory in the stacked image sensor.
6 more elements
An electronic device comprising:
- a processor,
- a memory,
- an optical lens;
- a communications bus;
- and
- a stacked image sensor, wherein
- the stacked image sensor comprises:
- a pixel array layer configured to capture an image and transfer image data of the captured image;
- and
- a logic and single object class detection deep neural network (DNN) layer, the logic and single object class detection DNN layer including a single object class detection DNN,
- the logic and the single object class detection DNN layer is configured to:
- receive the image data directly from the pixel array layer;
- perform preprocessing of the image data to generate preprocessed image data;
- perform single object class detection with the single object class detection DNN using at least a portion of the preprocessed image data;
- perform postprocessing of single object class detection data output by the single object class detection DNN;
- and
- output the image data in combination with the postprocessed single object class detection data to the communications bus, and
- wherein the logic and single object class detection DNN layer is configured to implement a feature extractor module, a box predictor module and a class predictor module, each of which comprise a plurality of convolutional blocks and inverted residual blocks.
15 more elements
Dependencies are inferred from the claim text.