Methods and apparatus of brain-like in-pixel intelligent processing system

A brain-like in-pixel intelligent processing system, comprising:

• an in-pixel processing array, wherein the in-pixel processing array comprises a plurality of in-pixel processing units, a saccadic pixel column selector, and a saccadic pixel row selector, wherein the saccadic pixel column selector and the saccadic pixel row selector are used for selecting output from selected in-pixel processing units according to a saccadic eye movement algorithm based on an acquired image, and each of the in-pixel processing units comprises:
• a photogate sensor, wherein the photogate sensor captures a pixel of an image of an object and produces an Iout current to the in-pixel processing array;
• an average circuit, wherein the average circuit receives and averages Iout current from the in-pixel processing unit and a plurality of neighboring in-pixel processing units around the in-pixel processing unit and an averaged Iout current forms a periphery (P) element of the in-pixel processing unit;
• a subtraction circuit, wherein the periphery (P) element of the in-pixel processing unit formed is used by the subtraction circuit to generate a fovea (F) element of the in-pixel processing unit;
• and
• an absolute circuit, wherein the F element of the in-pixel processing unit generated in pixel level is sent to the absolute circuit via pixel mapping to generate lateral geniculate nucleus (LGN) element of the in-pixel processing unit;
• and
• wherein the in-pixel processing array forms an image acquisition and bio-inspired processing imager configured to processes raw gray information in a pixel level;
• wherein an output of the in-pixel processing array is from a plurality of selected in-pixel processing units of the in-pixel processing array, and the plurality of selected in-pixel processing units of the in-pixel processing array is selected according to the saccadic eye movement algorithm based on the acquired image;
• a feature vector generator, wherein the feature vector generator receives and processes the P element, the F element, and the LGN element from each of selected in-pixel processing units of the in-pixel processing array, and generates the P feature vector, the F feature vector, and the LGN feature vector of the in-pixel processing array;
• a neural network processor, wherein the neural network processor receives and processes the P feature vector, the F feature vector, and the LGN feature vector of the in-pixel processing array received;
• a region and object of interest identifier, wherein the region and object of interest identifier identifies the region and object of interest from the processed P, F, and LGN feature vectors of the in-pixel processing array, and
• wherein each of the plurality of selected in-pixel processing units produces a periphery (P) element, a fovea (F) element, and a lateral geniculate nucleus (LGN) element of the in-pixel processing unit, the feature vector generator generates a P feature vector, an F feature vector, and an LGN feature vector of the in-pixel processing array of the image of the object from the P element, the F element, and the LGN element of each of the plurality of selected in-pixel processing units received, the neural network processor receives and processes the P feature vector, the F feature vector, and the LGN feature vector of the in-pixel processing array, detects the object, recognizes the object, and determines the location of the object, and the region and object of interest identifier provides feedback of the region and the object of interest to the in-pixel processing array to improve the object detection and identification.

A method of using a brain-like in-pixel intelligent processing system, comprising:

• exposing an object to the brain-like in-pixel intelligent processing system, wherein the brain-like in-pixel intelligent processing system comprises:
• an in-pixel processing array configured to acquire raw gray information of the object and to process the raw gray information acquired in a pixel level, a feature vector generator, a neural network processor, and a region and object of interest identifier, wherein the in-pixel processing array comprises a plurality of in-pixel processing units, a saccadic pixel column selector, and a saccadic pixel row selector for selecting output from selected in-pixel processing units according to a saccadic eye movement algorithm based on an acquired image, and each of the plurality of in-pixel processing units comprises a photogate sensor, an average circuit, a subtraction circuit, and an absolute circuit;
• producing, by the photogate sensor of a plurality of selected in-pixel processing units, an Iout current of each selected in-pixel processing unit in response to the exposure to the object, wherein the plurality of selected in-pixel processing units of the in-pixel processing array is selected according to the saccadic eye movement algorithm based on the acquired image;
• averaging, by the average circuit of each of the plurality of selected in-pixel processing units, the Iout current from the selected in-pixel processing unit and a plurality of neighboring in-pixel processing units around the selected in-pixel processing unit to generate a periphery (P) element of each selected in-pixel processing unit;
• subtracting, by the subtraction circuit of each of the plurality of selected in-pixel processing units, the P element from the each of the plurality of selected in-pixel processing units to generate a corresponding fovea (F) element for each of the plurality of selected in-pixel processing units;
• producing, by the absolute circuit of each of the plurality of selected in-pixel processing units, a corresponding lateral geniculate nucleus (LGN) element for each of the plurality of selected in-pixel processing units from each of the corresponding F elements of the plurality of selected in-pixel processing units;
• generating, by the feature vector generator of the brain-like in-pixel intelligent processing system, a corresponding P feature vector, F feature vector, and LGN feature vector from each of the corresponding P elements, F elements, and LGN elements of each of the plurality of selected in-pixel processing units;
• processing, by the neural network processor, the P feature vector, the F feature vector, and the LGN feature vector of the plurality of selected in-pixel processing units to detect the object, to identify the object, and obtain the location of the object;
• identifying, by the region and object of interest identifier, the region and object of interest from the processed P, F, and LGN feature vectors of the brain-like in-pixel intelligent processing system;
• and
• transmitting, by the region and object of interest identifier, the identified region and object of interest and the processed P, F, and LGN feature vectors of the brain-like in-pixel intelligent processing system back to in-pixel processing array to improve the object detection and identification.