# US 2017/0256239 A1 · Real-time video frame pre-processing hardware > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0256239 A1 - Document ID: US20170256239A1 - Status: Published application - Kind: A1 - Application number: 15/469,392 - Filed: 2017-03-24 - Published: 2017-09-07 - Earliest claimed: 2010-07-07 - Assignees: FotoNation Limited - Applicants: FotoNation Limited - Inventors: Corneliu ZAHARIA; Petronel BIGIOI; Peter CORCORAN - Web page: https://patentgrep.com/patent/US20170256239A1 ## CPC - G06T 1/0007: {Image acquisition} - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 1/60: Memory management - G09G 5/393: Arrangements for updating the contents of the bit-mapped memory - H04N 9/67: for matrixing (camera processing pipelines for matrixing of colour signals H04N23/85) - H04N 25/00: Circuitry of solid-state image sensors [SSIS]; Control thereof - G06F 15/8046: {Systolic arrays} - G09G 2360/12: Frame memory handling - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A dynamically reconfigurable heterogeneous systolic array is configured to process a first image frame, and to generate image processing primitives from the image frame, and to store the primitives and the corresponding image frame in a memory store. A characteristic of the image frame is determined. Based on the characteristic, the array is reconfigured to process a following image frame. ## Claims ### Claim 1 An image processing hardware unit comprising: a frame receiving component configured to: receive frame data of a first image frame; a frame processing component configured to: generate, based on the frame data of the first image frame, an image processing primitive; wherein the image processing primitive is generated by identifying a sub-region of the first image frame and generating the image processing primitive from data of the sub-region; wherein the image processing primitive comprises an image map; an integral image accumulator configured to: generate, based on the image map, an integral image (II) primitive by computing for a first pixel of the II primitive a sum of luminance values of all pixels located above and to the left from the first pixel in the image map; and store the II primitive and the first image frame in a memory store. ### Claim 2 (depends on 1) The image processing hardware unit of claim 1, wherein the integral image accumulator is further configured to: generate, based on the II primitive, an integral image square (II2) primitive as a product of the II primitive multiplied by the II primitive; and store the II2 primitive in the memory store. ### Claim 3 (depends on 2) The image processing hardware unit of claim 2, wherein the integral image accumulator is further configured to: for each pixel of the sub-region, determine a similarity score indicating a similarity between a color of the pixel and a skin color; determine a skin II primitive based on the similarity scores determined for the sub-region; and store the skin II primitive in the memory store. ### Claim 4 (depends on 3) The image processing hardware unit of claim 3, wherein the II primitive is used to match one or more Haar classifiers to an image luminance value to determine whether a face is depicted in the sub-region; wherein the II2 primitive is used to determine a variance in color values within the sub-region; and wherein the skin II primitive is used to determine a probability that a skin region is depicted in the sub-region. ### Claim 5 (depends on 4) The image processing hardware unit of claim 4, wherein the frame processing component is further configured to: determine whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and determine whether the variance is high, and if so, determine that the sub-region probably depicts the face. ### Claim 6 (depends on 5) The image processing hardware unit of claim 5, wherein the first image frame is a video frame. ### Claim 7 (depends on 6) The image processing hardware unit of claim 6, wherein the frame processing component is further configured to: generate, based on the frame data of the first image frame, a skin-map primitive; wherein the skin-map primitive is a bit-map; and store the skin-map primitive in the memory store. ### Claim 8 An image processing method comprising: receiving frame data of a first image frame; generating, based on the frame data of the first image frame, an image processing primitive; wherein the image processing primitive is generated by identifying a sub-region of the first image frame and generating the image processing primitive from data of the sub-region; wherein the image processing primitive comprises an image map; generating, based on the image map, an integral image (II) primitive by computing for a first pixel of the II primitive a sum of luminance values of all pixels located above and to the left from the first pixel in the image map; and storing the II primitive and the first image frame in a memory store; wherein the method is performed using one or more computing devices. ### Claim 9 (depends on 8) The image processing method of claim 8, further comprising: generating, based on the II primitive, an integral image square (II2) primitive as a product of the II primitive multiplied by the II primitive; and storing the II2 primitive in the memory store. ### Claim 10 (depends on 9) The image processing method of claim 9, further comprising: for each pixel of the sub-region, determining a similarity score indicating a similarity between a color of the pixel and a skin color; determining a skin II primitive based on the similarity scores determined for the sub-region; and storing the skin II primitive in the memory store. ### Claim 11 (depends on 10) The image processing method of claim 10, wherein the II primitive is used to match one or more Haar classifiers to an image luminance value to determine whether a face is depicted in the sub-region; wherein the II2 primitive is used to determine a variance in color values within the sub-region; and wherein the skin II primitive is used to determine a probability that a skin region is depicted in the sub-region. ### Claim 12 (depends on 11) The image processing method of claim 11, further comprising: determining whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and determining whether the variance is high, and if so, determine that the sub-region probably depicts the face. ### Claim 13 (depends on 12) The image processing method of claim 12, wherein the first image frame is a video frame. ### Claim 14 (depends on 13) The image processing method of claim 13, further comprising: generating, based on the frame data of the first image frame, a skin-map primitive; wherein the skin-map primitive is a bit-map; and storing the skin-map primitive in the memory store. ### Claim 15 A non-transitory computer-readable storage medium storing one or more computer instructions which, when executed, cause one or more processors to perform: receiving frame data of a first image frame; generating, based on the frame data of the first image frame, an image processing primitive; wherein the image processing primitive is generated by identifying a sub-region of the first image frame and generating the image processing primitive from data of the sub-region; wherein the image processing primitive comprises an image map; generating, based on the image map, an integral image (II) primitive by computing for a first pixel of the II primitive a sum of luminance values of all pixels located above and to the left from the first pixel in the image map; and storing the II primitive and the first image frame in a memory store. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, further comprising additional instructions which, when executed, cause the one or more processors to perform: generating, based on the II primitive, an integral image square (II2) primitive as a product of the II primitive multiplied by the II primitive; and storing the II2 primitive in the memory store. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, further comprising additional instructions which, when executed, cause the one or more processors to perform: for each pixel of the sub-region, determining a similarity score indicating a similarity between a color of the pixel and a skin color; determining a skin II primitive based on the similarity scores determined for the sub-region; and storing the skin II primitive in the memory store. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein the II primitive is used to match one or more Haar classifiers to an image luminance value to determine whether a face is depicted in the sub-region; wherein the II2 primitive is used to determine a variance in color values within the sub-region; and wherein the skin II primitive is used to determine a probability that a skin region is depicted in the sub-region. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium of claim 18, further comprising additional instructions which, when executed, cause the one or more processors to perform: determining whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and determining whether the variance is high, and if so, determine that the sub-region probably depicts the face. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein the first image frame is a video frame.