Image sensor data formats and memory addressing techniques for image signal processing

An image signal processing system comprising:

• an input configured to receive image frames comprising pixel data acquired by an image sensor, wherein the image sensor is configured to acquire the pixel data in a plurality of pixel formats, wherein each pixel format is associated with a first parameter, a second parameter, a third parameter, and a fourth parameter, wherein the first parameter comprises a minimum pixel unit (MPU), wherein the second parameter comprises a log^2 parameter based on the MPU;
• wherein the third parameter comprises an offset mask, and wherein the fourth parameter comprises a value representing a number of bytes corresponding to the MPU (BPPU);
• a memory device comprising a memory array comprising a plurality of memory blocks and being configured to store the pixel data in accordance with a memory format corresponding to the pixel format of the pixel data, wherein each memory block comprises a plurality of bytes;
• and
• a memory input/output (I/O) controller configured to receive a read request issued to the memory device, wherein in response to the read request, the memory I/O controller is configured to identify a requested image frame comprising pixels stored in the memory array using one of the pixel formats, determine a starting byte address of the requested image frame based at least partially upon the BBPU and the log^2 parameter corresponding to the pixel format of the requested pixels, use the starting byte address to identify a starting memory block within the memory array, and use the starting byte address, the starting memory block, the log^2 parameter, BPPU, and offset mask associated with the pixel format of the requested pixels to determine a width of the image frame in terms of memory blocks;
• wherein the requested image frame is read from the memory array using the starting byte address, the starting memory block, and the memory block width of the image frame.

An electronic device comprising:

• a memory array;
• an image sensor configured to acquire image frames comprising pixels, wherein the image frames are acquired using one of a plurality of available pixel memory formats, the plurality of available pixel memory formats comprising at least one pixel memory format for storing pixel data into bytes of the memory array in an aligned manner and at least one pixel memory format for storing pixel data into bytes of the memory array in an unaligned manner;
• a sensor interface configured to communicate with the image sensor;
• an image signal processing sub-system system comprising:
• an input configured to receive the pixels via the sensor interface;
• image signal processing logic configured to process the pixels;
• and
• a memory controller configured to store the pixels in the memory array by determining whether the received pixels comprise raw image data, color RGB image data or luma and chroma image data, to select a pixel memory format based upon the type of the received pixels, and to store the received pixels in the memory array using the selected memory format;
• wherein:
• when the received pixels comprise raw image data:
• if the raw pixels have a precision of 8 bits (RAW8), a first raw pixel memory format is used to store each raw pixel into a corresponding byte of the memory array;
• if the raw pixels have a precision of 10 bits (RAW 10), a second raw pixel memory format is used store every four of the 10-bit raw pixels into five bytes of the memory array, wherein the upper 8 bits of a first raw pixel of the four 10-bit raw pixels is stored in a first byte of the five bytes, wherein the upper 8 bits of a second raw pixel of the four 10-bit raw pixels is stored in a second byte of the five bytes, wherein the upper 8 bits of a third raw pixel of the four 10-bit raw pixels is stored in a third byte of the five bytes, wherein the upper 8 bits of a fourth raw pixel of the four 10-bit raw pixels is stored in a fourth byte of the five bytes, and wherein the lower 2 bits of each of the first, second, third, and fourth 10-bit raw pixels are stored in a fifth byte of the five bytes;
• if the raw pixels have a precision of 12 bits (RAW 12), a third raw pixel memory format is used store every two of the 12-bit raw pixels into three bytes of the memory array, wherein the upper 8 bits of a first raw pixel of the two 12-bit raw pixels is stored in a first byte of the three bytes, wherein the upper 8 bits of a second raw pixel of the two 12-bit raw pixels is stored in a second byte of the three bytes, and wherein the lower 4 bits of each of the first and second 12-bit raw pixels are stored in a third byte of the three bytes;
• if the raw pixels have a precision of 14 bits (RAW 14), a fourth raw pixel memory format is used store every four of the 14-bit into seven bytes of the memory array, wherein the upper 8 bits of a first raw pixel of the four 14-bit raw pixels is stored in a first byte of the seven bytes, wherein the upper 8 bits of a second raw pixel of the four 14-bit raw pixels is stored in a second byte of the seven bytes, wherein the upper 8 bits of a third raw pixel of the four 14-bit raw pixels is stored in a third byte of the seven bytes, wherein the upper 8 bits of a fourth raw pixel of the four 14-bit raw pixels is stored in a fourth byte of the seven bytes, wherein the lower 6 bits of the first raw pixel of the four 14-bit raw pixels is stored in a first portion of a fifth byte of the seven bytes, wherein the lower 6 bits of the second raw pixel of the four 14-bit raw pixels is stored in a second portion of the fifth byte of the seven bytes and in a first portion of a sixth byte of the seven bytes, wherein the lower 6 bits of the third raw pixel of the four 14-bit raw pixels is stored in a second portion of the sixth byte of the seven bytes and in a first portion of a seventh byte of the seven bytes, and wherein the lower 6 bits of the fourth raw pixel of the four 14-bit raw pixels is stored in a second portion of the seventh byte of the seven bytes;
• and
• if the raw pixels have a precision of 16 bits (RAW 16), a fifth raw pixel memory format is used to store each of the 16-bit raw pixels into two bytes of the memory array;
• or
• when the received pixels comprise color RGB pixels:
• if the color RGB pixels comprise RGB-565 pixels, a first RGB pixel memory format is used to store sixteen bits of color RGB pixel data into two bytes of the memory array, wherein the sixteen bits of color RGB pixel data comprises a 5-bit red pixel, a 6-bit green pixel, and a 5-bit blue pixel;
• if the color RGB pixels comprise RGB-888 pixels, a second RGB pixel memory format is used to store each of an 8-bit blue pixel, an 8-bit green pixel, and an 8-bit red pixel into respective first, second, and third bytes of the memory array;
• and
• if the color RGB pixels comprise RGB-666 pixels, a third RGB pixel memory format is uses to store a 6-bit red pixel into a first memory byte in a left justified manner, a 6-bit green pixel into a second memory byte in a left justified manner, and a 6-bit blue pixel into a third memory byte in a left justified manner, wherein the remaining two bites of each of the first, second, and third memory bytes are set to zero;
• or
• when the received pixels comprise luma and chroma pixels:
• if the luma and chroma pixels comprises YCC-4:2:0, 2-plane, pixels, a first luma and chroma pixel memory format is used to store an 8-bit luma pixel in a first byte of a first plane of the memory array and two 8-bit chroma pixels in first and second bytes, respectively, of a second plane of the memory array;
• and
• if the luma and chroma pixels comprise YCC-4:2:2, 1-plane, pixels, a second luma and chroma pixel memory format is used to store an 8 bit luma pixel in a first byte of a first plane of the memory array, and two 8-bit chroma pixels in second and third bytes of the first plane of the memory array, wherein chroma samples are sub-sampled by a factor of two in both horizontal and vertical directions of the memory array;
• or
• any combination thereof.

A method comprising:

• using an image sensor configured to acquire image frames representative of an image scene, wherein each image frame comprises a plurality of pixels;
• providing a plurality of pixel memory formats;
• providing the plurality of pixels to an image signal processing system, wherein the plurality of pixels are of one of a plurality of pixel types, each of the plurality of pixel types being configured to be stored in a memory array of a memory device using a respective one of the plurality of pixel memory formats;
• based upon the pixel type of the plurality of pixels, selecting a pixel memory format corresponding to the pixel type;
• and
• using a memory input/output controller associated with the memory device to store the plurality of pixels into the memory array based upon the selected pixel memory format;
• wherein the pixel memory formats comprise at least one pixel memory format that is aligned with bytes of the memory array and at least one pixel memory format that is not aligned with bytes of the memory array;
• wherein the plurality of pixel types comprises at least one of raw pixels, color RGB pixels, and luma and chroma pixels;
• wherein the pixel type of the plurality of pixels comprises color RGB pixels;
• wherein if the color RGB pixels comprise RGB-565 pixels, a first RGB pixel memory format is used to store sixteen bits of color RGB pixel data into two bytes of the memory array, wherein the sixteen bits of color RGB pixel data comprises a 5-bit red pixel, a 6-bit green pixel, and a 5-bit blue pixel;
• wherein if the color RGB pixels comprise RGB-888 pixels, a second RGB pixel memory format is used to store each of an 8-bit blue pixel, an 8-bit green pixel, and an 8-bit red pixel into respective first, second, and third bytes of the memory array;
• and
• wherein if the color RGB pixels comprise RGB-666 pixels, a third RGB pixel memory format is uses to store a 6-bit red pixel into a first memory byte in a left justified manner, a 6-bit green pixel into a second memory byte in a left justified manner, and a 6-bit blue pixel into a third memory byte in a left justified manner, wherein the remaining two bites of each of the first, second, and third memory bytes are set to zero.

A method comprising:

• using an image sensor configured to acquire image frames representative of an image scene, wherein each image frame comprises a plurality of pixels;
• providing a plurality of pixel memory formats;
• providing the plurality of pixels to an image signal processing system, wherein the plurality of pixels are of one of a plurality of pixel types, each of the plurality of pixel types being configured to be stored in a memory array of a memory device using a respective one of the plurality of pixel memory formats;
• based upon the pixel type of the plurality of pixels, selecting a pixel memory format corresponding to the pixel type;
• and
• using a memory input/output controller associated with the memory device to store the plurality of pixels into the memory array based upon the selected pixel memory format;
• wherein the pixel memory formats comprise at least one pixel memory format that is aligned with bytes of the memory array and at least one pixel memory format that is not aligned with bytes of the memory array;
• wherein the plurality of pixel types comprises at least one of raw pixels, color RGB pixels, and luma and chroma pixels;
• wherein the pixel type of the plurality of pixels comprise luma and chroma pixels;
• wherein if the luma and chroma pixels comprises YCC-4:2:0, 2-plane, pixels, a first luma and chroma pixel memory format is used to store an 8-bit luma pixel in a first byte of a first plane of the memory array and two 8-bit chroma pixels in first and second bytes, respectively, of a second plane of the memory array;
• and
• wherein if the luma and chroma pixels comprise YCC-4:2:2, 1-plane, pixels, a second luma and chroma pixel memory format is used to store an 8 bit luma pixel in a first byte of a first plane of the memory array, and two 8-bit chroma pixels in second and third bytes of the first plane of the memory array, wherein chroma samples are sub-sampled by a factor of two in both horizontal and vertical directions of the memory array.