# US 2024/0007771 A1 · Readout methods for foveated sensing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0007771 A1
- Document ID: US20240007771A1
- Status: Published application
- Kind: A1
- Application number: 18/208,236
- Filed: 2023-06-09
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Andrew Samuel Berkovich; Jack Thomas Lavier; Reid Frederick Pinkham; Shlomo Alkalay
- Web page: https://patentgrep.com/patent/US20240007771A1

## CPC

- H04N 25/443: by reading pixels from selected two-dimensional [2D] regions of the array, e.g. for windowing or digital zooming
- H04N 25/78: Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- H04N 19/103: Selection of coding mode or of prediction mode
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/172: the region being a picture, frame or field
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N 25/40: Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N 25/42: by switching between different modes of operation using different resolutions or aspect ratios, e.g. switching between interlaced and non-interlaced mode
- H04N 25/445: by skipping some contiguous pixels within the read portion of the array
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and readout methods for foveated sensing may include a pixel array, readout circuitry, and processing logic. The pixel array may have a number of pixels. The readout circuitry may be configured to read image data from the pixel array for each of the plurality of pixels. The processing logic may be configured to identify a number of regions of interest (ROIs) within the pixel array. The processing logic may be configured to associate the image data for the pixels with one or more ROIs. The processing logic may be configured to arrange the image data into data frames. The data frames may include the image data ordered by ROI. Image data for inactive pixels may be removed from the image data and data frames prior to transmission. The processing logic may be configured to transmit the data frames in an order that is based on the ROIs.

## Claims

### Claim 1

An image sensor, comprising:
a pixel array having a plurality of pixels;
readout circuitry configured to read image data from the pixel array for each of the plurality of pixels; and
processing logic configured to:
identify a plurality of regions of interest (ROIs) within the pixel array;
associate the image data for at least one of the plurality of pixels with at least one of the plurality of ROIs;
arrange the image data into a plurality of data frames, wherein the plurality of data frames include the image data ordered by the plurality of ROIs; and
transmit the plurality of data frames in an order that is based on the plurality of ROIs.

### Claim 2 (depends on 1)

The image sensor of claim 1, wherein the plurality of data frames include the image data grouped by the plurality of ROIs for each row of the pixel array, wherein the plurality of data frames are transmitted row-by-row for the plurality of pixels in the pixel array.

### Claim 3 (depends on 2)

The image sensor of claim 2, wherein the plurality of pixels include active pixels and inactive pixels, wherein the plurality of data frames include the image data of the active pixels and omit the image data of the inactive pixels.

### Claim 4 (depends on 1)

The image sensor of claim 1, wherein each of the plurality of data frames is ordered and transmitted as an entire one of the plurality of ROIs, wherein the image data of a first of the plurality of ROIs is transmitted prior to the image data of a second of the plurality of ROIs.

### Claim 5 (depends on 4)

The image sensor of claim 4, wherein the readout circuitry is configured to read the image data one ROI at a time.

### Claim 6 (depends on 1)

The image sensor of claim 1, wherein a first of the plurality of ROIs overlaps with a portion of a second of the plurality of ROIs.

### Claim 7 (depends on 1)

The image sensor of claim 1, wherein the processing logic is configured to i) define active pixels and inactive pixels in the pixel array and ii) remove the image data for the inactive pixels from the plurality of data frames.

### Claim 8 (depends on 1)

The image sensor of claim 1, wherein the processing logic is configured to: i) receive ROI configuration settings and ii) define a size and a sampling ratio for each of the plurality of ROIs based on the ROI configuration settings.

### Claim 9

A method, comprising:
identifying a plurality of regions of interest (ROIs) in a pixel array of an image sensor;
defining active pixels and inactive pixels for each of the plurality of ROIs;
reading, on the image sensor, image data for the active pixels and inactive pixels from each row of the pixel array;
compressing the image data by removing the image data for the inactive pixels; and
transmitting the image data to a computing device.

### Claim 10 (depends on 9)

The method of claim 9 further comprising:
arranging the image data into a plurality of data frames, wherein each of the plurality of data frames represents a corresponding one of the plurality of ROIs,
wherein transmitting the image data includes transmitting the plurality of data frames in an order of the plurality of ROIs, one ROI at a time.

### Claim 11 (depends on 10)

The method of claim 10, wherein reading the image data includes reading a particular row of the pixel array once for each of the plurality of ROIs included in the particular row of the pixel array.

### Claim 12 (depends on 9)

The method of claim 9 further comprising:
buffering, on the image sensor, the image data for the active pixels and the inactive pixels prior to compressing the image data.

### Claim 13 (depends on 9)

The method of claim 9 further comprising:
arranging the image data for each row of the pixel array into a plurality of data frames, wherein each of the plurality of data frames includes sub-frames of the image data, wherein each of the sub-frames represents a portion of the image data for a particular one of the plurality of ROIs in a particular row of the pixel array,
wherein transmitting the image data includes transmitting the plurality of data frames by each row of the pixel array, wherein transmitting the plurality of data frames includes transmitting the sub-frames in an order of the plurality of ROIs that are included in the plurality of data frames.

### Claim 14 (depends on 9)

The method of claim 9, wherein each of the plurality of ROIs corresponds with one of a plurality of sampling ratios, wherein reading the image data includes reading the image data with one or more of the plurality of sampling ratios.

### Claim 15 (depends on 9)

The method of claim 9 further comprising:
receiving ROI configuration settings; and
defining a size and a sampling ratio for each of the plurality of ROIs based on the ROI configuration settings.

### Claim 16

A head-mounted device comprising:
an image sensor configured to provide image data from a pixel array of a plurality of pixels;
a display;
processing logic; and
one or more memories coupled to the processing logic, the one or more memories storing instructions that when executed by the processing logic, cause the head-mounted device to perform operations comprising:
estimate a gaze vector;
define a plurality of regions of interest (ROIs) for the pixel array at least partially based on the gaze vector;
transmit ROI configuration settings to the image sensor;
receive a plurality of data frames from the image sensor, wherein the plurality of data frames include the image data that is ordered by the plurality of ROIs; and
provide display data on the display at least partially based on the plurality of data frames.

### Claim 17 (depends on 16)

The head-mounted device of claim 16, wherein the ROI configuration settings include:
a quantity of the plurality of ROIs;
a size of each of the plurality of ROIs;
a sampling ratio for each of the plurality of ROIs; and
a data frame format.

### Claim 18 (depends on 17)

The head-mounted device of claim 17, wherein the data frame format is at least one of a plurality of data frame formats, wherein the plurality of data frame formats include:
each of the plurality of data frames represents one of the plurality of ROIs;
each of the plurality of data frames represents one row of the pixel array, wherein each of the plurality of data frames includes sub-frames of the image data ordered based on ones of the plurality of ROIs that are represented in the image data; or
each of the plurality of data frames includes the image data for active pixels of the pixel array and excludes the image data for inactive pixels of the pixel array, wherein the active pixels and inactive pixels are at least partially defined by the sampling ratio for each of the plurality of ROIs.

### Claim 19 (depends on 16)

The head-mounted device of claim 16, wherein the one or more memories store further instructions that when executed by the processing logic, cause the head-mounted device to perform further operations comprising:
determine the gaze vector based on an orientation of the head-mounted device.

### Claim 20 (depends on 16)

The head-mounted device of claim 16, wherein the one or more memories store further instructions that when executed by the processing logic, cause the head-mounted device to perform further operations comprising:
determine an orientation of an eye of a user; and
define the gaze vector based on the orientation of the eye of the user.
