# US 2025/0156985 A1 · Method and device for latency reduction of an image processing pipeline

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156985 A1
- Document ID: US20250156985A1
- Status: Published application
- Kind: A1
- Application number: 18/922,265
- Filed: 2024-10-21
- Published: 2025-05-15
- Earliest claimed: 2020-04-08
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Bertrand Nepveu; Marc-Andre Chenier; Yan Cote; Yves Millette
- Web page: https://patentgrep.com/patent/US20250156985A1

## CPC

- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- H04N 21/816: {involving special video data, e.g 3D video}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06T 5/73: Deblurring; Sharpening
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 7/73: using feature-based methods
- G06T 15/005: {General purpose rendering architectures}
- G06T 15/20: Perspective computation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 21/42201: {biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user (input arrangements for interaction with the human body based on nervous system activity detection G06F3/015)}
- H04N 21/42202: {environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/44004: {involving video buffer management, e.g. video decoder buffer or video display buffer}
- H04N 21/440281: {by altering the temporal resolution, e.g. by frame skipping (television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783)}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In some implementations, a method includes: determining a complexity value for first image data associated with of a physical environment that corresponds to a first time period; determining an estimated composite setup time based on the complexity value for the first image data and virtual content for compositing with the first image data; in accordance with a determination that the estimated composite setup time exceeds the threshold time: forgoing rendering the virtual content from the perspective that corresponds to the camera pose of the device relative to the physical environment during the first time period; and compositing a previous render of the virtual content for a previous time period with the first image data to generate the graphical environment for the first time period.

## Claims

### Claim 1

A method comprising:
at a device including one or more processors, non-transitory memory, and an image capture device including an image sensor:
obtaining, via the image sensor, image data;
determining a focal region of the image data based on contextual information; and
generating a quadtree associated with the image data based on the focal region.

### Claim 2 (depends on 1)

The method of claim 1, wherein pixels within the focal region of the image data are associated with a first size of quadtree node within the quadtree and pixels outside of the focal region of the image data are associated with a second size of quadtree node within the quadtree larger than the first size of quadtree node, wherein the first size of quadtree node results in higher resolution as compared to second size of quadtree node.

### Claim 3 (depends on 1)

The method of claim 1, wherein the contextual information includes at least one of head pose information, body pose information, limb pose information, or gaze direction information.

### Claim 4 (depends on 1)

The method of claim 1, wherein the image capture device further includes a and a front-end architecture, and the method further comprising:
generating, via the front-end architecture, a color filter array (CFA) mosaic image based on the image data, wherein the image data corresponds to an analog image.

### Claim 5 (depends on 4)

The method of claim 4, wherein generating the quadtree associated with the image data based on the focal region includes transforming the CFA mosaic into a quadtree CFA mosaic based on the focal region.

### Claim 6 (depends on 4)

The method of claim 4, wherein the front-end architecture includes an analog-to-digital converter (ADC) that converts the analog image into the CFA mosaic.

### Claim 7 (depends on 4)

The method of claim 4, wherein the front-end architecture includes an analog amplifier that amplifies the analog image and a correction engine that performs at least one of lens correction or defect pixel correction on the CFA mosaic.

### Claim 8 (depends on 4)

The method of claim 4, wherein the analog image corresponds to RAW image data, and wherein the image frame corresponds to an RGB image frame.

### Claim 9 (depends on 4)

The method of claim 4, wherein the device further includes an image processing architecture, and the method further comprising:
reading a first slice of the quadtree CFA mosaic into one or more input buffers of the image processing architecture;
performing, at the image processing architecture, one or more image processing operations on the first slice of the quadtree CFA mosaic to obtain a first portion of image data;
reading a second slice of the quadtree CFA mosaic into the one or more input buffers of the image processing architecture;
performing, at the image processing architecture, the one or more image processing operations on the second slice of the quadtree CFA mosaic to obtain a second portion of the image data; and
generating an image frame of the physical environment based at least in part on the first and second portions of the image data.

### Claim 10 (depends on 9)

The method of claim 9, wherein the one or more image processing operations correspond to at least one of: white balance correction, de-mosaicking, color correction, gamma correction, or sharpening.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
generating a graphical environment by compositing the image frame with virtual content.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
rendering the virtual content based on a camera pose of the image capture device relative to the virtual content.

### Claim 13 (depends on 11)

The method of claim 11, further comprising:
presenting the graphical environment via a display device.

### Claim 14 (depends on 13)

The method of claim 13, wherein the device corresponds to a near-eye system that includes the display device, and wherein the image capture device corresponds to a scene-facing image sensor.

### Claim 15

A device comprising:
one or more processors;
a non-transitory memory;
an image capture device including an image sensor; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
obtain, via the image sensor, image data;
determine a focal region of the image data based on contextual information; and
generate a quadtree associated with the image data based on the focal region.

### Claim 16 (depends on 15)

The device of claim 15, wherein pixels within the focal region of the image data are associated with a first size of quadtree node within the quadtree and pixels outside of the focal region of the image data are associated with a second size of quadtree node within the quadtree larger than the first size of quadtree node, wherein the first size of quadtree node results in higher resolution as compared to second size of quadtree node.

### Claim 17 (depends on 15)

The device of claim 15, wherein the contextual information includes at least one of head pose information, body pose information, limb pose information, or gaze direction information.

### Claim 18 (depends on 15)

The device of claim 15, wherein the image capture device further includes a and a front-end architecture, and wherein the one or more programs further cause the device to:
generate, via the front-end architecture, a color filter array (CFA) mosaic image based on the image data, wherein the image data corresponds to an analog image.

### Claim 19 (depends on 18)

The device of claim 18, wherein generating the quadtree associated with the image data based on the focal region includes transforming the CFA mosaic into a quadtree CFA mosaic based on the focal region.

### Claim 20

A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device with an image processing architecture and an image capture device including a photodiode and a front-end architecture, cause the device to:
obtain, via the image capture device, first image data corresponding to a physical environment;
determine a focal region based on contextual information; and
perform a pixel binning operation on the first image data based on the focal region to generate a quadtree version of the first image data;
read a first slice of the quadtree version of the first image data into one or more input buffers of the image processing architecture;
perform, at the image processing architecture, one or more image processing operations on the first slice of the first image data to obtain a first portion of second image data;
read a second slice of the quadtree version of the first image data into the one or more input buffers of the image processing architecture;
perform, at the image processing architecture, the one or more image processing operations on the second slice of the first image data to obtain a second portion of the second image data; and
generate an image frame of the physical environment based at least in part on the first and second portions of the second image data.
