# US 2023/0058472 A1 · Sensor prioritization for composite image capture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0058472 A1
- Document ID: US20230058472A1
- Status: Published application
- Kind: A1
- Application number: 17/795,523
- Filed: 2021-01-27
- Published: 2023-02-23
- Earliest claimed: 2020-01-28
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Sylvain Leroy; Guillaume Matthieu Guerin; Yoël Taïeb
- Web page: https://patentgrep.com/patent/US20230058472A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 7/292: Multi-camera tracking
- G06V 40/161: {Detection; Localisation; Normalisation}
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/51: Housings
- H04N 23/61: based on recognised objects
- H04N 23/62: Control of parameters via user interfaces
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)}
- G06T 2200/28: involving image processing hardware
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

Systems and methods are disclosed for sensor prioritization for composite image capture. For example, methods may include selecting an image sensor as a prioritized sensor from among an array of two or more image sensors; determining one or more image processing parameters based on one or more images captured using the prioritized sensor; applying image processing using the one or more image processing parameters to images captured with each image sensor in the array of two or more image sensors to obtain respective processed images for the array of two or more image sensors; and stitching the respective processed images for the array of two or more image sensors to obtain a composite image.

## Claims

### Claim 1

A system comprising:
an array of two or more image sensors configured to capture images; and
a processing apparatus that is configured to:
select an image sensor as a prioritized sensor from among the array of two or more image sensors;
access one or more images captured using the prioritized sensor;
determine one or more image processing parameters based on the one or more images captured using the prioritized sensor;
access a second image captured using a second image sensor of the array of two or more image sensors, wherein the second image sensor is other than the prioritized sensor; and
apply image processing using the one or more image processing parameters to the second image to obtain a processed image.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more image processing parameters are determined without consideration of image data from image sensors other than the prioritized sensor.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processed image is a first processed image, and the processing apparatus is configured to:
apply image processing using the one or more image processing parameters to an image captured using the prioritized sensor to obtain a second processed image; and
stitch the first processed image to the second processed image to obtain a composite image.

### Claim 4 (depends on 3)

The system of claim 3, wherein the composite image is a spherical image.

### Claim 5 (depends on 1)

The system of claim 1, wherein the prioritized sensor and the second image sensor are positioned back-to-back, and further comprising:
a first hyper-hemispherical lens positioned to cover the prioritized sensor; and
a second hyper-hemispherical lens positioned to cover the second image sensor.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processing apparatus is configured to:
receive a user input signal identifying the prioritized sensor; and
select the prioritized sensor from among the array of two or more image sensors based on the user input signal.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processing apparatus is configured to:
detect a face in an image captured using the prioritized sensor; and
select the prioritized sensor from among the array of two or more image sensors based on detection of the face.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processing apparatus is configured to:
track an object appearing in images captured using the array of two or more image sensors; and
select the prioritized sensor from among the array of two or more image sensors based on appearance of the object in an image captured using the prioritized sensor.

### Claim 9 (depends on 1)

The system of claim 1, comprising an array of two or more microphones that is attached to the array of two or more image sensors, and wherein the processing apparatus that is configured to:
determine a direction of arrival of an audio signal based on audio recordings captured using the array of two or more microphones; and
select the prioritized sensor from among the array of two or more image sensors based on correspondence between the direction of arrival of the audio signal and a field of view the prioritized sensor.

### Claim 10 (depends on 9)

The system of claim 9, wherein the audio signal is a human speech signal.

### Claim 11 (depends on 1)

The system of claim 1, wherein the one or more image processing parameters include parameters of an auto exposure algorithm, and the image processing applied includes auto exposure processing.

### Claim 12

(canceled)

### Claim 13 (depends on 1)

The system of claim 1, wherein the one or more image processing parameters include parameters of a global tone mapping algorithm, and the image processing applied includes global tone mapping processing.

### Claim 14

A method comprising:
selecting an image sensor as a prioritized sensor from among an array of two or more image sensors;
determining one or more image processing parameters based on one or more images captured using the prioritized sensor;
applying image processing using the one or more image processing parameters to images captured with each image sensor in the array of two or more image sensors to obtain respective processed images for the array of two or more image sensors; and
stitching the respective processed images for the array of two or more image sensors to obtain a composite image.

### Claim 15 (depends on 14)

The method of claim 14, wherein the one or more image processing parameters are determined without consideration of image data from image sensors other than the prioritized sensor.

### Claim at position 16

-17. (canceled)

### Claim 18 (depends on 14)

The method of claim 14, wherein selecting the prioritized sensor from among the array of two or more image sensors comprises:
detecting a face in an image captured using the prioritized sensor; and
selecting the prioritized sensor from among the array of two or more image sensors based on detection of the face.

### Claim 19 (depends on 14)

The method of claim 14, wherein selecting the prioritized sensor from among the array of two or more image sensors comprises:
tracking an object appearing in images captured using the array of two or more image sensors; and
selecting the prioritized sensor from among the array of two or more image sensors based on appearance of the object in an image captured using the prioritized sensor.

### Claim 20 (depends on 14)

The method of claim 14, wherein selecting the prioritized sensor from among the array of two or more image sensors comprises:
determining a direction of arrival of an audio signal based on audio recordings captured using an array of two or more microphones that is attached to the array of two or more image sensors; and
selecting the prioritized sensor from among the array of two or more image sensors based on correspondence between the direction of arrival of the audio signal and a field of view the prioritized sensor.

### Claim at position 20

-24. (canceled)

### Claim 25

An image capture device, comprising:
a first image sensor configured to capture images;
a second image sensor configured to capture images;
a memory that stores instructions for producing spherical images based on images captured using the first image sensor and the second image sensor; and
a processor that executes the instructions to:
select, from among the first image sensor and the second image sensor, an image sensor as a prioritized sensor and the other image sensor as a deprioritized sensor;
determine an average global luminance value for a spherical image to be produced using a first image captured using the first image sensor and a second image captured using the second image sensor, wherein the average global luminance value is determined as a weighted average using weights determined based on the selection of the prioritized sensor to weight pixel values captured using the prioritized sensor more heavily than pixel values captured using the deprioritized sensor;
determine luminance values for each of the first image and the second image;
determine delta luminance values for each of the first image and the second image based on the average global luminance value and the luminance values;
update the first image and the second image using the delta luminance values; and
produce the spherical image based on the updated first image and the updated second image.

### Claim 26 (depends on 25)

The image capture device of claim 25, wherein the instructions to determine the delta luminance values for each of the first image and the second image based on the average global luminance value and the luminance values include instructions to:
determine clipped luminance variance values for the delta luminance values based on a threshold value representative of a maximum luminance variance between the first image sensor and the second image sensor; and
produce smoothed delta luminance values by applying temporal smoothing against the clipped luminance variance values.

### Claim 27 (depends on 25)

The image capture device of claim 25, wherein the instructions to update the first image and the second image using the delta luminance values include instructions to:
determine constrained luminance values for the first image and the second image based on the average global luminance value and the smoothed delta luminance values; and
limit the update to the first image and the second image based on the constrained luminance values.
