# US 2019/0356832 A1 · Image processing apparatus, image processing method, image capturing apparatus, and storage medium for panoramic image composition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0356832 A1
- Document ID: US20190356832A1
- Status: Published application
- Kind: A1
- Application number: 16/404,328
- Filed: 2019-05-06
- Published: 2019-11-21
- Earliest claimed: 2018-05-17
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Yoichi Takahashi
- Web page: https://patentgrep.com/patent/US20190356832A1

## CPC

- H04N 23/60: Control of cameras or camera modules
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/73: by influencing the exposure time
- H04N 23/95: Computational photography systems, e.g. light-field imaging systems
- H04N 25/57: Control of the dynamic range
- H04N 25/771: comprising storage means other than floating diffusion
- H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing apparatus performs first calculation configured to calculate a conversion coefficient, with respect to images having a same exposure, with respect to a plurality of images, second calculation configured to calculate a conversion coefficient of images having different exposures, positional adjustment configured to execute positional adjustment with respect to the plurality of images by using the conversion coefficient calculated in the second calculation, and composition configured to execute composition with respect to the images to generate a composite image having a dynamic range and a viewing angle wider than the images, and in the second calculation an image used for detection with respect to a part of images from among the plurality of images is newly generated.

## Claims

### Claim 1

An image processing apparatus comprising:
at least one memory configured to store instructions; and
at least one processor in communication with the at least one memory, configured to execute the instructions to:
execute first calculation in which a conversion coefficient for executing positional adjustment of images captured at a same exposure is calculated with respect to a plurality of images captured at different exposures;
execute second calculation different from the first calculation, in which a conversion coefficient for executing positional adjustment of images captured at different exposures is calculated with respect to the plurality of images;
execute positional adjustment with respect to the plurality of images; and
execute composition with respect to the images on which the positional adjustment is executed to composite a panoramic image having a dynamic range and a viewing angle wider than a dynamic range and a viewing angle of the image,
wherein, the at least one processor further executes the instructions to, in the second calculation, when calculating a conversion coefficient with respect to an image in a part of images from the plurality of images, newly generate an image used for detection having an exposure that is the same as an exposure of an image that is to be positionally adjusted to the image in the part of images by using an exposure of the image, and calculate a conversion coefficient by using the image used for detection, and to calculate a conversion coefficient for executing positional adjustment with respect to images other than the part of images from the plurality of images based on the conversion coefficient calculated in the first calculation.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein the plurality of images is captured through a panning operation.

### Claim 3 (depends on 1)

The image processing apparatus according to claim 1, wherein the at least one processor further executes the instruction to composite a plurality of composite images having a dynamic range wider than the dynamic range of the image to composite the panoramic image having a viewing angle wider than a viewing angle of the composite image by using the plurality of composite images.

### Claim 4 (depends on 1)

The image processing apparatus according to claim 1, wherein, in the second calculation, the at least one processor further executes the instruction to calculate a conversion coefficient for executing positional adjustment of images other than the part of images from the plurality of images by using the conversion coefficient calculated by using the image used for detection, with respect to the image other than the part of images from the plurality of images.

### Claim 5 (depends on 1)

The image processing apparatus according to claim 1, wherein the part of images at least include images captured in a predetermined order from the plurality of images.

### Claim 6 (depends on 1)

The image processing apparatus according to claim 1, wherein, the part of images at least includes an image from which the conversion coefficient cannot be calculated in the first calculation, from the plurality of images.

### Claim 7 (depends on 6)

The image processing apparatus according to claim 6,
wherein the at least one processor further executes the instruction to calculate the conversion coefficient by detecting a motion vector from the image in the first calculation, and
wherein the image from which the conversion coefficient cannot be calculated is an image from which motion vectors of a number less than a predetermined minimum number are detected in the first calculation.

### Claim 8 (depends on 7)

The image processing apparatus according to claim 7, wherein the conversion coefficient is an affine coefficient, and the predetermined minimum number is three.

### Claim 9 (depends on 1)

The image processing apparatus according to claim 1, wherein the plurality of images is captured at the exposures which rotationally are changed.

### Claim 10 (depends on 1)

The image processing apparatus according to claim 1, wherein the at least one processor further executes the instruction to executes the positional adjustment with respect to each two images from the plurality of images by using the conversion coefficient for executing positional adjustment calculated in the second calculation.

### Claim 11

An image pickup apparatus, comprising:
an image sensor configured to pick up a plurality of images;
at least one memory configured to store instructions; and
at least one processor in communication with the at least one memory, configured to execute the instructions to:
execute first calculation in which a conversion coefficient for executing positional adjustment of images captured at a same exposure is calculated with respect to a plurality of images captured at different exposures;
execute second calculation different from the first calculation, in which a conversion coefficient for executing positional adjustment of images captured at different exposures is calculated, with respect to the plurality of images;
execute positional adjustment with respect to the plurality of images; and
execute composition with respect to the images on which the positional adjustment is executed to composite a panoramic image having a dynamic range and a viewing angle wider than a dynamic range and a viewing angle of the image,
wherein, the at least one processor further executes the instructions to, in the second calculation, when calculating a conversion coefficient with respect to an image in a part of images from the plurality of images, newly generate an image used for detection having an exposure that is the same as an exposure of an image that is to be positionally adjusted to the image in the part of images by using an exposure of the image and calculate a conversion coefficient by using the image used for detection, and to calculate a conversion coefficient for executing positional adjustment based on the conversion coefficient calculated in the first calculation with respect to images other than the part of images from the plurality of images.

### Claim 12

A control method to control an image processing apparatus, the method comprising:
executing first calculation in which a conversion coefficient for executing positional adjustment of images captured at a same exposure is calculated with respect to a plurality of images captured at different exposures;
executing second calculation different from the first calculation, in which a conversion coefficient for executing positional adjustment of images captured at different exposures is calculated with respect to the plurality of images;
executing positional adjustment with respect to the plurality of images; and
executing composition with respect to the images on which the positional adjustment is executed to composite a panoramic image having a dynamic range and a viewing angle wider than a dynamic range and a viewing angle of the image,
wherein, in the second calculation, when calculating a conversion coefficient with respect to an image in a part of images from the plurality of images, an image used for detection having an exposure that is the same as an exposure of an image that is to be positionally adjusted to the image is newly generated in the part of images by using an exposure of the image and calculate a conversion coefficient by using the image used for detection, and a conversion coefficient is calculated for executing positional adjustment based on the conversion coefficient calculated in the first calculation with respect to images other than the part of images from the plurality of images.

### Claim 13

A computer-readable storage medium storing instructions that cause a computer to execute a method for controlling an image processing apparatus, the method comprising:
executing first calculation in which a conversion coefficient for executing positional adjustment of images captured at a same exposure is calculated with respect to a plurality of images captured at different exposures;
executing second calculation different from the first calculation, in which a conversion coefficient for executing positional adjustment of images captured at different exposures is calculated with respect to the plurality of images;
executing positional adjustment with respect to the plurality of images; and
executing composition with respect to the images on which the positional adjustment is executed to composite a panoramic image having a dynamic range and a viewing angle wider than a dynamic range and a viewing angle of the image,
wherein, in the second calculation, when calculating a conversion coefficient with respect to an image in a part of images from the plurality of images, an image used for detection having an exposure that is the same as an exposure of an image that is to be positionally adjusted to the image is newly generated in the part of images by using an exposure of the image and calculate a conversion coefficient by using the image used for detection, and a conversion coefficient is calculated for executing positional adjustment based on the conversion coefficient calculated in the first calculation with respect to images other than the part of images from the plurality of images.
