# US 2021/0344846 A1 · Image processing apparatus and image processing method, and image capturing apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0344846 A1
- Document ID: US20210344846A1
- Status: Published application
- Kind: A1
- Application number: 17/367,657
- Filed: 2021-07-06
- Published: 2021-11-04
- Earliest claimed: 2019-03-06
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Naoko Watazawa
- Web page: https://patentgrep.com/patent/US20210344846A1

## CPC

- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06F 18/211: Selection of the most significant subset of features
- G06T 7/50: Depth or shape recovery
- G06V 10/141: Control of illumination
- G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 40/166: {using acquisition arrangements}
- H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability}
- H04N 9/74: for obtaining special effects (H04N9/65 - H04N9/73 take precedence)
- H04N 23/10: for generating image signals from different wavelengths
- H04N 23/611: where the recognised objects include parts of the human body
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/76: by influencing the image signals
- G06T 2207/10024: Color image
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing apparatus comprises: an acquisition unit that acquires a first image obtained through shooting and distance information of the first image; a detection unit that detects a main subject from the first image; an extraction unit that extracts another subject from the first image based on the distance information of the main subject; a setting unit that sets parameters of one or more virtual light sources that emit virtual light to the main subject and the extracted other subject; and a processing unit that generates from the first image a second image in which the main subject and the other subject are illuminated with the virtual light using the parameters set by the setting unit.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

An image processing apparatus comprising:
an acquisition unit that acquires (a) a first image obtained through shooting and (b) distance information of the first image, wherein the distance information includes information on a distance to a subject in the first image in a depth direction;
a detection unit that detects a main subject from the first image;
an extraction unit that extracts another subject that exists in a predetermined region of the first image, the predetermined region being within a predetermined distance from the main subject, based on the distance information of the main subject;
a setting unit that sets parameters of one or more virtual light sources that emit virtual light to the main subject and the another subject; and
a processing unit that generates, from the first image, a second image in which the main subject and the another subject are illuminated with the virtual light using the parameters set by the setting unit,
wherein each unit is implemented by one or more processors, circuitry, or a combination thereof.

### Claim 22 (depends on 21)

The image processing apparatus according to claim 21, wherein the extraction unit extracts, based on the distance information of the main subject, the another subject that exists within a predetermined range of a distance in the depth direction from a distance to the main subject in the depth direction.

### Claim 23 (depends on 21)

The image processing apparatus according to claim 21, wherein in a case where no other subject is extracted by the extraction unit, (a) the setting unit sets the parameters of the one or more virtual light sources that emit the virtual light to the main subject, and (b) the processing unit generates the second image in which the main subject is illuminated with the virtual light.

### Claim 24 (depends on 21)

The image processing apparatus according to claim 21, wherein the processing unit processes the first image based on (a) shapes of the main subject and the another subject, (b) the distance information, and (c) the parameters of the one or more virtual light sources to generate the second image in which the main subject and the another subject are illuminated with the virtual light by the one or more virtual light sources.

### Claim 25 (depends on 21)

The image processing apparatus according to claim 21, wherein, in a case where a plurality of main subjects are detected by the detection unit, (a) the extraction unit extracts the another subject based on the distance information of each of the plurality of the main subjects, (b) the setting unit sets the parameters of the one or more virtual light sources that emit the virtual light to the main subject and the another subject with respect to each of the main subjects, and (c) the processing unit combines effects of the virtual light using the parameters on each of the main subjects.

### Claim 26 (depends on 21)

The image processing apparatus according to claim 21, further comprising a selection unit that selects at least one of the main subject and the another subject,
wherein the setting unit sets the parameters of one or more virtual light sources that emit virtual light to at least one of the main subject and the another subject as selected by the selection unit, and
wherein the selection unit is implemented by one or more processors, circuitry, or a combination thereof.

### Claim 27 (depends on 26)

The image processing apparatus according to claim 26, further comprising:
a control unit that controls to superimpose a display indicating the main subject and the another subject on the first image on a display device; and
an operation unit used for an operation to select the main subject and an operation to select the another subject, based on the display indicating the main subject and the another subject displayed on the display device,
wherein the selection unit selects at least one of the main subject and the another subject in accordance with an operation of the operation unit, and
wherein each of the control unit and the operation unit is implemented by one or more processors, circuitry, or a combination thereof.

### Claim 28 (depends on 21)

The image processing apparatus according to claim 21, wherein the parameters of the one or more virtual light sources include (a) a position of each virtual light source, and (b) an illumination area and intensity of virtual light.

### Claim 29 (depends on 21)

The image processing apparatus according to claim 21, wherein the parameters of the one or more virtual light sources include parameters of virtual light of one of a plurality of types including (a) addition light that increases brightness of a subject, (b) subtraction light that darkens a subject, and (c) specular reflection light.

### Claim 30 (depends on 21)

The image processing apparatus according to claim 21, further comprising a second detection unit that detects a subject using a method different from a method used by the detection unit,
wherein the extraction unit extracts the another subject based on (a) the distance information of the main subject and (b) distance information of the subject detected by the second detection unit, and
wherein the second detection unit is implemented by one or more processors, circuitry, or a combination thereof.

### Claim 31

An image processing method comprising:
acquiring (a) a first image obtained through shooting and (b) distance information of the first image, wherein the distance information includes information on a distance to a subject in the first image in a depth direction;
detecting a main subject from the first image;
extracting another subject that exists in a predetermined region of the first image, the predetermined region being within a predetermined distance from the main subject, based on the distance information of the main subject;
setting parameters of one or more virtual light sources that emit virtual light to the main subject and the another subject; and
generating, from the first image, a second image in which the main subject and the another subject are illuminated with the virtual light using the parameters set by the setting.

### Claim 32

A non-transitory computer-readable storage medium, the storage medium storing a program that is executable by a computer, wherein the program includes program code for causing the computer to function as an image processing apparatus comprising:
an acquisition unit that acquires (a) a first image obtained through shooting and (b) distance information of the first image, wherein the distance information includes information on a distance to a subject in the first image in a depth direction;
a detection unit that detects a main subject from the first image;
an extraction unit that extracts another subject that exists in a predetermined region of the first image, the predetermined region being within a predetermined distance from the main subject, based on the distance information of the main subject;
a setting unit that sets parameters of one or more virtual light sources that emit virtual light to the main subject and the another subject; and
a processing unit that generates, from the first image, a second image in which the main subject and the another subject are illuminated with the virtual light using the parameters set by the setting unit.
