# US 2019/0378326 A1 · Image processing apparatus, method and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378326 A1
- Document ID: US20190378326A1
- Status: Published application
- Kind: A1
- Application number: 16/421,576
- Filed: 2019-05-24
- Published: 2019-12-12
- Earliest claimed: 2018-06-11
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Hironao Ito
- Web page: https://patentgrep.com/patent/US20190378326A1

## CPC

- G06T 15/10: Geometric effects
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 15/04: Texture mapping
- G06T 15/20: Perspective computation
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597)
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- G06T 2207/10024: Color image
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In an image processing apparatus, three-dimensional shape data indicating a shape of an object is generated based on at least one of a plurality of images captured from different directions by a plurality image capturing devices. Color information indicating a color of each element of the three-dimensional shape data is generated based on the generated three-dimensional shape data and the plurality of images. The three-dimensional shape data and the color information are associated with each other and written to a storage unit as material data for generating a virtual viewpoint image corresponding to virtual viewpoint information indicating a virtual viewpoint.

## Claims

### Claim 1

An image processing apparatus comprising:
a first generation unit configured to generate three-dimensional shape data indicating a shape of an object based on at least one of a plurality of images captured from different directions by a plurality image capturing devices;
a second generation unit configured to generate color information indicating a color of each element of the three-dimensional shape data based on the three-dimensional shape data generated by the first generation unit and the plurality of images; and
a write unit configured to write the three-dimensional shape data and the color information to a storage unit in association with each other as material data for generating a virtual viewpoint image corresponding to virtual viewpoint information indicating a virtual viewpoint.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein
the second generation unit generates a pixel value of a pixel of the image, which corresponds to an element of the three-dimensional shape data, as the color information.

### Claim 3 (depends on 2)

The image processing apparatus according to claim 2, wherein
the second generation unit generates the color information from an image of the image capturing device selected based on a distance between an element of the object and the image capturing device having captured an image including the element of the object.

### Claim 4 (depends on 3)

The image processing apparatus according to claim 3, wherein
the second generation unit generates the color information from an image of the image capturing device selected from the two image capturing devices whose angles formed with an element of the object are large compared to others of the image capturing devices whose distance is the same.

### Claim 5 (depends on 4)

The image processing apparatus according to claim 4, wherein
the second generation unit generates the color information from an image of the image capturing device facing an element of the object.

### Claim 6 (depends on 3)

The image processing apparatus according to claim 3, wherein
the second generation unit generates, in a case where the two or more image capturing devices have the same distance, the color information from an image of the image capturing device whose priority given to the image capturing device is high.

### Claim 7 (depends on 2)

The image processing apparatus according to claim 2, wherein
the second generation unit generates, in a case where there is a plurality of pixels of the image corresponding to an element of the three-dimensional shape data, an average value of pixel values of the plurality of pixels as the color information.

### Claim 8 (depends on 1)

The image processing apparatus according to claim 1, comprising:
a third generation unit configured to generate color correction information for correcting a color of each element of the three-dimensional shape data based on the generated three-dimensional shape data and the plurality of images, wherein
the write unit writes the color correction information to the storage unit as the material data by associating the color correction information with the three-dimensional shape data and the color information.

### Claim 9 (depends on 8)

The image processing apparatus according to claim 8, wherein
the third generation unit generates the color correction information from an image of the image capturing device selected based on a distance between an element of the object and the image capturing device having captured an image including the element of the object.

### Claim 10 (depends on 9)

The image processing apparatus according to claim 9, wherein
the third generation unit generates the color correction information from an image of the selected image capturing device, with the two image capturing devices whose angles formed with an element of the object are large compared to others being selected from the image capturing devices whose distance is the same.

### Claim 11 (depends on 8)

The image processing apparatus according to claim 8, wherein
the third generation unit generates, in a case where there is a plurality of pixels of the image corresponding to an element of the object, an average of pixel values of the plurality of pixels as the color correction information.

### Claim 12 (depends on 1)

The image processing apparatus according to claim 1, wherein
the write unit does not write a foreground image generated from the plurality of images captured by the plurality of image capturing devices and including the object to the storage unit and writes a background image not including the object to the storage unit.

### Claim 13 (depends on 12)

The image processing apparatus according to claim 12, comprising:
a data acquisition unit configured to acquire the three-dimensional shape data, color information on the three-dimensional shape data, and a background image from the storage unit based on virtual viewpoint information indicating a virtual viewpoint;
a coloring unit configured to generate colored three-dimensional shape data by coloring each element of the acquired three-dimensional shape data based on the acquired color information on the three-dimensional shape data; and
a fourth generation unit configured to generate a virtual viewpoint image corresponding to the virtual viewpoint information by using the generated colored three-dimensional shape data and the background image.

### Claim 14 (depends on 13)

The image processing apparatus according to claim 13, wherein
the data acquisition unit acquires color correction information for correcting the color information included in the material data from the storage unit,
the image processing apparatus further includes a color correction unit configured to correct color information on the three-dimensional shape data based on the three-dimensional shape data, the color information, the color correction information, and the virtual viewpoint information, which are acquired, and
the coloring unit generates the colored three-dimensional shape data based on color information on the three-dimensional shape data corrected by the color correction unit.

### Claim 15 (depends on 13)

The image processing apparatus according to claim 13, wherein
weighting of the color information and the color correction information is changed based on the virtual viewpoint information.

### Claim 16 (depends on 15)

The image processing apparatus according to claim 15, wherein
in a case where the virtual viewpoint is in an area surrounded by the generation-target image capturing device of each of the color information and the color correction information and an element of the three-dimensional shape data, weighting of the color information and the color correction information is set in accordance with a magnitude of an angle formed by a line connecting the virtual viewpoint and the element of the three-dimensional shape data and an image capturing direction of the generation-target image capturing device of each of the color information and the color correction information.

### Claim 17 (depends on 15)

The image processing apparatus according to claim 15, wherein
the coloring unit generates the colored three-dimensional shape data by using the color correction information corresponding to the generation-target image capturing device of the color correction information, which is close to the virtual viewpoint, in a case where the virtual viewpoint is outside an area surrounded by the generation-target image capturing device of each of the color information and the color correction information and an element of the three-dimensional shape data.

### Claim 18

An image processing method comprising:
generating three-dimensional shape data indicating a shape of an object based on at least one of a plurality of images captured from different directions by a plurality image capturing devices;
generating color information indicating a color of each element of the three-dimensional shape data based on the generated three-dimensional shape data and the plurality of images; and
writing the three-dimensional shape data and the color information to a storage unit in association with each other as material data for generating a virtual viewpoint image corresponding to virtual viewpoint information indicating a virtual viewpoint.

### Claim 19

A non-transitory computer readable storage medium storing a program for causing a computer to perform an image processing method comprising:
generating three-dimensional shape data indicating a shape of an object based on at least one of a plurality of images captured from different directions by a plurality image capturing devices;
generating color information indicating a color of each element of the three-dimensional shape data based on the generated three-dimensional shape data and the plurality of images; and
writing the three-dimensional shape data and the color information to a storage unit in association with each other as material data for generating a virtual viewpoint image corresponding to virtual viewpoint information indicating a virtual viewpoint.
