# US 2026/0017874 A1 · Information processing system, operation method of information processing system, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0017874 A1
- Document ID: US20260017874A1
- Status: Published application
- Kind: A1
- Application number: 18/994,621
- Filed: 2023-07-21
- Published: 2026-01-15
- Earliest claimed: 2022-08-04
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Kendai FURUKAWA
- Web page: https://patentgrep.com/patent/US20260017874A1

## CPC

- G06T 15/20: Perspective computation
- G06T 7/90: Determination of colour characteristics
- G06T 15/205: {Image-based rendering}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

There is provided an information processing system capable of easily correcting a failure occurring in a virtual viewpoint image (free viewpoint image, volumetric image), an operation method of the information processing system, and a program. A virtual viewpoint image is generated by synthesizing on the basis of a color synthesis weight set for each of multi-viewpoint images, an input of a failure region of the virtual viewpoint image is received, a correction input of a color synthesis weight of the failure region of each of the multi-viewpoint images used for synthesis is received, the color synthesis weight is corrected to a plurality of the color synthesis weights based on the correction input, a plurality of the virtual viewpoint images is re-synthesized as a correction candidate image on the basis of the plurality of color synthesis weights, the color synthesis weights applied in a correction candidate image selected from a plurality of the correction candidate images as the failure is regarded as being corrected are determined as correction information for correcting the failure. The present disclosure can be applied to a generation device of a volumetric image.

## Claims

### Claim 1

An information processing system comprising:
a virtual viewpoint image generation unit that generates a virtual viewpoint image by synthesizing multi-viewpoint images on a basis of a color synthesis weight set for each of the multi-viewpoint images according to a virtual viewpoint position;
a failure region acquisition unit that receives an input of a failure region designated by a user as a region in which a failure occurs in the virtual viewpoint image;
a re-synthesis unit that receives a correction input of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image, corrects the color synthesis weight to a plurality of the color synthesis weights based on the correction input, and re-synthesizes a plurality of the virtual viewpoint images as a correction candidate image on a basis of the plurality of color synthesis weights; and
a correction information determination unit that receives selection information of the correction candidate image selected from a plurality of the correction candidate images as the failure is regarded as being corrected, and on a basis of the selection information, determines the color synthesis weights applied to the selected correction candidate image as correction information for correcting the failure.

### Claim 2 (depends on 1)

The information processing system according to claim 1, further comprising
a color synthesis weight visualization unit that generates weight visualization information visualizing a magnitude of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image,
wherein the re-synthesis unit receives a correction input, using the weight visualization information, of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image.

### Claim 3 (depends on 2)

The information processing system according to claim 2, wherein
the color synthesis weight visualization unit generates a weight map visualizing a magnitude of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image, and
the re-synthesis unit receives a correction input, using the weight map, of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image.

### Claim 4 (depends on 3)

The information processing system according to claim 3, wherein
the color synthesis weight visualization unit generates the weight map visualized with at least one of a color and a pattern according to a magnitude of the color synthesis weight of the failure region, and
the re-synthesis unit receives a correction input associated with correction of at least one of the color and the pattern of the weight map.

### Claim 5 (depends on 2)

The information processing system according to claim 2, wherein
the color synthesis weight visualization unit visualizes, with a slidack, a magnitude of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image, and
the re-synthesis unit receives a correction input, using the slidack, of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image.

### Claim 6 (depends on 1)

The information processing system according to claim 1, further comprising
a correction information propagation unit that stores the correction information determined by the correction information determination unit,
wherein when the correction information is stored in the correction information propagation unit, the virtual viewpoint image generation unit generates a virtual viewpoint image by synthesizing the multi-viewpoint images using the color synthesis weights as the correction information.

### Claim 7 (depends on 1)

The information processing system according to claim 1, wherein
the multi-viewpoint images are generated on a basis of a shape and texture data of a three-dimensional model, and
the virtual viewpoint image generation unit performs, to generate a virtual viewpoint image in time series, tracking of a position of a subject in the failure region in which the correction input is made on a basis of the shape and the texture data of the three-dimensional model supplied in time series to generate the multi-viewpoint images, and generates a virtual viewpoint image by synthesizing the multi-viewpoint images using the color synthesis weights as the correction information according to a tracked position of the subject.

### Claim 8 (depends on 7)

The information processing system according to claim 7, wherein
the virtual viewpoint image generation unit specifies a type of the failure according to the tracked position of the subject, and generates a virtual viewpoint image by synthesizing the multi-viewpoint images using the color synthesis weights as the correction information according to the type of the failure.

### Claim 9 (depends on 8)

The information processing system according to claim 8, wherein
the type of the failure includes a failure caused by low quality of the multi-viewpoint images having the large color synthesis weights, a failure in which a background is reflected in a portion of a foreground, and a failure in which a foreground is reflected in a portion of a background.

### Claim 10 (depends on 7)

The information processing system according to claim 7, wherein
the tracking includes Mesh-tracking.

### Claim 11 (depends on 7)

The information processing system according to claim 7, further comprising
a correction case learning unit that acquires the shape and the texture data of the three-dimensional model and the correction information as correction cases, and learns learning information for realizing detecting a failure region of a virtual viewpoint image generated by synthesizing multi-viewpoint images generated from the shape and texture data of the three-dimensional model and correction of the detected failure region by learning using the correction cases.

### Claim 12 (depends on 11)

The information processing system according to claim 11, further comprising
a learning correction unit that detects the failure region on a basis of the learning information for a virtual viewpoint image generated by synthesizing the multi-viewpoint images and generated by the virtual viewpoint image generation unit, and corrects the detected failure region.

### Claim 13 (depends on 1)

The information processing system according to claim 1, further comprising
a quality improvement unit that improves quality of the multi-viewpoint images when the multi-viewpoint images used to generate the virtual viewpoint image have quality lower than a predetermined level of quality.

### Claim 14

An operation method of an information processing system, the operation method comprising the steps of:
generating a virtual viewpoint image by synthesizing multi-viewpoint images on a basis of a color synthesis weight set for each of the multi-viewpoint images according to a virtual viewpoint position;
receiving an input of a failure region designated by a user as a region in which a failure occurs in the virtual viewpoint image;
receiving a correction input of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image, correcting the color synthesis weight to a plurality of the color synthesis weights based on the correction input, and re-synthesizing a plurality of the virtual viewpoint images as a correction candidate image on a basis of the plurality of color synthesis weights; and
receiving selection information of the correction candidate image selected from a plurality of the correction candidate images as the failure is regarded as being corrected, and on a basis of the selection information, determining the color synthesis weights applied to the selected correction candidate image as correction information for correcting the failure.

### Claim 15

A program for causing a computer to function as:
a virtual viewpoint image generation unit that generates a virtual viewpoint image by synthesizing multi-viewpoint images on a basis of a color synthesis weight set for each of the multi-viewpoint images according to a virtual viewpoint position;
a failure region acquisition unit that receives an input of a failure region designated by a user as a region in which a failure occurs in the virtual viewpoint image;
a re-synthesis unit that receives a correction input of the color synthesis weight of the failure region in each of the multi-viewpoint images used for synthesis of the virtual viewpoint image, corrects the color synthesis weight to a plurality of the color synthesis weights based on the correction input, and re-synthesizes a plurality of the virtual viewpoint images as a correction candidate image on a basis of the plurality of color synthesis weights; and
a correction information determination unit that receives selection information of the correction candidate image selected from a plurality of the correction candidate images as the failure is regarded as being corrected, and on a basis of the selection information, determines the color synthesis weights applied to the selected correction candidate image as correction information for correcting the failure.
