# US 2026/0094433 A1 · Image processing system, image processing method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094433 A1
- Document ID: US20260094433A1
- Status: Published application
- Kind: A1
- Application number: 19/414,003
- Filed: 2025-12-09
- Published: 2026-04-02
- Earliest claimed: 2023-06-26
- Assignees: Sony Group Corporation; Sony Interactive Entertainment Inc.
- Applicants: Sony Group Corporation; Sony Interactive Entertainment Inc.
- Inventors: Toshinori Ihara; Hirotaka Asayama; Hisashi Kobiki; Ryota Ito; Kenichiro Hosokawa; Shoichi Ikenboue; Takafumi Morifuji; Kaoru Saso; Takuro Kawai
- Web page: https://patentgrep.com/patent/US20260094433A1

## CPC

- G06V 10/993: {Evaluation of the quality of the acquired pattern}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/73: Deblurring; Sharpening
- G06T 13/20: Three-dimensional [3D] animation
- G06T 15/00: Three-dimensional [3D] image rendering
- G06V 10/82: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques include acquiring 1st to Nth input frames (N is a natural number equal to or greater than 2) having a prescribed input pixel number. The techniques further include acquiring, based on each of the input frames, 1st to Nth intermediate frames by generating an intermediate frame for each input frame which corresponds to the input frame and which includes an intermediate pixel number equal to or greater than the input pixel number. The techniques further include inputting each of the intermediate frames to a machine learning model. The techniques further include acquiring 1st to Nth estimation frames including an estimated pixel number equal to or greater than the intermediate pixel number which is greater than the input pixel number.

## Claims

### Claim 1

An image processing system comprising:
one or more storage media storing instructions; and
one or more processors configured to execute the instructions to cause the image processing system to:
acquire 1st to Nth input frames (N is a natural number equal to or greater than 2) having a prescribed input pixel number;
acquire, based on each of the input frames, 1st to Nth intermediate frames by generating an intermediate frame for each input frame which corresponds to the input frame and which includes an intermediate pixel number equal to or greater than the input pixel number; and
input each of the intermediate frames to a machine learning model; and
acquire 1st to Nth estimation frames including an estimated pixel number equal to or greater than the intermediate pixel number which is greater than the input pixel number, wherein the machine learning model includes:
a cumulative feature information output layer to which the nth intermediate frame (n=2, 3, . . . , N) and (n−1)th auxiliary information based on (n−1)th cumulative feature information indicating the features of the 1st to (n−1)th intermediate frames are inputted, wherein the cumulative feature information output layer outputs nth cumulative feature information indicating the features of the 1st to nth intermediate frames; and
an estimation frame output layer to which the nth cumulative feature information is inputted, wherein the estimation frame output layer outputs the nth estimation frame, wherein the machine learning model was trained using a plurality of training data including:
a learning intermediate frame including the intermediate pixel number generated based on a learning input frame having the input pixel number; and
a learning estimation frame including the estimated pixel number.

### Claim 2 (depends on 1)

The image processing system of claim 1, wherein each of the input frames includes an image obtained by executing rendering of three-dimensional data indicating one or more objects as seen from a prescribed viewpoint.

### Claim 3 (depends on 2)

The image processing system of claim 2, wherein each of the input frames includes an image obtained by executing the rendering so that the viewpoint changes for each of the input frames, and wherein the instructions further cause the image processing system to:
acquire change information including information relating to a change of the viewpoint for each of the input frames in the rendering; and
obtain a pixel value of a position corresponding to each pixel before the change by interpolation in the input frame, based on the change information and each pixel of each of the input frames; and
generate each of the intermediate frames.

### Claim 4 (depends on 2)

The image processing system of claim 2, wherein the instructions further cause the image processing system to:
acquire the (n−1)th motion information including information indicating an amount and a direction of motion from the (n−1)th input frame towards the nth input frame; and
acquire the (n−1)th auxiliary information by applying motion compensation on the (n−1)th cumulative feature information, based on the (n−1)th motion information.

### Claim 5 (depends on 4)

The image processing system of claim 4, wherein the instructions further cause the image processing system to:
acquire (n−1)th depth information indicating each pixel depth of the (n−1)th input frame, and nth depth information indicating each pixel depth of the nth input frame;
specify, amongst the nth intermediate frame pixels, an nth appearance pixel as a fully or partially displayed pixel of the object which is not displayed in the (n−1)th intermediate frame, based on the (n−1)th depth information and the nth depth information; and
acquire the (n−1)th auxiliary information by converting a pixel value of the nth appearance pixel in the (n−1)th cumulative feature information to a prescribed value.

### Claim 6 (depends on 1)

The image processing system of claim 1, wherein the 1st intermediate frame and a given auxiliary information are input to the cumulative feature information output layer which outputs the 1st cumulative feature information.

### Claim 7 (depends on 1)

The image processing system of claim 1, wherein the cumulative feature information includes image information having the same pixel number as the intermediate pixel number.

### Claim 8

A method comprising:
acquiring 1st to Nth input frames (N is a natural number equal to or greater than 2) having a prescribed input pixel number;
acquiring, based on each of the input frames, 1st to Nth intermediate frames by generating an intermediate frame for each input frame which corresponds to the input frame and which includes an intermediate pixel number equal to or greater than the input pixel number; and
inputting each of the intermediate frames to a machine learning model; and
acquiring 1st to Nth estimation frames including an estimated pixel number equal to or greater than the intermediate pixel number which is greater than the input pixel number, wherein the machine learning model includes:
a cumulative feature information output layer to which the nth intermediate frame (n=2, 3, . . . , N) and (n−1)th auxiliary information based on (n−1)th cumulative feature information indicating the features of the 1st to (n−1)th intermediate frames are inputted, wherein the cumulative feature information output layer outputs nth cumulative feature information indicating the features of the 1st to nth intermediate frames; and
an estimation frame output layer to which the nth cumulative feature information is inputted, wherein the estimation frame output layer outputs the nth estimation frame, wherein the machine learning model was trained using a plurality of training data including:
a learning intermediate frame including the intermediate pixel number generated based on a learning input frame having the input pixel number; and
a learning estimation frame including the estimated pixel number.

### Claim 9 (depends on 8)

The method of claim 8, wherein each of the input frames includes an image obtained by executing rendering of three-dimensional data indicating one or more objects as seen from a prescribed viewpoint.

### Claim 10 (depends on 9)

The method of claim 9, wherein each of the input frames includes an image obtained by executing the rendering so that the viewpoint changes for each of the input frames, and wherein the method further comprises:
acquiring change information including information relating to a change of the viewpoint for each of the input frames in the rendering; and
obtaining a pixel value of a position corresponding to each pixel before the change by interpolation in the input frame, based on the change information and each pixel of each of the input frames; and
generating each of the intermediate frames.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
acquiring the (n−1)th motion information including information indicating an amount and a direction of motion from the (n−1)th input frame towards the nth input frame; and
acquiring the (n−1)th auxiliary information by applying motion compensation on the (n−1)th cumulative feature information, based on the (n−1)th motion information.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
acquiring (n−1)th depth information indicating each pixel depth of the (n−1)th input frame, and nth depth information indicating each pixel depth of the nth input frame;
specifying, amongst the nth intermediate frame pixels, an nth appearance pixel as a fully or partially displayed pixel of the object which is not displayed in the (n−1)th intermediate frame, based on the (n−1)th depth information and the nth depth information; and
acquiring the (n−1)th auxiliary information by converting a pixel value of the nth appearance pixel in the (n−1)th cumulative feature information to a prescribed value.

### Claim 13 (depends on 8)

The method of claim 8, wherein the 1st intermediate frame and a given auxiliary information are input to the cumulative feature information output layer which outputs the 1st cumulative feature information.

### Claim 14 (depends on 8)

The method of claim 8, wherein the cumulative feature information includes image information having the same pixel number as the intermediate pixel number.

### Claim 15

One or more non-transitory computer-readable storage media storing instructions that, upon execution by one or more processors of a system, cause the system to perform operations comprising:
acquiring 1st to Nth input frames (N is a natural number equal to or greater than 2) having a prescribed input pixel number;
acquiring, based on each of the input frames, 1st to Nth intermediate frames by generating an intermediate frame for each input frame which corresponds to the input frame and which includes an intermediate pixel number equal to or greater than the input pixel number; and
inputting each of the intermediate frames to a machine learning model; and
acquiring 1st to Nth estimation frames including an estimated pixel number equal to or greater than the intermediate pixel number which is greater than the input pixel number, wherein the machine learning model includes:
a cumulative feature information output layer to which the nth intermediate frame (n=2, 3, . . . , N) and (n−1)th auxiliary information based on (n−1)th cumulative feature information indicating the features of the 1st to (n−1)th intermediate frames are inputted, wherein the cumulative feature information output layer outputs nth cumulative feature information indicating the features of the 1st to nth intermediate frames; and
an estimation frame output layer to which the nth cumulative feature information is inputted, wherein the estimation frame output layer outputs the nth estimation frame, wherein the machine learning model was trained using a plurality of training data including:
a learning intermediate frame including the intermediate pixel number generated based on a learning input frame having the input pixel number; and
a learning estimation frame including the estimated pixel number.

### Claim 16 (depends on 15)

The computer-readable storage media of claim 15, wherein each of the input frames includes an image obtained by executing rendering of three-dimensional data indicating one or more objects as seen from a prescribed viewpoint.

### Claim 17 (depends on 16)

The computer-readable storage media of claim 16, wherein each of the input frames includes an image obtained by executing the rendering so that the viewpoint changes for each of the input frames, and wherein the operations further comprise:
acquiring change information including information relating to a change of the viewpoint for each of the input frames in the rendering; and
obtaining a pixel value of a position corresponding to each pixel before the change by interpolation in the input frame, based on the change information and each pixel of each of the input frames; and
generating each of the intermediate frames.

### Claim 18 (depends on 16)

The computer-readable storage media of claim 16, wherein the operations further comprise:
acquiring the (n−1)th motion information including information indicating an amount and a direction of motion from the (n−1)th input frame towards the nth input frame; and
acquiring the (n−1)th auxiliary information by applying motion compensation on the (n−1)th cumulative feature information, based on the (n−1)th motion information.

### Claim 19 (depends on 15)

The computer-readable storage media of claim 15, wherein the 1st intermediate frame and a given auxiliary information are input to the cumulative feature information output layer which outputs the 1st cumulative feature information.

### Claim 20 (depends on 15)

The computer-readable storage media of claim 15, wherein the cumulative feature information includes image information having the same pixel number as the intermediate pixel number.
