# US 2020/0288100 A1 · Image reconstruction method, system, device and computer-readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0288100 A1
- Document ID: US20200288100A1
- Status: Published application
- Kind: A1
- Application number: 16/810,695
- Filed: 2020-03-05
- Published: 2020-09-10
- Earliest claimed: 2019-03-07
- Assignees: Alibaba Group Holding Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Xiaojie Sheng
- Web page: https://patentgrep.com/patent/US20200288100A1

## CPC

- G06T 15/205: {Image-based rendering}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 15/04: Texture mapping
- G06T 15/20: Perspective computation
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
- H04N 13/128: Adjusting depth or disparity
- H04N 13/139: Format conversion, e.g. of frame-rate or size
- H04N 13/156: Mixing image signals
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/172: image signals comprising non-image signal components, e.g. headers or format information
- H04N 13/189: Recording image signals; Reproducing recorded image signals
- H04N 13/268: based on depth image-based rendering [DIBR]
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/293: Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 13/349: Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking (for viewing without the aid of special glasses using fly-eye lenses H04N13/307)
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2210/36: Level of detail
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Image reconstruction methods, systems, devices, and computer-readable storage media are provided. The method includes: acquiring a multi-angle free-perspective image combination, parameter data of the image combination, and virtual viewpoint position information based on user interaction, where the image combination includes multiple groups of texture images and depth maps that are synchronized at multiple angles and have corresponding relationships; selecting a corresponding group of texture images and depth maps in the image combination at a user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment.

## Claims

### Claim 1

A method, comprising:
acquiring a multi-angle free-perspective image combination, parameter data of the image combination, and virtual viewpoint position information based on user interaction, wherein the image combination includes multiple groups of texture images and depth maps, and the multiple groups of texture images and depth maps are synchronized at multiple angles and have corresponding relationships;
selecting a corresponding group of texture images and depth maps in the image combination at a user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and
combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, to obtain a reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

### Claim 2 (depends on 1)

The method of claim 1, wherein selecting the corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the preset rule according to the virtual viewpoint position information and the parameter data of the image combination comprises:
selecting the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies a preset positional relationship and/or a preset quantitative relationship with the virtual viewpoint position according to the virtual viewpoint position information and the parameter data of the image combination.

### Claim 3 (depends on 2)

The method of claim 2, wherein selecting the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies the preset positional relationship and/or the preset quantitative relationship with the virtual viewpoint position according to the virtual viewpoint position information and the parameter data of the image combination comprises:
selecting a preset number of corresponding groups of texture images and the depth maps that are closest to the virtual viewpoint position in the image combination at the user interaction moment according to the virtual viewpoint position information and the parameter data of the image combination.

### Claim 4 (depends on 3)

The method of claim 3, wherein selecting the preset number of corresponding groups of texture images and the depth maps that are closest to the virtual viewpoint position in the image combination at the user interaction moment according to the virtual viewpoint position information and the parameter data of the image combination comprises:
selecting texture images and depth maps corresponding to 2 to N capturing devices closest to the virtual viewpoint position according to the virtual viewpoint position information and the parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, wherein N is the number of all capturing devices that capture the image combination.

### Claim 5 (depends on 1)

The method of claim 1, wherein combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment comprises:
performing forward projection on the selected depth maps of the corresponding group in the image combination at the user interaction moment respectively, to project to the virtual position at the user interaction moment;
performing post-processing on the depth maps after the forward projection respectively;
performing backward projection on the selected texture images of the corresponding group in the image combination at the user interaction moment respectively; and
fusing respective virtual texture images generated after the backward projection.

### Claim 6 (depends on 5)

The method of claim 5, wherein after fusing respective the virtual texture images generated after the backward projection, the method further comprises:
performing inpainting on the fused texture image to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

### Claim 7 (depends on 5)

The method of claim 5, wherein performing post-processing on the depth maps after the forward projection respectively comprises at least one of the following:
performing foreground padding processing on the depth maps after the forward projection respectively; and
performing pixel-level filtering processing on the depth maps after the forward projection respectively.

### Claim 8 (depends on 5)

The method of claim 5, wherein fusing respective virtual texture images generated after the backward projection comprises:
fusing the respective virtual texture images generated after the backward projection using a global weight determined by a distance between the virtual viewpoint position and a position of a capturing device that captures a corresponding texture image in the image combination, according to the virtual viewpoint position information and the parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment.

### Claim 9 (depends on 1)

The method of claim 1, wherein combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment comprises:
respectively projecting the depth maps of the corresponding group in the image combination at the user interaction moment to the virtual viewpoint position at the user interaction moment according to a spatial geometric relationship, to form a depth map of the virtual viewpoint position; and
copying from pixel points in the texture images of the corresponding group to the generated virtual texture images corresponding to the virtual viewpoint position according to the projected depth maps, to form the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment; and
fusing the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment, to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment.

### Claim 10 (depends on 9)

The method of claim 9, wherein fusing the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment comprises:
performing weighting processing on pixels in corresponding positions in the virtual texture images corresponding to respective corresponding groups in the image combination at the user interaction moment, to obtain pixel values of the corresponding positions in the reconstructed image of the virtual viewpoint position at the user interaction moment; and
for a first pixel where a pixel value is zero in the reconstructed image of the virtual viewpoint position at the user interaction moment, performing inpainting using pixels around the first pixel in the reconstructed image, to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment.

### Claim 11 (depends on 9)

The method of claim 9, wherein fusing the virtual texture images corresponding to the corresponding group in the image combination at the user interaction moment to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment comprises:
for a first pixel where a pixel value is zero in the virtual texture images corresponding to respective corresponding groups in the image combination at the user interaction moment, performing inpainting on the first pixel using around pixel values respectively; and
performing weighting processing on pixel values in corresponding positions in the virtual texture images corresponding to the respective corresponding groups after the inpainting, to obtain the reconstructed image of the virtual viewpoint position at the user interaction moment.

### Claim 12 (depends on 1)

The method of claim 1, wherein acquiring a multi-angle free-perspective image combination and parameter data of the image combination comprises:
decoding acquired compressed multi-angle free-perspective image data, to obtain the multi-angle free-perspective image combination and the parameter data corresponding to the image combination.

### Claim 13

A system, comprising:
an acquiring unit, configured to acquire multiple-angle free-perspective image combination, parameter data of the image combination, and the virtual viewpoint position information based on user interaction, wherein the image combination includes multiple groups of texture images and the depth maps, and the multiple groups of texture images and depth maps are synchronized at multiple angles and have corresponding relationships;
a selecting unit, configured to select a corresponding group of texture images and the depth maps in the image combination at the user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and
an image reconstruction unit, configured to combine and render the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, to obtain a reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

### Claim 14

A computer-readable storage medium having computer instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform acts comprising:
acquiring a multi-angle free-perspective image combination, parameter data of the image combination, and virtual viewpoint position information based on user interaction, wherein the image combination includes multiple groups of texture images and depth maps, and the multiple groups of texture images and depth maps are synchronized at multiple angles and have corresponding relationships;
selecting a corresponding group of texture images and depth maps in the image combination at a user interaction moment based on a preset rule according to the virtual viewpoint position information and the parameter data of the image combination; and
combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, to obtain a reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

### Claim 15 (depends on 14)

The computer-readable storage medium of claim 14, wherein selecting the corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the preset rule according to the virtual viewpoint position information and the parameter data of the image combination comprises:
selecting the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies a preset positional relationship and/or a preset quantitative relationship with the virtual viewpoint position according to the virtual viewpoint position information and the parameter data of the image combination.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein selecting the corresponding group of texture images and the depth maps in the image combination at the user interaction moment that satisfies the preset positional relationship and/or the preset quantitative relationship with the virtual viewpoint position according to the virtual viewpoint position information and the parameter data of the image combination comprises:
selecting a preset number of corresponding groups of texture images and the depth maps that are closest to the virtual viewpoint position in the image combination at the user interaction moment according to the virtual viewpoint position information and the parameter data of the image combination.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein selecting the preset number of corresponding groups of texture images and the depth maps that are closest to the virtual viewpoint position in the image combination at the user interaction moment according to the virtual viewpoint position information and the parameter data of the image combination comprises:
selecting texture images and depth maps corresponding to 2 to N capturing devices closest to the virtual viewpoint position according to the virtual viewpoint position information and the parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment, wherein N is the number of all capturing devices that capture the image combination.

### Claim 18 (depends on 14)

The computer-readable storage medium of claim 14, wherein combining and rendering the selected corresponding group of texture images and depth maps in the image combination at the user interaction moment based on the virtual viewpoint position information and parameter data corresponding to the corresponding group of texture images and depth maps in the image combination at the user interaction moment to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment comprises:
performing forward projection on the selected depth maps of the corresponding group in the image combination at the user interaction moment respectively, to project to the virtual position at the user interaction moment;
performing post-processing on the depth maps after the forward projection respectively;
performing backward projection on the selected texture images of the corresponding group in the image combination at the user interaction moment respectively; and
fusing respective virtual texture images generated after the backward projection.

### Claim 19 (depends on 14)

The computer-readable storage medium of claim 14, wherein after fusing respective the virtual texture images generated after the backward projection, the method further comprises:
performing inpainting on the fused texture image to obtain the reconstructed image corresponding to the virtual viewpoint position at the user interaction moment.

### Claim 20 (depends on 14)

The computer-readable storage medium of claim 14, wherein performing post-processing on the depth maps after the forward projection respectively comprises at least one of the following:
performing foreground padding processing on the depth maps after the forward projection respectively; and
performing pixel-level filtering processing on the depth maps after the forward projection respectively.