# US 10,484,658 B2 · Apparatus and method for generating image of arbitrary viewpoint using camera array and multi-focus image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,658 B2 - Document ID: US10484658B2 - Status: Granted - Kind: B2 - Application number: 15/834,237 - Filed: 2017-12-07 - Published: 2018-06-14 - Granted: 2019-11-19 - Earliest claimed: 2016-12-08 - Assignees: Electronics and Telecommunications Research Institute - Applicants: Electronics and Telecommunications Research Institute - Inventors: Seong Jun Bae; Ho Wook Jang; Wook Ho Son; Jae Woo Kim; Do Hyung Kim; Seong Jin Park - Web page: https://patentgrep.com/patent/US10484658B2 ## CPC - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - G06T 7/73: using feature-based methods - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/957: Light-field or plenoptic cameras or camera modules - G06T 2207/10012: Stereo images - G06T 2207/10148: Varying focus - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An arbitrary viewpoint image generation method includes obtaining an original image set photographed by a plurality of cameras included in a camera array at each of at least one focal distance at a same time point; obtaining a multi-focus image set by generating a multi-focus image from the original image set for each of the at least one focal distance; and generating an arbitrary viewpoint image at a position where a viewpoint is to be moved from the multi-focus image set. ## Claims ### Claim 1 An arbitrary viewpoint image generation method comprising: obtaining an original image set of a plurality of original images photographed by a plurality of cameras included in a camera array at each of at least one focal distance at a same time point; obtaining a multi-focus image set by generating a multi-focus image from the original image set a each of the at least one focal distance; and generating an arbitrary viewpoint image at a position where a viewpoint is to be moved from the multi-focus image set, wherein the obtaining of the multi-focus image set comprises: moving each of the original images in parallel as much as a positional coordinate of each corresponding camera included in the camera array; and generating the multi-focus image by taking an average of the parallel-moved original images. ### Claim 2 (depends on 1) The arbitrary viewpoint image generation method according to claim 1, further comprising, after the obtaining of the original image set, performing correction on the original image set by reflecting characteristics of the camera array. ### Claim 3 (depends on 2) The arbitrary viewpoint image generation method according to claim 2, wherein the performing of correction comprises: obtaining a characteristic or position of each of the plurality of cameras included in the camera array using the original image set; and performing correction on each original image included in the original image set by reflecting the obtained characteristic or position. ### Claim 4 (depends on 1) The arbitrary viewpoint image generation method according to claim 1, wherein the positional coordinate of each corresponding camera is determined in x pixel coordinate and y pixel coordinate with respect to a reference camera of the camera array. ### Claim 5 (depends on 1) The arbitrary viewpoint image generation method according to claim 1, wherein the each original image is moved in parallel by correcting the positional coordinate of each corresponding camera based on a relative distance between a reference focal distance, which is selected from the at least one focal distance, and each of the at least one focal distance. ### Claim 6 (depends on 5) The arbitrary viewpoint image generation method according to claim 5, wherein the generating of the arbitrary viewpoint image comprises: moving each multi-focus image included in the obtained multi-focus image set in parallel according to a viewpoint movement direction and a viewpoint movement distance; and generating the arbitrary viewpoint image by taking an average of the parallel-moved multi-focus images. ### Claim 7 (depends on 6) The arbitrary viewpoint image generation method according to claim 6, wherein the each multi-focus image is moved in parallel by correcting the viewpoint movement distance based on the relative distance. ### Claim 8 (depends on 1) The arbitrary viewpoint image generation method according to claim 1, wherein the generating of the arbitrary viewpoint image comprises generating the arbitrary viewpoint image by reflecting a variance of pixel values of each of the parallel-moved original images at the at least one focal distance. ### Claim 9 (depends on 8) The arbitrary viewpoint image generation method according to claim 8, wherein the variance is calculated for each of the at least one focal distance by using a deviation between pixel values of each of the parallel-moved original images and pixel values of the generated multi-focus images. ### Claim 10 An arbitrary viewpoint image generation apparatus comprising a processor and a memory storing at least one instruction executed by the processor, wherein the at least one instruction is configured to: obtain an original image set of a plurality of original images photographed by a plurality of cameras included in a camera array at each of at least one focal distance at a same time point; obtain a multi-focus image set by generating a multi-focus image from the original image set at each of the at least one focal distance; and generate an arbitrary viewpoint image at a position where a viewpoint is to be moved from the multi-focus image set, wherein the obtaining of the multi-focus image set comprises: moving each of the original images in parallel as much as a positional coordinate of each corresponding camera included in the camera array; and generating the multi-focus image by taking an average of the parallel-moved original images. ### Claim 11 (depends on 10) The arbitrary viewpoint image generation apparatus according to claim 10, wherein the at least one instruction is further configured to perform correction on the original image set by reflecting characteristics of the camera array after the obtaining of the original image set. ### Claim 12 (depends on 11) The arbitrary viewpoint image generation apparatus according to claim 11, wherein the at least one instruction is further configured to; obtain a characteristic or position of each of the plurality of cameras included in the camera array using the original image set; and perform correction on each original image included in the original image set by reflecting the obtained characteristic or position. ### Claim 13 (depends on 10) The arbitrary viewpoint image generation apparatus according to claim 10, wherein the positional coordinate of each corresponding camera is determined in x pixel coordinate and y pixel coordinate with respect to a reference camera of the camera array. ### Claim 14 (depends on 10) The arbitrary viewpoint image generation apparatus according to claim 10, wherein the at least one instruction is further configured to move the each original image in parallel by correcting the positional coordinate of each corresponding camera based on a relative distance between a reference focal distance, which is selected from the at least one focal distance, and each of the at least one focal distance. ### Claim 15 (depends on 14) The arbitrary viewpoint image generation apparatus according to claim 14, wherein the at least one instruction is configured to: move each multi-focus image included in the obtained multi-focus image set in parallel according to a viewpoint movement direction and a viewpoint movement distance; and generate the arbitrary viewpoint image by taking an average for of the parallel-moved multi-focus images. ### Claim 16 (depends on 15) The arbitrary viewpoint image generation apparatus according to claim 15, wherein the at least one instruction is further configured to move the each multi-focus image in parallel by correcting the viewpoint movement distance based on the relative distance. ### Claim 17 (depends on 10) The arbitrary viewpoint image generation apparatus according to claim 10, wherein the at least one instruction is further configured to generate the arbitrary viewpoint image by reflecting a variance of pixel values of each of the parallel-moved original images at the at least one focal distance. ### Claim 18 (depends on 17) The arbitrary viewpoint image generation apparatus according to claim 17, wherein the variance is calculated for each of the at least one focal distance by using a deviation between pixel values of each of the parallel-moved original images and pixel values of the generated multi-focus images.