# US 10,771,758 B2 · Immersive viewing using a planar array of cameras > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,758 B2 - Document ID: US10771758B2 - Status: Granted - Kind: B2 - Application number: 16/139,880 - Filed: 2018-09-24 - Published: 2019-02-07 - Granted: 2020-09-08 - Earliest claimed: 2018-09-24 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Oscar Nestares; Kalpana Seshadrinathan; Vladan Popovic; Horst Haussecker - Web page: https://patentgrep.com/patent/US10771758B2 ## CPC - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G11B 27/022: Electronic editing of analogue information signals, e.g. audio or video signals - H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation - H04N 23/10: for generating image signals from different wavelengths - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Techniques related to generating a virtual view from multi-view images for presentation to a viewer are discussed. Such techniques include determining, based on a viewer position relative to a display region, first and second crop positions of planar image and cropping the planar image to a cropped planar image to fill the display region using the first and second crop positions such that the first and second crop positions define an asymmetric frustum between the cropped planar image and a virtual window corresponding to the display region. ## Claims ### Claim 1 A system for generating a virtual view from multi-view images comprising: a memory to store a plurality of planar images of a scene; and a processor coupled to the memory, the processor to: attain a first planar image representative of the scene based on the plurality of planar images; determine, based on a viewer position relative to a display region, a first crop position of the first planar image and a second crop position of the first planar image; crop the first planar image to a cropped planar image to fill the display region based on the first and second crop positions, wherein the first and second crop positions define an asymmetric frustum of the first planar image corresponding to a virtual window representing the display region, the first crop position is at a position away from a midpoint of the first planar image toward a first edge of the first planar image by a ratio of a product of a focal length corresponding to the first planar image and a lateral position of virtual viewer position from the first edge of the virtual window to a distance of the virtual viewer position from the virtual window, and the virtual viewer position corresponds to the viewer position; and provide the cropped planar image for presentation to the viewer. ### Claim 2 (depends on 1) The system of claim 1, wherein the first edge of the first planar image and the first edge of the virtual window are one of corresponding top, bottom, left, or right edges. ### Claim 3 (depends on 1) The system of claim 1, wherein the second crop position is at a second position away from the midpoint of the first planar image toward a second edge of the first planar image by a ratio of a product of the focal length and a difference between the width of the virtual window and the lateral position of the virtual viewer position from the first edge of the virtual window to the distance of the user from the virtual window. ### Claim 4 (depends on 1) The system of claim 1, wherein the second crop position is at a position away from the midpoint of the first planar image toward the first edge of the first planar image by a ratio of a product of the focal length and a difference between the lateral position of the virtual viewer position from the first edge of the virtual window and a width of the virtual window to the distance of the virtual viewer position from the virtual window. ### Claim 5 (depends on 1) The system of claim 1, wherein the processor to crop crops the first planar image at the first and second crop positions in a first dimension and at third and fourth crop positions in a second dimension based on an aspect ratio of the display region. ### Claim 6 (depends on 1) The system of claim 1, the processor further to: attain, after a viewer moves to a second viewer position, a second planar image representative of the scene based on a second plurality of planar images of the scene; and determine, based on the second viewer position relative to the display region, a third crop position of the second planar image and a fourth crop position of the second planar image, wherein a first virtual viewer position corresponding to the viewer position is between first and second edges of the virtual window and a second virtual viewer position corresponding to the second viewer position is outside one of the first and second edges of the virtual window and wherein the first and second crop positions are on opposite sides of a midpoint of the first planar image and the third and fourth crop positions are on the same side of a midpoint of the second planar image. ### Claim 7 (depends on 1) The system of claim 1, wherein the plurality of planar images comprise images captured via an array of cameras, wherein the viewer position translates to an image capture position of one of the array of cameras, and wherein the first planar image comprises a corresponding one of the plurality of captured planar images. ### Claim 8 (depends on 1) The system of claim 1, wherein the plurality of planar images comprise images captured via an array of cameras, wherein the viewer position translates to a position between first and second cameras of the array of cameras, and wherein the processor to attain the first planar image comprises the processor to: synthesize the first planar image between a second planar image and a third planar image corresponding to the first and second cameras, respectively, by generation of bi-directional disparity maps based on the second and third planar images and warp of the second and third planar images to the first planar image based on the bi-directional disparity maps. ### Claim 9 (depends on 1) The system of claim 1, wherein the processor to provide the cropped planar image for presentation to the viewer comprises the processor to transmit the cropped planar image to a receiving device having a display screen comprising the display region. ### Claim 10 (depends on 1) The system of claim 1, wherein the processor to provide the cropped planar image for presentation to the viewer comprises the processor to transmit the cropped planar image to a display screen comprising the display region. ### Claim 11 A method for generating a virtual view from multi-view images comprising: attaining a first planar image representative of a scene based on a plurality of planar images of the scene; determining, based on a viewer position relative to a display region, a first crop position of the first planar image and a second crop position of the first planar image; cropping the first planar image to a cropped planar image to fill the display region based on the first and second crop positions, wherein the first and second crop positions define an asymmetric frustum of the first planar image corresponding to a virtual window corresponding to the display region, the first crop position is at a position away from a midpoint of the first planar image toward a first edge of the first planar image by a ratio of a product of a focal length corresponding to the first planar image and a lateral position of virtual viewer position from the first edge of the virtual window to a distance of the virtual viewer position from the virtual window, and the virtual viewer position corresponds to the viewer position; and providing the cropped planar image for presentation to the viewer. ### Claim 12 (depends on 11) The method of claim 11, wherein the second crop position is at a second position away from the midpoint of the first planar image toward a second edge of the first planar image by a ratio of a product of the focal length and a difference between the width of the virtual window and the lateral position of the virtual viewer position from the first edge of the virtual window to the distance of the user from the virtual window. ### Claim 13 (depends on 11) The method of claim 11, wherein the second crop position is at a position away from the midpoint of the first planar image toward the first edge of the first planar image by a ratio of a product of the focal length and a difference between the lateral position of the virtual viewer position from the first edge of the virtual window and a width of the virtual window to the distance of the virtual viewer position from the virtual window. ### Claim 14 (depends on 11) The method of claim 11, wherein said cropping crops the first planar image at the first and second crop positions in a first dimension and at third and fourth crop positions in a second dimension based on an aspect ratio of the display region. ### Claim 15 (depends on 11) The method of claim 11, further comprising: attaining, after a viewer moves to a second viewer position, a second planar image representative of the scene based on a second plurality of planar images of the scene; and determining, based on the second viewer position relative to the display region, a third crop position of the second planar image and a fourth crop position of the second planar image, wherein a first virtual viewer position corresponding to the viewer position is between first and second edges of the virtual window and a second virtual viewer position corresponding to the second viewer position is outside one of the first and second edges of the virtual window and wherein the first and second crop positions are on opposite sides of a midpoint of the first planar image and the third and fourth crop positions are on the same side of a midpoint of the second planar image. ### Claim 16 At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to generate a virtual view from multi-view images by: attaining a first planar image representative of a scene based on a plurality of planar images of the scene; determining, based on a viewer position relative to a display region, a first crop position of the first planar image and a second crop position of the first planar image; cropping the first planar image to a cropped planar image to fill the display region based on the first and second crop positions, wherein the first and second crop positions define an asymmetric frustum of the first planar image corresponding to a virtual window corresponding to the display region, the first crop position is at a position away from a midpoint of the first planar image toward a first edge of the first planar image by a ratio of a product of a focal length corresponding to the first planar image and a lateral position of virtual viewer position from the first edge of the virtual window to a distance of the virtual viewer position from the virtual window, and the virtual viewer position corresponds to the viewer position; and providing the cropped planar image for presentation to the viewer. ### Claim 17 (depends on 16) The machine readable medium of claim 16, wherein the second crop position is at a second position away from the midpoint of the first planar image toward a second edge of the first planar image by a ratio of a product of the focal length and a difference between the width of the virtual window and the lateral position of the virtual viewer position from the first edge of the virtual window to the distance of the user from the virtual window. ### Claim 18 (depends on 16) The machine readable medium of claim 16, wherein the second crop position is at a position away from the midpoint of the first planar image toward the first edge of the first planar image by a ratio of a product of the focal length and a difference between the lateral position of the virtual viewer position from the first edge of the virtual window and a width of the virtual window to the distance of the virtual viewer position from the virtual window. ### Claim 19 (depends on 16) The machine readable medium of claim 16, wherein said cropping crops the first planar image at the first and second crop positions in a first dimension and at third and fourth crop positions in a second dimension based on an aspect ratio of the display region. ### Claim 20 (depends on 16) The machine readable medium of claim 16, further comprising instructions that, in response to being executed on the computing device, cause the computing device to generate a virtual view from multi-view images by: attaining, after a viewer moves to a second viewer position, a second planar image representative of the scene based on a second plurality of planar images of the scene; and determining, based on the second viewer position relative to the display region, a third crop position of the second planar image and a fourth crop position of the second planar image, wherein a first virtual viewer position corresponding to the viewer position is between first and second edges of the virtual window and a second virtual viewer position corresponding to the second viewer position is outside one of the first and second edges of the virtual window and wherein the first and second crop positions are on opposite sides of a midpoint of the first planar image and the third and fourth crop positions are on the same side of a midpoint of the second planar image.