# US 2016/0366392 A1 · Image encoding and display > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2016/0366392 A1 - Document ID: US20160366392A1 - Status: Published application - Kind: A1 - Application number: 15/121,518 - Filed: 2015-02-25 - Published: 2016-12-15 - Earliest claimed: 2014-02-26 - Assignees: Sony Computer Entertainment Europe Limited - Applicants: Sony Computer Entertainment Europe Limited - Inventors: Sharwin Winesh Raghoebardajal; Ian Henry Bickerstaff - Web page: https://patentgrep.com/patent/US20160366392A1 ## CPC - H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) - H04N 13/10: Processing, recording or transmission of stereoscopic or multi-view image signals - H04N 13/122: Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) - H04N 13/156: Mixing image signals - H04N 13/189: Recording image signals; Reproducing recorded image signals - H04N 13/194: Transmission of image signals - H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD] - H04N 13/366: using viewer tracking - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - H04N 19/17: the unit being an image region, e.g. an object - H04N 21/440245: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment} - H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region - H04N 21/816: {involving special video data, e.g 3D video} - H04N 13/344: with head-mounted left-right displays - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 2013/0074: {Stereoscopic image analysis} - H04N 2213/008: Aspects relating to glasses for viewing stereoscopic images - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image encoding method includes, for each of left and right images of an image stereo pair, identifying a region of interest (ROI) in an input image, generating multiple representations of that input image such that one representation is a high field of view (FOV) representation including substantially all of the input image and another representation is a low FOV representation having a lower FOV than the high FOV representation and including at least the ROI. The relationship between pixel resolution and image size for the high and low FOV representations is such that, in relation to the ROI, the low FOV representation provides more image detail than the high FOV representation. The method also includes generating one or more multiplexed images by juxtaposing portions of the left and right, high and low FOV representations so as to include all image content thereof in fewer than four multiplexed images. ## Claims ### Claim 1 An image encoding method comprising: for each of a left and a right image of a stereo pair of images, identifying, in an input image, a region of interest; generating, by one or more processors, at least two representations of that input image, such that one of the representations is a high field of view (FOV) representation including substantially all of the input image, and another of the representations is a low FOV representation having a lower FOV than the high FOV representation; and including at least the region of interest; in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to the region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation; generating, by the one or more processors, one or more multiplexed images by juxtaposing at least portions of the left and right, high FOV and low FOV representations, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images; and performing one or both of transmitting or storing the multiplexed images. ### Claim 2 (depends on 1) A method according to claim 1, in which the step of generating the one or more multiplexed images comprises generating a first multiplexed image by juxtaposing a whole of a first one of the left and right, high FOV and low FOV representations next to a whole of a second one of the left and right, high FOV and low FOV representations, and generating a second multiplexed image by juxtaposing a whole of a third one of the left and right, high FOV and low FOV representations next to a whole of a fourth one of the left and right, high FOV and low FOV representations. ### Claim 3 (depends on 1) A method according to claim 1, in which the performing step comprises one or both of transmitting or storing the multiplexed images at a higher image repetition rate than an image repetition rate of the input images. ### Claim 4 (depends on 1) A method according to claim 1, in which performing step comprises one or both of transmitting or storing the multiplexed images as respective left and right dummy 3D images. ### Claim 5 An image display method comprising: receiving multiplexed images, each multiplexed image including juxtaposed portions of left and right, high field of view (FOV) and low FOV representations of an input stereo image pair, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images, in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to a region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation; demutiplexing, by one or more processors, the multiplexed images to generate sets of left and right, high FOV and low FOV representations; and combining, by the one or more processors, the low FOV representation and the high FOV representation of each of the left and right images to provide respective left and right display images. ### Claim 6 (depends on 5) A method according to claim 5, in which the combining step comprises: using primarily image content from the low FOV representation in respect of image regions represented by the low FOV representation, and using image content from the respective high FOV representation for other image regions. ### Claim 7 (depends on 6) A method according to claim 6, in which the combining step comprises: for at least a central region of the low FOV representation, replacing image content of the high FOV representation by corresponding image content of the low FOV representation. ### Claim 8 (depends on 7) A method according to claim 7, in which the combining step comprises: for at least a peripheral region of the low FOV representation, mixing image content of the high and low FOV representations. ### Claim 9 (depends on 5) A method according to claim 5, in which the combining step comprises re-projecting to form a respective display image, according to a desired display viewpoint. ### Claim 10 (depends on 9) A method according to claim 9, comprising displaying the re-projected image using a display. ### Claim 11 (depends on 10) A method according to claim 10, in which the display is a head-mountable display and the desired display viewpoint is detected by detecting one or both of a position and an orientation of the head-mountable display. ### Claim 12 (canceled) ### Claim 13 A non-transitory computer readable storage medium storing computer readable instructions thereon, the instructions, when executed by one or more processors, casing the one or more processors to perform an image encoding method, the method comprising: for each of a left and a right image of a stereo pair of images, identifying, in an input image, a region of interest; generating at least two representations of that input image, such that one of the representations is a high field of view (FOV) representation including substantially all of the input image, and another of the representations is a low FOV representation having a lower FOV than the high FOV representation and including at least the region of interest; in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to the region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation; generating one or more multiplexed images by juxtaposing at least portions of the left and right, high FOV and low FOV representations, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images; and performing one or both of transmitting or storing the multiplexed images. ### Claim at position 14 14-17. (canceled) ### Claim 18 An image encoding apparatus comprising: a representation generator configured, for each of a left and a right image of a stereo pair of images, to: identify, in an input image, a region of interest; and generate at least two representations of the input image, such that one of the representations is a high field of view (FOV) representation including substantially all of the input image, and another of the representations is a low FOV representation having a lower FOV than the high FOV representation; and including at least the region of interest; in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to the region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation; a multiplexer operatively coupled to the representation generator and configured to generate one or more multiplexed images by juxtaposing at least portions of the left and right, high FOV and low FOV representations, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images; and an image output operatively coupled to the multiplexer and configured to output the multiplexed images. ### Claim 19 (depends on 18) The image encoding apparatus of claim 18, wherein the image encoding apparatus is a videogame machine. ### Claim 20 An image display apparatus comprising: an image input configured to receive multiplexed images output by an image encoding apparatus, each multiplexed image including juxtaposed portions of left and right, high field of view (FOV) and low FOV representations of an input stereo image pair, so as to include all image content of the left and right, high FOV and low FOV representations in fewer than four multiplexed images, in which a relationship between pixel resolution and image size for the high FOV representation and the low FOV representation is such that, in relation to a region of interest, the low FOV representation provides a higher amount of image detail than the high FOV representation; a demultiplexer configured to demultiplex the multiplexed images to generate sets of left and right, high FOV and low FOV representations; and a combiner configured to combine the low FOV representation and the high FOV representation of each of the left and right images to provide respective left and right display images. ### Claim 21 (depends on 20) The image display apparatus of claim 20, wherein the image display apparatus is a head-mountable display system.