# US 12,005,350 B2 · Image transmission/reception system, image transmission apparatus, image reception apparatus, image transmission/reception method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,005,350 B2 - Document ID: US12005350B2 - Status: Granted - Kind: B2 - Application number: 17/435,857 - Filed: 2019-12-25 - Published: 2022-05-19 - Granted: 2024-06-11 - Earliest claimed: 2019-03-25 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Katsushi Otsuka - Web page: https://patentgrep.com/patent/US12005350B2 ## CPC - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/52: involving aspects of the displayed game scene - A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - H04N 7/01: Conversion of standards {, e.g. involving analogue television standards or digital television standards processed at pixel level} - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/172: the region being a picture, frame or field - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - H04N 19/87: involving scene cut or scene change detection in combination with video compression - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - H04N 21/4402: involving reformatting operations of video signals for household redistribution, storage or real-time display {(details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764)} - A63F 2300/538: for performing operations on behalf of the game client, e.g. rendering - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An upscaling unit generates a first to-be-displayed image on the basis of first image data. An alternative image generation unit generates an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on the basis of second image data. A display control unit displays, after displaying the image generated on the basis of the first image data, any one of the image generated on the basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded. A transmission control unit controls whether or not to transmit, to a terminal, the second image data on the basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit. ## Claims ### Claim 1 An image transmission/reception system comprising: an image transmission apparatus; and an image reception apparatus, the image transmission apparatus including an image generation unit configured to generate a first image and generate a second image after generating the first image, an image data transmission unit configured to transmit first image data generated on a basis of the first image, and a transmission control unit configured to control whether or not to transmit second image data generated on a basis of the second image, the image reception apparatus including an image data reception unit configured to receive the first image data, a to-be-displayed image generation unit configured to generate a first to-be-displayed image on a basis of the first image data, an alternative image generation unit configured to generate an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on a basis of the second image data by the to-be-displayed image generation unit, and a display control unit configured to display, after displaying the first to-be-displayed image, any one of the second to-be-displayed image generated on the basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded, wherein the transmission control unit controls whether or not to transmit the second image data on a basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image. ### Claim 2 (depends on 1) The image transmission/reception system according to claim 1, wherein the transmission control unit controls whether or not to transmit the second image data on the basis of the possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image. ### Claim 3 (depends on 2) The image transmission/reception system according to claim 2, wherein the transmission control unit controls whether or not to transmit the second image data on the basis of the possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a chronological order of feature amounts including the feature amount of the first image and the feature amount of the second image. ### Claim 4 (depends on 1) The image transmission/reception system according to claim 1, wherein the transmission control unit controls, on the basis of the possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a feature amount that includes a parameter that is used for encoding a downscaled image obtained by downscaling the second image, whether or not to transmit the second image data indicating the downscaled image. ### Claim 5 (depends on 1) The image transmission/reception system according to claim 1, wherein the image data transmission unit transmits the first image data indicating a first downscaled image obtained by downscaling the first image, and the transmission control unit controls, on the basis of the possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a feature amount that includes a parameter that is used for encoding the first downscaled image and a feature amount that includes a parameter that is used for encoding a second downscaled image obtained by downscaling the second image, whether or not to transmit the second image data indicating the second downscaled image. ### Claim 6 (depends on 2) The image transmission/reception system according to claim 2, wherein the feature amount indicates at least any one of pieces of information indicating a feature point of an image, an edge strength of the image, a depth of a pixel included in the image, a texture type of the image, an optical flow in the image, and a moving direction and a speed of a rectangular region in the image. ### Claim 7 (depends on 2) The image transmission/reception system according to claim 2, wherein the transmission control unit performs control of giving priority to transmission of feature amount data indicating the feature amount to the image reception apparatus over transmission of image data indicating an image. ### Claim 8 (depends on 1) The image transmission/reception system according to claim 1, wherein the transmission control unit controls whether or not to transmit the second image data on a basis of whether or not the second image includes a key frame. ### Claim 9 (depends on 1) The image transmission/reception system according to claim 1, wherein the transmission control unit controls whether or not to transmit the second image data on a basis of whether or not a scene in the second image is switched from a scene in the first image. ### Claim 10 (depends on 1) The image transmission/reception system according to claim 1, wherein the transmission control unit controls whether or not to transmit the second image data on a basis of whether or not the second image includes an image of an object that does not appear in the first image. ### Claim 11 (depends on 1) The image transmission/reception system according to claim 1, wherein the alternative image generation unit starts generation of the alternative image on the basis of the first image data before whether or not the reception of the second image data has succeeded is checked. ### Claim 12 (depends on 1) The image transmission/reception system according to claim 1, wherein the transmission control unit performs, in a case where a transmission speed of the image transmission apparatus is higher than a predetermined speed, control of transmitting the second image data, irrespective of the possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit. ### Claim 13 An image transmission apparatus comprising: an image generation unit configured to generate a first image and generate a second image after generating the first image; an image data transmission unit configured to transmit first image data generated on a basis of the first image to an image reception apparatus, the image reception apparatus including an image data reception unit configured to receive the first image data, a to-be-displayed image generation unit configured to generate a first to-be-displayed image on a basis of the first image data, an alternative image generation unit configured to generate an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on a basis of second image data generated on a basis of the second image by the to-be-displayed image generation unit, and a display control unit configured to display, after displaying the first to-be-displayed image, any one of the second to-be-displayed image generated on the basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded; and a transmission control unit configured to control whether or not to transmit the second image data generated on the basis of the second image to the image reception apparatus on a basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image. ### Claim 14 An image reception apparatus comprising: an image data reception unit configured to receive first image data from an image transmission apparatus; a to-be-displayed image generation unit configured to generate a first to-be-displayed image on a basis of the first image data; an alternative image generation unit configured to generate an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on a basis of second image data by the to-be-displayed image generation unit; and a display control unit configured to display, after displaying the first to-be-displayed image, any one of the second to-be-displayed image generated on the basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded, whether or not to transmit the second image data being controlled by the image transmission apparatus on a basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data by the alternative image generation unit, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image. ### Claim 15 An image transmission/reception method comprising: generating a first image; generating a second image after generating the first image; transmitting first image data generated on a basis of the first image; controlling whether or not to transmit second image data generated on a basis of the second image; receiving the first image data; generating a first to-be-displayed image on a basis of the first image data; generating an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on a basis of the second image data in the to-be-displayed image generation; and displaying, after displaying the first to-be-displayed image, any one of the second to-be-displayed image generated on a basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded, wherein, the controlling controls whether or not to transmit the second image data on a basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data in the alternative image generation, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image. ### Claim 16 A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform an image transmission/reception method by carrying out actions, comprising: generating a first image; generating a second image after generating the first image; transmitting first image data generated on a basis of the first image to an image reception apparatus, wherein the generating the first image and the second image includes receiving the first image data, generating a first to-be-displayed image on a basis of the first image data, generating an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on a basis of second image data generated on a basis of the second image by the to-be-displayed image generation unit, and displaying, after displaying the first to-be-displayed image, any one of the second to-be-displayed image generated on the basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded; and controlling whether or not to transmit the second image data generated on the basis of the second image to the image reception apparatus on a basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image. ### Claim 17 A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform an image transmission/reception method by carrying out actions, comprising: receiving first image data from an image transmission apparatus; generating a first to-be-displayed image on a basis of the first image data; generating an alternative image by estimating, on the basis of the first image data, a second to-be-displayed image to be generated on a basis of second image data in the to-be-displayed image; and displaying, after displaying the first to-be-displayed image, any one of the second to-be-displayed image generated on the basis of the second image data and the alternative image, depending on whether or not reception of the second image data has succeeded, whether or not to transmit the second image data being controlled on a basis of a possibility of successful estimation of the second to-be-displayed image on the basis of the first image data in the alternative image, the possibility being determined on a basis of a feature amount of the first image and a feature amount of the second image.