# US 12,118,691 B2 · Image transmission/reception system, image transmission apparatus, image reception apparatus, image transmission/reception method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,691 B2 - Document ID: US12118691B2 - Status: Granted - Kind: B2 - Application number: 17/431,609 - Filed: 2019-12-25 - Published: 2022-03-10 - Granted: 2024-10-15 - Earliest claimed: 2019-03-01 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Katsushi Otsuka - Web page: https://patentgrep.com/patent/US12118691B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/52: involving aspects of the displayed game scene - G06T 3/4046: using neural networks - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - H04N 1/387: Composing, repositioning or otherwise {geometrically} modifying originals - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - H04N 21/431: Generation of visual interfaces {for content selection or interaction}; Content or additional data rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An image transmission/reception system, an image transmission apparatus, an image reception apparatus, an image transmission/reception method, and a program, generate, from image data representing a downscaled image, a higher resolution image than the downscaled image and is more similar to an original image. An encoding process section generates image data representing a downscaled image obtained by downscaling the original image. An additional data generation section generates additional data on the basis of the original image, the additional data being unidentifiable by the image data alone. A data transmission section transmits the image data and the additional data. A data reception section receives the image data and the additional data. An upscaling section generates an upscaled image on the basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image. A display control section causes the upscaled image to be displayed. ## Claims ### Claim 1 An image transmission/reception system comprising: an image transmission apparatus; and an image reception apparatus, wherein the image transmission apparatus includes an image data generation section configured to generate image data representing a downscaled image obtained by downscaling an original image, an additional data generation section configured to generate additional data on a basis of the original image, the additional data including a feature amount of the original image and a feature amount that is a parameter used for encoding the downscaled image, the parameter being unidentifiable by the image data alone, and a data transmission section configured to transmit the image data and the additional data, and the image reception apparatus includes a data reception section configured to receive the image data and the additional data, an upscaling section configured to generate an upscaled image on a basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image, and a display control section configured to cause the upscaled image to be displayed, wherein the additional data generation section is configured to determine whether or not it is necessary to perform super-resolution processing on the downscaled image, on the basis of the feature amounts, wherein the additional data includes data indicating the determination, and wherein the upscaling section generates algorithm data indicative of whether an image decoded from the received image data is suitable for super-resolution, and the algorithm data is generated on the basis of the decoded image and feature quantity data of the additional data, using an input determination function dictionary. ### Claim 2 (depends on 1) The image transmission/reception system according to claim 1, wherein the additional data generation section includes a feature quantity data generation section configured to generate feature quantity data indicative of at least either a feature quantity of the original image or a feature quantity that is a parameter used for encoding the downscaled image, the data transmission section transmits the additional data including the feature quantity data, and the upscaling section generates the upscaled image on a basis of the image data and the feature quantity data. ### Claim 3 (depends on 1) The image transmission/reception system according to claim 1, wherein the additional data generation section includes a feature quantity data generation section and a super-resolution processing method data generation section, the feature quantity data generation section being configured to generate feature quantity data indicative of at least either a feature quantity of the original image or a feature quantity that is a parameter used for encoding the downscaled image, the super-resolution processing method data generation section being configured to generate super-resolution processing method data including a parameter for use in super-resolution processing on a basis of the feature quantity data, the data transmission section transmits the additional data including the super-resolution processing method data, and the upscaling section generates the upscaled image on a basis of the image data and the super-resolution processing method data. ### Claim 4 (depends on 3) The image transmission/reception system according to claim 3, wherein the upscaling section includes a trained machine learning model configured to output the upscaled image upon receiving, as input, the super-resolution processing method data and an image generated on a basis of the image data. ### Claim 5 (depends on 1) The image transmission/reception system according to claim 1, wherein the data transmission section transmits, to the image reception apparatus, the additional data with higher priority than the image data. ### Claim 6 (depends on 1) The image transmission/reception system according to claim 1, wherein the upscaling section generates the upscaled image on a basis of a time series of the image data and a time series of the additional data, the upscaled image being higher in resolution than the downscaled image. ### Claim 7 An image transmission apparatus comprising: an image data generation section configured to generate image data representing a downscaled image obtained by downscaling an original image; an additional data generation section configured to generate additional data on a basis of the original image, the additional data including a feature amount of the original image and a feature amount that is a parameter used for encoding the downscaled image, the parameter being unidentifiable by the image data alone; and a data transmission section configured to transmit the image data and the additional data to an image reception apparatus including an upscaling section and a display control section, the upscaling section being configured to generate an upscaled image on a basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image, the display control section being configured to cause the upscaled image to be displayed, wherein the additional data generation section is configured to determine whether or not it is necessary to perform super-resolution processing on the downscaled image, on the basis of the feature amounts, wherein the additional data includes data indicating the determination, and wherein the upscaling section generates algorithm data indicative of whether an image decoded from the received image data is suitable for super-resolution, and the algorithm data is generated on the basis of the decoded image and feature quantity data of the additional data, using an input determination function dictionary. ### Claim 8 An image reception apparatus comprising: a data reception section configured to receive image data and additional data from an image transmission apparatus including an image data generation section, an additional data generation section, and a data transmission section, the image data generation section being configured to generate the image data representing a downscaled image obtained by downscaling an original image, the additional data generation section being configured to generate the additional data on a basis of the original image, the additional data including a feature amount of the original image and a feature amount that is a parameter used for encoding the downscaled image, the parameter being unidentifiable by the image data alone, the data transmission section being configured to transmit the image data and the additional data; an upscaling section configured to generate an upscaled image on a basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image; and a display control section configured to cause the upscaled image to be displayed, wherein the additional data generation section is configured to determine whether or not it is necessary to perform super-resolution processing on the downscaled image, on the basis of the feature amounts, wherein the additional data includes data indicating the determination, and wherein the upscaling section generates algorithm data indicative of whether an image decoded from the received image data is suitable for super-resolution, and the algorithm data is generated on the basis of the decoded image and feature quantity data of the additional data, using an input determination function dictionary. ### Claim 9 An image transmission/reception method comprising: causing an image transmission apparatus to generate image data representing a downscaled image obtained by downscaling an original image; causing the image transmission apparatus to generate additional data on a basis of the original image, the additional data including a feature amount of the original image and a feature amount that is a parameter used for encoding the downscaled image, the parameter being unidentifiable by the image data alone; causing the image transmission apparatus to transmit the image data and the additional data; causing an image reception apparatus to receive the image data and the additional data; causing the image reception apparatus to generate an upscaled image on a basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image; and causing the image reception apparatus to cause the upscaled image to be displayed, wherein an additional data generation section is configured to determine whether or not it is necessary to perform super-resolution processing on the downscaled image, on the basis of the feature amounts, wherein the additional data includes data indicating the determination, and wherein algorithm data is generated indicative of whether an image decoded from the received image data is suitable for super-resolution, and the algorithm data is generated on the basis of the decoded image and feature quantity data of the additional data, using an input determination function dictionary. ### Claim 10 A non-transitory, computer-readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform an image generation and transmission method by carrying out actions, comprising: generating image data representing a downscaled image obtained by downscaling an original image; generating additional data on a basis of the original image, the additional data including a feature amount of the original image and a feature amount that is a parameter used for encoding the downscaled image, the parameter being unidentifiable by the image data alone; and transmitting the image data and the additional data to an image reception apparatus including an upscaling section and a display control section, the upscaling section being configured to generate an upscaled image on a basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image, the display control section being configured to cause the upscaled image to be displayed, wherein an additional data generation section is configured to determine whether or not it is necessary to perform super-resolution processing on the downscaled image, on the basis of the feature amounts, and wherein the additional data includes data indicating the determination, and wherein the upscaling section generates algorithm data indicative of whether an image decoded from the received image data is suitable for super-resolution, and the algorithm data is generated on the basis of the decoded image and feature quantity data of the additional data, using an input determination function dictionary. ### Claim 11 A non-transitory, computer-readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform an image reception method by carrying out actions, comprising: receiving image data and additional data from an image transmission apparatus including an image data generation section, an additional data generation section, and a data transmission section, the image data generation section being configured to generate the image data representing a downscaled image obtained by downscaling an original image, the additional data generation section being configured to generate the additional data on a basis of the original image, the additional data including a feature amount of the original image and a feature amount that is a parameter used for encoding the downscaled image, the parameter being unidentifiable by the image data alone, the data transmission section being configured to transmit the image data and the additional data; generating an upscaled image on a basis of the image data and the additional data, the upscaled image being higher in resolution than the downscaled image; and causing the upscaled image to be displayed, wherein the additional data generation section is configured to determine whether or not it is necessary to perform super-resolution processing on the downscaled image, on the basis of the feature amounts, wherein the additional data includes data indicating the determination, and wherein algorithm data is generated indicative of whether an image decoded from the received image data is suitable for super-resolution, and the algorithm data is generated on the basis of the decoded image and feature quantity data of the additional data, using an input determination function dictionary.