# US 12,112,451 B2 · Image processing apparatus and server > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,451 B2 - Document ID: US12112451B2 - Status: Granted - Kind: B2 - Application number: 17/907,831 - Filed: 2021-03-23 - Published: 2023-05-11 - Granted: 2024-10-08 - Earliest claimed: 2020-03-25 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Katsushi Otsuka - Web page: https://patentgrep.com/patent/US12112451B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - A63F 13/53: involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game - G06T 3/4046: using neural networks - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - G06T 15/005: {General purpose rendering architectures} - H04N 7/01: Conversion of standards {, e.g. involving analogue television standards or digital television standards processed at pixel level} - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - A63F 13/355: Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client - A63F 13/52: involving aspects of the displayed game scene - G06T 2210/36: Level of detail - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A client terminal includes a super-resolution processing section and a display control section. The super-resolution processing section 48 of the client terminal executes a super-resolution process for an inputted image on the basis of scene information indicating a feature in the image. The display control section of the client terminal outputs, to a display panel, the image having undergone the super-resolution process at the super-resolution processing section. The scene information is acquired in a process prior to a super-resolution process for a super-resolution process target image. ## Claims ### Claim 1 An image processing apparatus comprising: a super-resolution processing section that executes a super-resolution process for an inputted image on a basis of scene information indicating a feature in the image; and a display control section that outputs, to a display section, the image having undergone the super-resolution process at the super-resolution processing section, wherein the scene information is acquired in a process prior to a super-resolution process for a super-resolution process target image; wherein the super-resolution processing section includes a scene analyzing section and a scene-information integrating section, and during the acquisition process, the scene analyzing section sequentially outputs to the scene-information integrating section results of scene analyses of partial images of the inputted image, the partial images having a plurality of different resolutions, without waiting for completion of all of the scene analyses, and when the outputted results are sufficient to generate the super-resolution image of the inputted image, the scene analyzing section abandons uncompleted ones of the scene analyses. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, further comprising: an expansion decoding section that expansion-decodes a compression-encoded image provided from a server, wherein the super-resolution processing section executes a super-resolution process for an image expansion-decoded by the expansion decoding section, on a basis of scene information provided from the server. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, further comprising: an expansion decoding section that expansion-decodes a compression-encoded image provided from a server, wherein the expansion decoding section acquires scene information regarding the image from information indicating a configuration of a compression encoding process for use in an expansion decoding process, and the super-resolution processing section executes a super-resolution process for an image expansion-decoded by the expansion decoding section, on a basis of scene information acquired by the expansion decoding section. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein, in a super-resolution process for an image having undergone lossy compression, the super-resolution processing section performs a scene analysis on a basis of internal drawing data held in an application that has drawn the image, and uses a result of the scene analysis for the super-resolution process. ### Claim 5 (depends on 1) The image processing apparatus according to claim 1, wherein, in a super-resolution process for an image having undergone lossy compression, the super-resolution processing section uses internal drawing data held in an application that has drawn the image, for the super-resolution process. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein, in a super-resolution process for an image in which a main content image and an additional content image are overlaid, the super-resolution processing section refers to data regarding an α value, and executes different super-resolution processes for a region of the main content image and a region of the additional content image. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, wherein the super-resolution processing section refers to blur information included in the internal drawing data held in the application that has generated the image, and switches a super-resolution process for a blurred region in the image. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the super-resolution processing section refers to motion vector information regarding an object included in the image, and refrains from executing a super-resolution process on a region in which a moving object is drawn, or refrains from executing a super-resolution process on an edge portion of a region in which a moving object is drawn. ### Claim 9 (depends on 1) The image processing apparatus according to claim 1, wherein the super-resolution processing section refers to an artifact strength in the image, and enhances a smoothing process for a region where intense image degradation has occurred. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein, when a region of an internal process unit in a super-resolution process is referred to as a region of an analysis unit granularity, the super-resolution processing section refers to coding unit allocation information regarding compression encoding of the image, and allocates one region of an analysis unit granularity to one coding unit in the image. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein, when a region of an internal process unit in a super-resolution process is referred to as a region of an analysis unit granularity, in a case where a limited number of regions of an analysis unit granularity are allocated to each of a plurality of regions in the image, the super-resolution processing section allocates a large number of regions of an analysis unit granularity to a region for which a relatively high score is obtained by calculation based on a predetermined standard of a scene analysis, compared to a region for which a relatively low score is obtained. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein, when a region of an internal process unit in a super-resolution process is referred to as a region of an analysis unit granularity, in a case where a limited number of regions of an analysis unit granularity are allocated to each of a plurality of regions in the image, the super-resolution processing section allocates a large number of regions of an analysis unit granularity to a region for which a relatively high score is obtained by calculation based on a predetermined standard of a human skin inference process, compared to a region for which a relatively low score is obtained. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein, when a region of an internal process unit in a super-resolution process is referred to as a region of an analysis unit granularity, the super-resolution processing section sets a total allocation number of regions of the analysis unit granularity in the image to be great in a case where the image is an I frame, compared to the total allocation number in a case where the image is not an I frame. ### Claim 14 A server comprising: an image generating section that generates an image of an application; a scene-information acquiring section that acquires scene information indicating a feature in the image generated by the image generating section; and a transmission section that transmits data regarding the image and the scene information to a client terminal to thereby cause the client terminal to execute a super-resolution process for the image on a basis of the scene information; wherein the super-resolution processing section includes a scene analyzing section and a scene-information integrating section, and during the acquisition process, the scene analyzing section sequentially outputs to the scene-information integrating section results of scene analyses of partial images of the inputted image, the partial images having a plurality of different resolutions, without waiting for completion of all of the scene analyses, and when the outputted results are sufficient to generate the super-resolution image of the inputted image, the scene analyzing section abandons uncompleted ones of the scene analyses. ### Claim 15 (depends on 14) The server according to claim 14, further comprising: a compression encoding section that compression-encodes the image generated by the image generating section, wherein the scene-information acquiring section acquires scene information regarding the image, in parallel with the compression encoding of the image at the compression encoding section. ### Claim 16 (depends on 15) The server according to claim 15, further comprising: a scene analyzing section that analyzes contents in the image, in parallel with the compression encoding of the image at the compression encoding section, wherein the scene-information acquiring section acquires scene information regarding the image on a basis of an analysis result obtained by the scene analyzing section.