# US 2023/0054523 A1 · Enhancing 360-degree video using convolutional neural network (cnn)-based filter > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0054523 A1 - Document ID: US20230054523A1 - Status: Published application - Kind: A1 - Application number: 17/793,348 - Filed: 2020-02-17 - Published: 2023-02-23 - Earliest claimed: 2020-02-17 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Huan Dou; Lidong Xu; Xiaoxia Cai; Chen Wang; Yi-Jen Chiu - Web page: https://patentgrep.com/patent/US20230054523A1 ## CPC - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 5/77: Retouching; Inpainting; Scratch removal - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) - H04N 19/86: involving reduction of coding artifacts, e.g. of blockiness - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An example apparatus for enhancing video includes a decoder to decode a received 360-degree projection format video bitstream to generate a decoded 360-degree projection format video. The apparatus also includes a viewport generator to generate a viewport from the decoded 360-degree projection format video. The apparatus further includes a convolutional neural network (CNN)-based filter to remove an artifact from the viewport to generate an enhanced image. The apparatus further includes a displayer to send the enhanced image to a display. ## Claims ### Claim at position 1 -25. (canceled) ### Claim 26 An apparatus for enhancing video, the apparatus comprising: a decoder to decode a received 360-degree projection format video bitstream to generate a decoded 360-degree projection format video; a viewport generator to generate a viewport from the decoded 360-degree projection format video; a convolutional neural network (CNN)-based filter to remove an artifact from the viewport to generate an enhanced image; and a displayer to send the enhanced image to a display. ### Claim 27 (depends on 26) The apparatus of claim 26, wherein the decoder is to receive a compressed conventional video and decompress the conventional video, the CNN-based filter to remove a second artifact from the decompressed conventional video. ### Claim 28 (depends on 26) The apparatus of claim 26, wherein the CNN-based filter includes a CNN with a light complexity (LC) topology. ### Claim 29 (depends on 26) The apparatus of claim 26, wherein the CNN-based filter includes a CNN with a high complexity (HC) topology. ### Claim 30 (depends on 26) The apparatus of claim 26, wherein the CNN-based filter includes a CNN with a stride two convolutional layer and a stride two deconvolutional layer. ### Claim 31 (depends on 26) The apparatus of claim 26, wherein the CNN-based filter includes an out-loop filter. ### Claim 32 (depends on 26) The apparatus of claim 26, wherein the artifact includes a blocky artifact. ### Claim 33 (depends on 26) The apparatus of claim 26, wherein the artifact includes a ringing artifact. ### Claim 34 (depends on 26) The apparatus of claim 26, wherein the artifact includes a radial artifact. ### Claim 35 (depends on 26) The apparatus of claim 26, wherein the artifact includes a seam artifact. ### Claim 36 A method for enhancing video, the method comprising: receiving, via a processor, a 360-degree projection format video bitstream; decoding, via the processor, the 360-degree projection format video bitstream to generate a decoded 360-degree projection format video; generating, via the processor, a viewport from the decoded 360-degree projection format video; and removing, via the processor, an artifact from the generated viewport via a convolutional neural network (CNN)-based filter to generate an enhanced viewport. ### Claim 37 (depends on 36) The method of claim 36, wherein the generating of the viewport includes receiving view information from a display device and generating the viewport from the decoded 360-degree projection format video based on the view information. ### Claim 38 (depends on 36) The method of claim 36, wherein the removing of the artifact includes using a CNN with a low complexity topology. ### Claim 39 (depends on 36) The method of claim 36, wherein the removing of the artifact includes using a CNN with a high complexity topology. ### Claim 40 (depends on 36) The method of claim 36, wherein the removing of the artifact includes using a CNN-based filter with a stride two convolutional layer and a stride two deconvolutional layer. ### Claim 41 (depends on 36) The method of claim 36, further including: receiving a compressed conventional video bitstream; decompressing the conventional video bitstream to generate conventional video frames; and removing a second artifact from the conventional video frames via the CNN-based filter. ### Claim 42 (depends on 36) The method of claim 36, further including training the CNN-based filter to remove blocky artifacts using training pairs of ground truth images and corresponding images with blocky artifacts added. ### Claim 43 (depends on 36) The method of claim 36, further including training the CNN-based filter to remove ringing artifacts using training pairs of ground truth images and corresponding images with ringing artifacts added. ### Claim 44 (depends on 36) The method of claim 36, further including training the CNN-based filter to remove blocky radial using training pairs of ground truth images and corresponding images with radial artifacts added. ### Claim 45 (depends on 36) The method of claim 36, further including training the CNN-based filter to remove blocky artifacts using training pairs of ground truth images and corresponding images with seam artifacts added. ### Claim 46 At least one hard drive, optical drive, thumbdrive, array of drives, solid-state drive, random access memory (RAM), read only memory (ROM), flash memory, or dynamic random access memory (DRAM) comprising machine-readable instructions that, when executed, cause processor circuitry to at least: receive a 360-degree projection format video bitstream; decode the 360-degree projection format video bitstream to generate a decoded 360-degree projection format video; generate a viewport from the decoded 360-degree projection format video; and remove an artifact from the generated viewport via a convolutional neural network (CNN)-based filter to generate an enhanced viewport. ### Claim 47 (depends on 46) The at least one hard drive, optical drive, thumbdrive, array of drives, solid-state drive, RAM, ROM, flash memory, or DRAM of claim 46, wherein the machine-readable instructions are to cause the processor circuitry to generate the viewport by receiving view information from a display device and generating the viewport from the decoded 360-degree projection format video based on the view information. ### Claim 48 (depends on 46) The at least one hard drive, optical drive, thumbdrive, array of drives, solid-state drive, RAM, ROM, flash memory, or DRAM of claim 46, wherein the machine-readable instructions are to cause the processor circuitry to remove the artifact using a CNN with a high complexity topology. ### Claim 49 (depends on 46) The at least one hard drive, optical drive, thumbdrive, array of drives, solid-state drive, RAM, ROM, flash memory, or DRAM of claim 46, wherein the machine-readable instructions are to cause the processor circuitry to remove the artifact using a CNN-based filter with a stride two convolutional layer and a stride two deconvolutional layer. ### Claim 50 (depends on 46) The at least one hard drive, optical drive, thumbdrive, array of drives, solid-state drive, RAM, ROM, flash memory, or DRAM of claim 46, wherein the machine-readable instructions are to cause the processor circuitry to: receive a compressed conventional video bitstream; decompress the conventional video bitstream to generate conventional video frames; and remove a second artifact from the conventional video frames via the CNN-based filter.