# US 12,003,719 B2 · Method, apparatus and storage medium for image encoding/decoding using segmentation map > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,719 B2 - Document ID: US12003719B2 - Status: Granted - Kind: B2 - Application number: 17/534,770 - Filed: 2021-11-24 - Published: 2022-05-26 - Granted: 2024-06-04 - Earliest claimed: 2020-11-26 - Assignees: Electronics and Telecommunications Research Institute - Applicants: Electronics and Telecommunications Research Institute - Inventors: Hyon-Gon Choo; Hee-Kyung Lee; Han-Shin Lim; Jeong-Il Seo; Won-Sik Cheong - Web page: https://patentgrep.com/patent/US12003719B2 ## CPC - H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks - H04N 19/17: the unit being an image region, e.g. an object - G06T 7/11: Region-based segmentation - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06T 7/90: Determination of colour characteristics - H04N 19/12: Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - G06T 2207/10016: Video; Image sequence - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Disclosed herein are a method, an apparatus and a storage medium for image encoding/decoding using a segmentation map. A feature vector for an image may be extracted using a segmentation map. The image may be encoded using the segmentation map and the feature vector. An output stream from an encoding apparatus may include a video stream and a feature stream. An input stream to a decoding apparatus may include a video stream and a feature stream. The image may be reconstructed using a reconstructed segmentation map and a reconstructed feature vector. ## Claims ### Claim 1 An encoding method, comprising: generating a segmentation map image indicating a segmentation map of an input image generating a video stream including an encoded segmentation map image; generating a feature vector using the input image and the segmentation map image; generating a feature stream including an encoded feature vector; generating a reconstructed image using the segmentation map image and the feature vector; generating a residual image between the reconstructed image and the input image; generating an encoded residual image by encoding the residual image; and generating an output stream by performing multiplexing on the video stream and the feature stream, wherein the encoded segmentation map image is generated by encoding the segmentation map image, wherein the encoded feature vector is generated by encoding the feature vector, wherein the output stream includes a residual stream, and wherein the residual stream includes the encoded residual image. ### Claim 2 (depends on 1) The encoding method of claim 1, wherein the segmentation map is a region segmentation map indicating one or more regions of the input image. ### Claim 3 (depends on 1) The encoding method of claim 1, wherein information about a label of each object in the segmentation map image and a feature of the object are extracted. ### Claim 4 (depends on 3) The encoding method of claim 3, wherein the feature includes color and/or texture information of the object. ### Claim 5 (depends on 3) The encoding method of claim 3, wherein: the feature of the object includes one or more blocks, each of the one or more blocks includes position and color set, and the color set is representative color components of the corresponding block. ### Claim 6 (depends on 1) The encoding method of claim 1, wherein: the feature vector includes color map information, and the color map information includes pieces of color information of each of one or more objects in the input image. ### Claim 7 (depends on 6) The encoding method of claim 6, wherein the pieces of color information are pieces of color information of one or more blocks forming each object in the input image. ### Claim 8 (depends on 1) The encoding method of claim 1, wherein a filter is applied to the input image to generate the segmentation map. ### Claim 9 (depends on 1) A non-transitory computer-readable storage medium, comprising: a bitstream generated by the encoding method of claim 1. ### Claim 10 A decoding method, comprising: receiving an input stream; separating a video stream and a feature stream from the input stream by performing demultiplexing on the input stream, generating a segmentation map image; generating a feature vector; generating a reconstructed image using the segmentation map image and the feature vector; and generating a final image using the reconstructed image and a residual image, wherein the video stream includes an encoded segmentation map image, wherein the feature stream includes an encoded feature vector, wherein the segmentation map image is reconstructed by decoding the encoded segmentation map image, wherein the feature vector is reconstructed by decoding the encoded feature vector, wherein a residual stream is output from the input stream, and wherein the residual image is reconstructed by decoding an encoded residual image of the residual stream. ### Claim 11 (depends on 10) The decoding method of claim 10, wherein the reconstructed image is generated by a generator network for performing an image translation method based on deep learning. ### Claim 12 (depends on 10) The decoding method of claim 10, wherein information about a label of each object in the segmentation map image and a feature of the object are extracted. ### Claim 13 (depends on 12) The decoding method of claim 12, wherein the feature includes color and/or texture information of the object. ### Claim 14 A non-transitory computer-readable storage medium storing a bitstream for image decoding, wherein the bitstream comprises: an input stream, wherein the input stream comprises a video stream, a feature stream and a residual stream, wherein the video stream and the feature stream are separated from the input stream by performing demultiplexing on the input stream, wherein a segmentation map image is generated using the video stream, wherein the video stream includes an encoded segmentation map image, wherein the segmentation map image is reconstructed by decoding the encoded segmentation map image, wherein a feature vector is generated using the feature stream, wherein the feature stream includes an encoded feature vector, wherein the feature vector is reconstructed by decoding the encoded feature vector, wherein a reconstructed image is generated using the segmentation map image and the feature stream, wherein a residual image is reconstructed by decoding an encoded residual image of the residual stream, and wherein a final image is generated using the reconstructed image and the residual image.