# US 2006/0039617 A1 · Method and assembly for video encoding, the video encoding including texture analysis and texture synthesis, and corresponding computer program and corresponding computer-readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2006/0039617 A1
- Document ID: US20060039617A1
- Status: Published application
- Kind: A1
- Application number: 11/208,103
- Filed: 2005-08-19
- Published: 2006-02-23
- Earliest claimed: 2003-02-28
- Assignees: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Inventors: Bela Makai; Patrick Ndjiki-Nya; Heiko Schwarz; Aljoscha Smolic; Thomas Wiegand
- Web page: https://patentgrep.com/patent/US20060039617A1

## CPC

- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- H04N 19/20: using video object coding
- H04N 19/27: involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
- H04N 19/537: Motion estimation other than block-based
- H04N 19/61: in combination with predictive coding
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The invention relates to a method and assembly for video coding comprising a texture analysis, texture synthesis, corresponding computer program and a computer-readable recording medium. Said invention can be used, in particular for reducing a data rate during a video data transmission. An analysis of video scene texture by an encoder is performed in such a way that synthesized texture areas are determined. The video scene coding is carried out with the aid of data for said synthesized texture areas. Information on the texture of said areas, metadata used for describing the determined areas and the synthesized texture are generated. Coded data and metadata are evaluated by a decoder in such a way that the video scenes are reconstructed by synthetically generated textures by means of evaluation of metadata for determined areas.

## Claims

### Claim 1

A method for video encoding, comprising:
performing a texture analysis of video scenes to identify areas of synthesizable textures;
encoding the video scenes and generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas; and
ensuring temporal consistency of recognizing synthesizable textures in a sequence of frames by means of a texture catalogue, by
storing the synthesizable textures of the identified areas of synthesizable textures in a first frame of the sequence in the texture catalogue in order to initialize same;
comparing the synthesizable textures of the identified areas of synthesizable textures in the following frames of the sequence with the synthesizable textures stored in the texture catalogue;
in the event of a match, assigning the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence to the respective synthesizable texture stored in the texture catalogue; and
in the event of no match, storing the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence in the texture catalogue.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein performing the texture analysis comprises subdividing frames into blocks using a multi-resolution quadtree.

### Claim 3 (depends on 2)

The method as claimed in claim 2, wherein the subdividing of a block is broken off if
the blocks resulting from the next subdivision step exhibit similar color and/or texture properties as this block, or
the sizes of blocks of the next subdivision step fall short of a predefined value.

### Claim 4 (depends on 3)

The method as claimed in claim 3, wherein blocks, the subdividing of which has been broken off because of similar color and/or texture properties, are marked as homogeneous blocks, and the blocks which differ from the homogeneous blocks once the subdividing of the frame has been completed, are marked as non-classifiable blocks.

### Claim 5 (depends on 3)

The method as claimed in claim 3, wherein after the subdivision of a frame, a similarity assessment is performed for those blocks whose subdivision was broken off because of similar color and/or texture properties, and wherein similar blocks are combined into synthesizable areas.

### Claim 6 (depends on 5)

The method as claimed in claim 5, wherein the similarity of blocks is assessed on the basis of MPEG-7 descriptors.

### Claim 7 (depends on 6)

The method as claimed in claim 6, wherein the similarity of blocks is assessed on the basis of the MPEG-7 descriptors “Edge Histogram” texture descriptor (EH) and/or “SCalable Color” descriptor (SCC).

### Claim 8 (depends on 1)

The method as claimed in claim 1, wherein the identified areas of synthesizable textures of a frame of a video sequence are matched with areas of synthesizable textures of preceding frames of the video sequence.

### Claim 9 (depends on 1)

The method as claimed in claim 1, wherein the first and last frames (the so-called key frames) of a “Group of Frames” (GoF) are encoded on the basis of a mean squared error method (MSE based), and intervening B frames comprising identified synthesizable areas are partially synthesized.

### Claim 10 (depends on 1)

The method as claimed in claim 1, wherein synthesizable areas in the partially synthesized frames are adapted to corresponding texture areas in the key frames by suitable warping.

### Claim 11 (depends on 10)

The method as claimed in claim 10, wherein the warping is performed by means of a planar perspective motion model described by the following equations:
x′=[(a1+a3x+a4y)/(1+a7x+a8y)]+x,
y′=[(a2+a5x+a6y)/(1+a7x+a8y)]+y,
wherein
(x, y) represent the coordinates of the starting point,
(x′, y′) represent the transformed coordinates of the starting point, and
a1, . . . , a8 represent model parameters.

### Claim 12 (depends on 10)

The method as claimed in claim 10, wherein the warping causes the texture of the first or last frames of the current GoF to be warped in the direction of the synthesizable texture area identified, wherein each such texture area is associated a motion parameter set and a control parameter, the control parameter indicating whether the first or last frame of a GoF is used for texture synthesis.

### Claim 13 (depends on 12)

The method as claimed in claim 12, wherein for texture synthesis, that frame of a GoF is used for which a smaller differential signal is determined between the synthesized and the original texture regions.

### Claim 14 (depends on 1)

The method as claimed in claim 1, wherein the following are generated per texture area as meta data of areas with synthesizable textures:
a segmentation mask,
a motion parameter set, and/or
a control parameter.

### Claim 15 (depends on 1)

The method as claimed in claim 1, further comprising, at the decoder side, the steps of assessing the encoded data and meta data, and reconstructing the video scenes by synthetically generating textures for areas identified by assessing the meta data.

### Claim 16

An apparatus for video encoding, comprising:
an analyzer for performing a texture analysis of video scenes to identify areas of synthesizable textures;
an encoder for encoding the video scenes and a generator for generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas; and
a unit for ensuring temporal consistency of recognizing synthesizable textures in a sequence of frames by means of a texture catalogue, by
storing the synthesizable textures of the identified areas of synthesizable textures in a first frame of the sequence in the texture catalogue in order to initialize same;
comparing the synthesizable textures of the identified areas of synthesizable textures in the following frames of the sequence with the synthesizable textures stored in the texture catalogue;
in the event of a match, assigning the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence to the respective synthesizable texture stored in the texture catalogue; and
in the event of no match, storing the respective synthesizable texture of an identified area of synthesizable texture among the following frames of the sequence in the texture catalogue.

### Claim 17 (depends on 1)

A computer program enabling a computer, once it has been loaded into the computer's memory, to perform a method as claimed in claim 1.

### Claim 18 (depends on 1)

A computer-readable storage medium having a program stored thereon which enables the computer, once it has been loaded into the computer's memory, to perform a method as claimed in claim 1.

### Claim 19

A method for video encoding, comprising:
performing a texture analysis of video scenes to identify areas of synthesizable textures;
encoding the video scenes and generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas,
the step of generating the meta data comprising the step of estimating motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, the motion parameters being part of the meta data.

### Claim 20 (depends on 19)

The method as claimed in claim 19, wherein performing the texture analysis comprises subdividing frames into blocks using a multi-resolution quadtree.

### Claim 21 (depends on 20)

The method as claimed in claim 20, wherein the subdividing of a block is broken off if
the blocks resulting from the next subdivision step exhibit similar color and/or texture properties as this block, or
the sizes of blocks of the next subdivision step fall short of a predefined value.

### Claim 22 (depends on 21)

The method as claimed in claim 21, wherein blocks, the subdividing of which has been broken off because of similar color and/or texture properties, are marked as homogeneous blocks, and the blocks which differ from the homogeneous blocks once the subdividing of the frame has been completed, are marked as non-classifiable blocks.

### Claim 23 (depends on 21)

The method as claimed in claim 21, wherein after the subdivision of a frame, a similarity assessment is performed for those blocks whose subdivision was broken off because of similar color and/or texture properties, and wherein similar blocks are combined into synthesizable areas.

### Claim 24 (depends on 23)

The method as claimed in claim 23, wherein the similarity of blocks is assessed on the basis of MPEG-7 descriptors.

### Claim 25 (depends on 24)

The method as claimed in claim 24, wherein the similarity of blocks is assessed on the basis of the MPEG-7 descriptors “Edge Histogram” texture descriptor (EH) and/or “SCalable Color” descriptor (SCC).

### Claim 26 (depends on 19)

The method as claimed in claim 19, wherein the identified areas of synthesizable textures of a frame of a video sequence are matched with areas of synthesizable textures of preceding frames of the video sequence.

### Claim 27 (depends on 19)

The method as claimed in claim 19, wherein the first and last frames (the so-called key frames) of a “Group of Frames” (GoF) are encoded on the basis of a mean squared error method (MSE based), and intervening B frames comprising identified synthesizable areas are partially synthesized.

### Claim 28 (depends on 27)

The method as claimed in claim 27, wherein the temporal consistency of recognizing synthesizable textures of a Group of Frames (GoF) is ensured by means of a texture catalogue.

### Claim 29 (depends on 19)

The method as claimed in claim 19, wherein the warping is performed by means of a planar perspective motion model described by the following equations:
x′=[(a1+a3x+a4y)/(1+a7x+a8y)]+x,
y′=[(a2+a5x+a6y)/(1+a7x+a8y)]+y,
wherein
(x, y) represent the coordinates of the starting point,
(x′, y′) represent the transformed coordinates of the starting point, and
a1, . . . , a8 represent model parameters.

### Claim 30 (depends on 29)

The method as claimed in claim 29, wherein the warping causes the texture of the first or last frames of the current GoF to be warped in the direction of the synthesizable texture area identified, wherein each such texture area is associated a motion parameter set and a control parameter, the control parameter indicating whether the first or last frame of a GoF is used for texture synthesis.

### Claim 31 (depends on 30)

The method as claimed in claim 30, wherein for texture synthesis, that frame of a GoF is used for which a smaller differential signal is determined between the synthesized and the original texture regions.

### Claim 32 (depends on 19)

The method as claimed in claim 19, wherein the following are generated per texture area as meta data of areas with synthesizable textures:
a segmentation mask,
a motion parameter set, and/or
a control parameter.

### Claim 33

A method for video decoding, comprising:
assessing encoded data of video scenes and meta data for describing identified areas of synthesizable textures in the video scenes and for describing the synthesizable textures of these areas; and
reconstructing the video scenes by synthetically generating synthetically generated textures for the areas identified,
wherein the meta data comprise motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, and
wherein the step of reconstructing comprises the step of warping the corresponding texture areas in the first or last frames of the group in the direction of the adapted synthesizable areas in the frames of the Group of Frames using the motion parameters.

### Claim 34

An assembly for video encoding, comprising:
an analyzer for performing a texture analysis of video scenes to identify areas of synthesizable textures;
an encoder for encoding the video scenes and a generator for generating meta data for describing the areas identified and for describing the synthesizable textures using information on identified areas of synthesizable textures, and information on the textures of these areas; and
the generator for generating the meta data being configured to estimate motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, the motion parameters being part of the meta data.

### Claim 35

An assembly for video decoding, comprising:
an assessor for assessing encoded data of video scenes and meta data for describing identified areas of synthesizable textures in the video scenes and for describing the synthesizable textures of these areas; and
a reconstructor for reconstructing the video scenes by synthetically generating synthetically generated textures for the areas identified,
wherein the meta data comprise motion parameters describing a warping so as to adapt synthesizable areas in frames of a Group of Frames to corresponding texture areas in first or last frames of this group by means of the warping, and
wherein the reconstructor is configured to warp the corresponding texture areas in the first or last frames of the group in the direction of the adapted synthesizable areas in the frames of the Group of Frames using the motion parameters.

### Claim 36 (depends on 19)

A computer program enabling a computer, once it has been loaded into the computer's memory, to perform a method as claimed in claim 19.

### Claim 37 (depends on 33)

A computer program enabling a computer, once it has been loaded into the computer's memory, to perform a method as claimed in claim 33.

### Claim 38 (depends on 19)

A computer-readable storage medium having a program stored thereon which enables the computer, once it has been loaded into the computer's memory, to perform a method as claimed in claim 19.

### Claim 39 (depends on 33)

A computer-readable storage medium having a program stored thereon which enables the computer, once it has been loaded into the computer's memory, to perform a method as claimed in claim 33.
