# US 2014/0153651 A1 · Method and apparatus for reframing and encoding a video signal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0153651 A1
- Document ID: US20140153651A1
- Status: Published application
- Kind: A1
- Application number: 14/233,556
- Filed: 2012-06-25
- Published: 2014-06-05
- Earliest claimed: 2011-07-19
- Assignees: Thomson Licensing
- Applicants: Thomson Licensing
- Inventors: Fabrice Urban; Christel Chamaret; Philippe Guillotel
- Web page: https://patentgrep.com/patent/US20140153651A1

## CPC

- H04N 7/01: Conversion of standards {, e.g. involving analogue television standards or digital television standards processed at pixel level}
- H04N 19/115: Selection of the code volume for a coding unit prior to coding
- H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence)
- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/196: being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters (processing of motion vectors H04N19/513)
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Reframing is used to re-size an image or video content, e.g. for displaying video signals with a given aspect ratio on a display having a different aspect ratio. Window cropping parameters (position and size over time) are constrained in order to optimise the rate/distortion of the encoded output video signal. Initial reframing is improved by computing a saliency map representing a user attention map which considers the video coding context and by providing a macroblock coding efficiency cost map and then taking the saliency map or a combined saliency/coding cost map into account so that the coding cost for said reframed video signal is smaller than the coding cost for other candidate reframing windows' sizes and positions.

## Claims

### Claim at position 1

1-10. (canceled)

### Claim 11

Method for reframing an original video signal, followed by an encoding of the reframed video signal, said method including the steps:
computing a saliency map for a current picture of said original video signal, said saliency map representing a user attention map;
providing a macroblock coding efficiency cost map for said current picture of said original video signal;
selecting a candidate reframing window size and position within said current picture of said original video signal;
by using said saliency map or said combined saliency and macroblock coding efficiency cost map, determining in said original video signal a different position and/or size of the reframing window, for which different position and/or size the coding cost for said reframed video signal are smaller than the coding cost for said candidate reframing window size and position;
encoding said corresponding reframed video signal.

### Claim 12 (depends on 11)

Method according to claim 11, wherein a Kalman filter is used to control the variation of the position and the size of the reframing window due to variations of the content of the saliency map over time.

### Claim 13 (depends on 12)

Method according to claim 12, wherein a Kalman filter is used in two steps for each picture, for predicting a current state xk with respect to a previous state xk-1 and a state transition model so as to stabilise the reframing window parameters, and for correcting the prediction using a noisy measure, wherein state xk defines the position of the centre of the reframing window and its size,
x
k
=
[
centerColumn
centerLine
windowWidth
windowHeight
]
.

### Claim 14 (depends on 11)

Method according to claim 11, wherein said macroblock coding efficiency cost map is provided and, if the aspect ratio of the candidate reframing window resulting from the current content of said saliency map is different than the aspect ratio of said original video signal, the candidate reframing window is extended correspondingly in horizontal or vertical direction so as to achieve said aspect ratio of said original video signal, wherein the direction of said extension is performed such that for the current candidate reframing window the macroblock coding cost derived from the current macroblock coding efficiency cost map is reduced.

### Claim 15 (depends on 11)

Method according to claim 11, wherein said saliency map is merged with said macro-block cost map, such that an expensive coding cost macro-block decreases the corresponding saliency value in a final saliency map applied.

### Claim 16

Method according to claim H, wherein said encoding is a H.264/AVC encoding, wherein said encoding means are H.264/AVC encoding means.

### Claim 17 (depends on 16)

Method according to claim 16, wherein said re-framing is included within the encoding loop but the full picture is encoded, and wherein SEI messages are used to signal to the decoder a reframing window to be used.

### Claim 18 (depends on 11)

Method according to claim 11, wherein said reframing window is aligned to macroblock borders.

### Claim 19

Apparatus for reframing an original video signal, followed by an encoding of the reframed video signal, said apparatus including:
computing means which compute a saliency map for a current picture of said original video signal, and for providing a macroblock coding efficiency cost map for said current picture, said saliency map representing a user attention map;
selecting means which select a candidate reframing window size and position within said current picture of said original video signal, and for determining in said original video signal a different position and/or size of the reframing window, by using said saliency map or said combined saliency and macroblock coding efficiency cost map, for which different position and/or size the coding cost for said reframed video signal are smaller than the coding cost for said candidate reframing window size and position;
encoding means for said corresponding reframed video signal.

### Claim 20 (depends on 19)

Apparatus according to claim 19, wherein a Kalman filter is used to control the variation of the position and the size of the reframing window due to variations of the content of the saliency map over time.

### Claim 21 (depends on 20)

Apparatus according to claim 20, wherein a Kalman filter is used in two steps for each picture, for predicting a current state xk with respect to a previous state xk-1 and a state transition model so as to stabilise the reframing window parameters, and for correcting the prediction using a noisy measure, wherein state xk defines the position of the centre of the reframing window and its size,
x
k
=
[
centerColumn
centerLine
windowWidth
windowHeight
]
.

### Claim 22 (depends on 19)

Apparatus according to claim 19, wherein said macroblock coding efficiency cost map is provided and, if the aspect ratio of the candidate reframing window resulting from the current content of said saliency map is different than the aspect ratio of said original video signal, the candidate reframing window is extended correspondingly in horizontal or vertical direction so as to achieve said aspect ratio of said original video signal, wherein the direction of said extension is performed such that for the current candidate reframing window the macroblock coding cost derived from the current macroblock coding efficiency cost map is reduced.

### Claim 23 (depends on 19)

Apparatus according to claim 19, wherein said saliency map is merged with said macro-block cost map, such that an expensive coding cost macroblock decreases the corresponding saliency value in a final saliency map applied.

### Claim 24 (depends on 19)

Apparatus according to claim 19, wherein said encoding means is an H.264/AVC encoder.

### Claim 25 (depends on 24)

Apparatus according to claim 24, wherein said re-framing is included within the encoding loop but the full picture is encoded, and wherein SEI messages are used to signal to the decoder a reframing window to be used.

### Claim 26 (depends on 19)

Apparatus according to claim 19, wherein said reframing window is aligned to macroblock borders.

### Claim 27 (depends on 11)

An encoded digital video signal that was, before the encoding, reframed according to the method of claim 11.
