# US 2024/0187582 A1 · Method for verifying video data encoded in an encoder unit

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0187582 A1
- Document ID: US20240187582A1
- Status: Published application
- Kind: A1
- Application number: 18/528,813
- Filed: 2023-12-05
- Published: 2024-06-06
- Earliest claimed: 2022-12-05
- Assignees: Matthias Auchmann
- Inventors: Matthias Auchmann
- Web page: https://patentgrep.com/patent/US20240187582A1

## CPC

- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567)
- H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147)
- H04N 19/17: the unit being an image region, e.g. an object
- H04N 19/172: the region being a picture, frame or field
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors}
- H04N 19/65: using error resilience
- H04N 19/82: involving filtering within a prediction loop
- H04N 19/86: involving reduction of coding artifacts, e.g. of blockiness
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present application relates to a method for verifying video data encoded in an encoder unit, wherein the encoder unit comprises a safe part and a non-safe part, wherein loop filtering in the encoder unit is disabled, and wherein the method (1) comprises the steps of: Acquiring video data in the safe part of the encoder unit (2,11,21); Encoding the acquired video data by a non-safe encoder in the non-safe part of the encoder unit (3,12,22); Decoding the encoded video data by a non-safe decoder in the non-safe part of the encoder unit (4,13,23); and Comparing the decoded video data with the acquired video data in the safe part of the encoder unit using at least one comparison algorithm (5,14,24), wherein if the decoded video data does not match the acquired video data exactly nor within an acceptable threshold for the comparison a safety focused reaction is initiated, and if the decoded video data matches the acquired video data exactly or within the acceptable threshold for the comparison a first safety sequence is generated based on the decoded video data in the safe part of the encoder unit and the encoded video data and the first safety sequence are transmitted to a decoder unit for decoding the encoded video data.

## Claims

### Claim 1

Method for verifying video data encoded in an encoder unit, wherein the encoder unit comprises a safe part and a non-safe part, wherein loop filtering in the encoder unit is disabled, and wherein the method (1) comprises the steps of:
Acquiring video data in the safe part of the encoder unit (2,11,21);
Encoding the acquired video data by a non-safe encoder in the non-safe part of the encoder unit (3,12,22);
Decoding the encoded video data by a non-safe decoder in the non-safe part of the encoder unit (4,13,23); and
Comparing the decoded video data with the acquired video data in the safe part of the encoder unit using at least one comparison algorithm (5,14,24), wherein if the decoded video data does not match the acquired video data exactly nor within an acceptable threshold for the comparison a safety focused reaction is initiated, and if the decoded video data matches the acquired video data exactly or within the acceptable threshold for the comparison a first safety sequence is generated based on the decoded video data in the safe part of the encoder unit and the encoded video data and the first safety sequence are transmitted to a decoder unit for decoding the encoded video data, wherein the step of comparing the decoded video data with the acquired video data is based on whether Rate Distortion Optimization is performed, wherein safe means that the corresponding part is safety-qualified and/or has a high integrity level and non-safe means that the corresponding part is not safety critical and does not have to be safety-qualified, wherein the safety focused reaction comprises flagging decoded video data that does not match the acquired video data exactly nor within the acceptable threshold for the comparison, respectively regions in the decoded video data that do not match the corresponding regions in the acquired video data exactly nor within the acceptable threshold for the comparison, wherein the flagged decoded video data respectively the flagged regions in the decoded video data are transmitted to the decoder unit, or flagging decoded video data that does not match the acquired video data exactly nor within the acceptable threshold for the comparison, respectively regions in the decoded video data that do not match the corresponding regions in the acquired video data exactly nor within the acceptable threshold for the comparison, wherein the flagged decoded video data respectively the flagged regions in the decoded video data are transmitted to the decoder unit and blanked in the decoder unit, or turning the corresponding source of video data off and/or deleting the video data acquired from the corresponding source of video data, and wherein the first safety sequence is one of a cryptographic hash, a checksum or a signature.

### Claim 2 (depends on 1)

Method according to claim 1, wherein the step (14) of comparing the decoded video data with the acquired video data comprises the steps of:
For each video frame in the acquired video data, respectively receiving a corresponding motion-predicted video frame from the non-safe encoder;
For each video frame in the acquired video data, respectively determining residual data for the corresponding video frame based on the acquired video frame and the corresponding motion-predicted video frame;
For each video frame in the acquired video frame, respectively block-wise transforming the residual data for the corresponding video frame, wherein transforming the residual data respectively includes utilizing the same transformation algorithms as used in the non-safe encoder to obtain transformed residual data, quantizing the transformed residual data using the same quantization step sizes as the non-safe encoder, inverse-quantizing the quantized residual data using the same quantization step sizes as the non-safe encoder, and inverse-transforming the inverse-quantized residual data by utilizing the same inverse transformation algorithms as used in the non-safe encoder;
For each video frame in the acquired video data, respectively determining a comparison frame by adding the corresponding motion-predicted video frame to the block-wise inverse-transformed residual data for the corresponding video frame; and
For each video frame in the acquired video data, respectively block-wise comparing the corresponding comparison frame with the corresponding decoded video frame.

### Claim 3 (depends on 1, 2)

Method according to claim 1 or 2, wherein the step (14) of comparing the decoded video data with the acquired video data comprises the steps of:
For each video frame in the acquired video data, respectively receiving a corresponding motion-predicted video frame including motion-predicted data from the non-safe encoder;
For each block of a plurality of blocks of the motion-predicted video frame, respectively determining residual data for the corresponding block by subtracting the motion-predicted data for the corresponding block from the corresponding data in the corresponding block of the acquired video frame, calculating an error metric for the determined residual data of the corresponding block and comparing the error metric to a predetermined threshold that is derived from the quantization parameter and the Lagrange multiplier lambda parameter in the Rate Distortion Optimization equation as used in the non-safe encoder, wherein the comparison is deemed successful for the corresponding block if the corresponding error metric does not exceed the predetermined threshold.

### Claim 4 (depends on 1)

Method according to claim 1, wherein the method further comprises a step of, for each transformed coefficient of each transformation carried out in the non-safe encoder, respectively computing a transformed coefficient threshold based on the quantization parameter used in the non-safe encoder, wherein the step (24) of comparing the decoded video data with the acquired video data comprises, for each video frame in the acquired video data, using the corresponding acquired video frame as a comparison frame and respectively block-wise comparing the corresponding comparison frame with the corresponding decoded video frame by dividing the corresponding decoded video frame into a plurality of blocks of pixels, for each of the plurality of blocks of pixels respectively transforming the corresponding block of pixels in the decoded video frame and a corresponding block of pixels of the corresponding comparison frame respectively using the same transformations as used in the non-safe encoder, and for each of the plurality of blocks of pixels respectively comparing the corresponding transformed block of pixels in the decoded video frame with the corresponding transformed block of pixels of the corresponding comparison frame by comparing the transformed coefficients of the transformation of the corresponding block of pixels of the decoded video frame with the transformed coefficients of the transformation of the corresponding block of pixels of the corresponding comparison frame using the transformed coefficient thresholds and accounting for numerical errors due to fixed point arithmetic in coefficients and rounding on all transformed coefficients.

### Claim 5 (depends on 1, 4)

Method according to claim 1 or 4, wherein the step (24) of comparing the decoded video data with the acquired video data comprises, for each video frame in the acquired video data, using the corresponding decoded video frame and subtracting the corresponding decoded video frame from the corresponding acquired video frame to determine an error frame including error data, block-wise calculating an error metric for the determined error data of the corresponding block in the error frame and comparing the corresponding error metric to a predetermined threshold that is derived from the quantization parameter and the Lagrange multiplier lambda parameter in the Rate Distortion Optimization equation as used in the non-safe encoder, wherein the comparison is deemed successful for the corresponding block if the corresponding error metric does not exceed the predetermined threshold.

### Claim 6 (depends on 2, 4)

Method according to claim 2 or 4, wherein the step (14,24) of block-wise comparing the corresponding comparison frame with the corresponding decoded video frame comprises block-wise comparing the corresponding comparison frame with the decoded video frame based on a first block size and block-wise comparing the corresponding comparison frame with the decoded video frame based on at least one second block size, wherein each of the first block size and the at least one second block size is a different block size, and wherein the decoded video data matches the acquired video data exactly or within the threshold for the comparison if for each pixel in the acquired video data respectively a block in the corresponding comparison frame containing the corresponding pixel matches a corresponding block in the corresponding decoded video frame containing the corresponding pixel based on at least one of the different block sizes.

### Claim 7 (depends on 3, 5)

Method according to claim 3 or 5, wherein the step of block-wise calculating an error metric for the determined error data of the corresponding block in the error frame and comparing the corresponding error metric to the predetermined threshold comprises block-wise calculating an error metric for the determined error data of the corresponding block in the error frame and comparing the corresponding error metric to the predetermined threshold based on a first block size and block-wise calculating an error metric for the determined error data of the corresponding block in the error frame and comparing the corresponding error metric to the predetermined threshold based on at least one second block size, wherein each of the first block size and the at least one second block size is a different block size, and wherein the comparison is deemed successful if for each pixel in the error frame the error metric does not exceeded the predetermined threshold for a block containing the corresponding pixel based on at least one of the different block sizes.

### Claim 8 (depends on 1, 2, 3, 4, 5, 6, 7)

Method for verifying video data transmitted from a source of video data to a processing unit for processing the video data by means of an encoder unit, wherein the encoder unit comprises a safe part and a non-safe part, and a decoder unit, wherein the decoder unit comprises a safe part and a non-safe part, wherein loop filtering in the encoder unit is disabled, wherein loop filtering in the decoder unit is disabled, and wherein the method (10,20) comprises the steps of:
Acquiring video data from the source of video data in the safe part of the encoder unit (11,21);
Encoding the video data in the non-safe part of the encoder unit (12,22);
Decoding the encoded video data by a non-safe decoder in the non-safe part of the encoder unit (13,23); and
Verifying the video data encoded in the encoder unit by a method for verifying video data encoded in an encoder unit according to one of claims 1 to 7;
Transmitting the encoded video data and a corresponding first safety sequence to the decoder unit (15,25);
Decoding the encoded video data by a non-safe decoder in the non-safe part of the decoder unit (16,26);
Generating a second safety sequence based on the decoded video data in the safe part of the decoder unit (17,27); and
Comparing the first safety sequence with the second safety sequence in the safe part of the decoder unit (18,28), wherein the decoded video data is transmitted to the processing unit for further processing the decoded video data if the first safety sequence corresponds to the second safety sequence, and wherein a safety focused reaction is initiated if the first safety sequence does not correspond to the second safety sequence, wherein the second safety sequence is one of a cryptographic hash, a checksum or a signature.

### Claim 9 (depends on 8)

Method according to claim 8, wherein the method further comprises filtering block artifacts of the decoded video data in the safe part of the decoder unit if the first safety sequence corresponds to the second safety sequence (19,29), wherein the filtered decoded video data is transmitted to the processing unit for further processing.

### Claim 10 (depends on 8, 9)

Method according to claim 8 or 9, wherein the step (15,25) of transmitting the first safety sequence to the decoder unit comprises transmitting the first safety sequence to the decoder unit over a safety layer.

### Claim 11

Encoder unit, wherein loop filtering in the encoder unit (32) is disabled, and wherein the encoder unit (32) comprises a safe acquiring unit (36), wherein the safe acquiring unit (36) is configured to acquire video data, a non-safe encoder (37), wherein the non-safe encoder (37) is configured to encode acquired video data, a non-safe decoder (38), wherein the non-safe decoder (38) is configured to decode the encoded video data, and a safe comparing unit (39), wherein the safe comparing unit (39) is configured to compare the decoded video data with the acquired video data using at least one comparison algorithm, and wherein the safe comparing unit (39) is configured to initiate a safety focused reaction if the decoded video data does not match the acquired video data exactly nor within an acceptable threshold for the comparison, and to generate a first safety sequence based on the decoded video data, wherein the encoded video data and the first safety sequence are transmitted to a decoder unit for decoding the encoded video data if the decoded video data matches the acquired video data exactly or within the acceptable threshold for the comparison, wherein the safe comparing unit is configured to compare the decoded video data with the acquired video data based on whether Rate Distortion Optimization is performed, wherein safe means that the corresponding part is safety-qualified and/or has a high integrity level and non-safe means that the corresponding part is not safety critical and does not have to be safety-qualified, wherein the safety focused reaction comprises flagging decoded video data that does not match the acquired video data exactly nor within the acceptable threshold for the comparison, respectively regions in the decoded video data that do not match the corresponding regions in the acquired video data exactly nor within the acceptable threshold for the comparison, wherein the flagged decoded video data respectively the flagged regions in the decoded video data are transmitted to the decoder unit, or flagging decoded video data that does not match the acquired video data exactly nor within the acceptable threshold for the comparison, respectively regions in the decoded video data that do not match the corresponding regions in the acquired video data exactly nor within the acceptable threshold for the comparison, wherein the flagged decoded video data respectively the flagged regions in the decoded video data are transmitted to the decoder unit and blanked in the decoder unit, or turning the corresponding source of video data off and/or deleting the video data acquired from the corresponding source of video data, and wherein the first safety sequence is one of a cryptographic hash, a checksum or a signature.

### Claim 12 (depends on 11)

Encoder unit according to claim 11, wherein the safe comparing unit (39) is configured to, for each video frame in the acquired video data, respectively receive a corresponding motion-predicted video frame from the non-safe encoder, respectively determine residual data for the corresponding video frame based on the acquired video frame and the corresponding motion-predicted video frame, and respectively block-wise transform the residual data for the corresponding video frame by utilizing the same transformation algorithms as used in the non-safe encoder to obtain transformed residual data, quantizing the transformed residual data using the same quantization step sizes as the non-safe encoder, inverse-quantizing the quantized residual data using the same quantization step sizes as the non-safe encoder, and inverse-transforming the inverse-quantized residual data by utilizing the same inverse transformation algorithms as used in the non-safe encoder, wherein the safe comparing unit is further configured to, for each video frame in the acquired video data, respectively determine a comparison frame by adding the corresponding motion-predicted video frame to the block-wise inverse-transformed residual data for the corresponding video frame, and to, for each video frame in the acquired video data, respectively block-wise compare the corresponding comparison frame with the corresponding acquired video frame.

### Claim 13 (depends on 11, 12)

Encoder unit according to claim 11 or 12, wherein the safe comparing unit is configured to, for each video frame in the acquired video data, respectively receive a corresponding motion-predicted video frame including motion-predicted data from the non-safe encoder, respectively, for each block of a plurality of blocks of the motion-predicted video frame, respectively determine residual data for the corresponding block by subtracting the motion-predicted data for the corresponding block from the corresponding data in the corresponding block of the acquired video frame, calculate an error metric for the determined residual data of the corresponding block and compare the error metric to a predetermined threshold that is derived from the quantization parameter and the Lagrange multiplier lambda parameter in the Rate Distortion Optimization equation as used in the non-safe encoder, wherein the comparison is deemed successful for the corresponding block if the corresponding error metric does not exceed the predetermined threshold.

### Claim 14 (depends on 11)

Encoder unit according to claim 11, wherein the safe comparing unit (39) is configured to, for each transformed coefficient of each transformation carried out in the non-safe encoder, compute a transformed coefficient threshold based on the quantization parameter used in the non-safe encoder, and to, for each video frame in the acquired video data, use the corresponding acquired video frame as a comparison frame and respectively block-wise compare the corresponding comparison frame with the corresponding decoded video frame by dividing the corresponding decoded video frame into a plurality of blocks of pixels, for each of the plurality of blocks of pixels respectively transform the corresponding block of pixels of the decoded video frame and a corresponding block of pixels of the corresponding comparison frame respectively using the same transformations as used in the non-safe encoder, and for each of the plurality of blocks of pixels respectively compare the corresponding transformed block of pixels of the decoded video frame with the corresponding transformed block of pixels of the corresponding comparison frame by comparing the transformed coefficients of the transformation of the corresponding block of pixels of the decoded video frame with the transformed coefficients of the transformation of the corresponding block of pixels of the corresponding comparison frame using the transformed coefficient thresholds and accounting for numerical errors due to fixed point arithmetic in coefficients and rounding on all transformed coefficients.

### Claim 15 (depends on 11, 14)

Encoder unit according to claim 11 or 14, wherein the safe comparing unit is configured to, for each video frame in the acquired video data, use the corresponding decoded video frame and subtracting the corresponding decoded video frame from the corresponding acquired video frame to determine an error frame including error data, block-wise calculate an error metric for the determined error data of the corresponding block in the error frame and compare the error metric to a predetermined threshold that is derived from the quantization parameter and the Lagrange multiplier lambda parameter in the Rate Distortion Optimization equation as used in the non-safe encoder, wherein the comparison is deemed successful for the corresponding block if the corresponding error metric does not exceed the predetermined threshold.

### Claim 16 (depends on 12, 14)

Encoder unit according to claim 12 or 14, wherein the safe comparing unit (39) is further configured to block-wise compare the corresponding comparison frame with the corresponding decoded video frame based on a first block size and to block-wise compare the corresponding comparison frame with the corresponding decoded video frame based on at least one second block size, wherein each of the first block size and the at least one second block size is a different block size, and wherein the decoded video data matches the acquired video data exactly or within the threshold for the comparison if for each pixel in the acquired video frame respectively a block in the corresponding comparison frame containing the corresponding pixel corresponds to a corresponding block in the corresponding decoded video frame containing the corresponding pixel based on at least one of the different block sizes.

### Claim 17 (depends on 13, 15)

Encoder unit according to claim 13 or 15, wherein the safe comparing unit (39) is configured to block-wise calculate an error metric for the determined error data of the corresponding block in the error frame and compare the corresponding error metric to the predetermined threshold based on a first block size and to block-wise calculate an error metric for the determined error data of the corresponding block in the error frame and compare the corresponding error metric to the predetermined threshold based on at least one second block size, wherein each of the first block size and the at least one second block size is a different block size, and wherein the comparison is deemed successful if for each pixel in the error frame the error metric does not exceeded the predetermined threshold for a block containing the corresponding pixel based on at least one of the different block sizes.

### Claim 18 (depends on 11, 12, 13, 14, 15, 16, 17)

Decoder unit, wherein loop filtering in the decoder unit is disabled, and wherein the decoder unit (34) comprises a receiver (41) to receive encoded video data, wherein the encoded video data has been verified by an encoder unit according to one of claims 11 to 17, and a corresponding first safety sequence that has been generated in the encoder unit in case of successful verification of the encoded video data, a non-safe decoder, wherein the non-safe decoder (42) is configured to decode the encoded video data, a safe safety sequence generating unit (43), wherein the safe safety sequence generating unit (43) is configured to generate a second safety sequence based on the decoded video data, and a safe safety sequence matching unit (44), wherein the safe safety sequence matching unit (44) is configured to compare the first safety sequence with the second safety sequence, wherein the safe safety sequence matching unit (44) is configured to allow transmission of the decoded video data to a processing unit for further processing the video data if the first safety sequence corresponds to the second safety sequence, and to initiate a safety focused reaction if the first safety sequence does not correspond to the second safety sequence, wherein the second safety sequence is one of a cryptographic hash, a checksum or a signature.

### Claim 19 (depends on 18)

Decoder unit according to claim 18, wherein the decoder unit (34) further comprises a safe filter unit (45) in the safe part of the decoder unit, wherein the safe filter unit is configured to filter block artifacts in the decoded video data if the first safety sequence corresponds to the second safety sequence.

### Claim 20 (depends on 11, 12, 13, 14, 15, 16, 17, 18, 19)

System for verifying video data transmitted from a source of video data to a processing unit for processing the video data, wherein the system (30) comprises a source of video data (31) for acquiring video data, an encoder unit (32) according to one of claims 11 to 17, a transmitting unit (33), a decoder unit (34) according to claim 18 or 19, and a processing unit (35) for processing the acquired video data, wherein the encoder unit (32) is configured to generate encoded video data acquired from the source of video data, wherein the transmitting unit (33) is configured to transmit the encoded video data from the encoder unit (32) to the decoder unit (34), wherein the decoder unit (34) is configured to decode the encoded video data, and wherein the processing unit (35) is configured to process the decoded video data.

### Claim 21 (depends on 20)

System according to claim 20, wherein the transmitting unit (33) is configured to transmit the first safety sequence to the decoder unit over a safety layer.
