# US 2021/0174473 A1 · Ai frame engine for mobile edge

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174473 A1
- Document ID: US20210174473A1
- Status: Published application
- Kind: A1
- Application number: 17/113,397
- Filed: 2020-12-07
- Published: 2021-06-10
- Earliest claimed: 2019-12-06
- Assignees: Mediatek Inc.
- Applicants: Mediatek Inc.
- Inventors: Jen Cheng LUNG; Pei-Kuei TSUNG; Chih-Wei CHEN; Yao-Sheng WANG; Shih-Che CHEN; Yu-Sheng LIN; Chih-Wen GOO; Shih-Chin LIN; Huang TSUNG-SHIAN; Ying-Chieh CHEN
- Web page: https://patentgrep.com/patent/US20210174473A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/254: involving subtraction of images
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T 2207/20224: Image subtraction
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Aspects of the disclosure provide a device for processing frames with aliasing artifacts. For example, the device can include a motion estimation circuit, a warping circuit coupled to the motion estimation circuit, and a temporal decision circuit coupled to the warping circuit. The motion estimation circuit can estimate a motion value between a current frame and a previous frame. The warping circuit can warp the previous frame based on the motion value such that the warped previous frame is aligned with the current frame and determine whether the current frame and the warped previous frame are consistent. The temporal decision circuit can generate an output frame, the output frame including either the current frame and the warped previous frame when the current frame and the warped previous frame are consistent, or the current frame when the current frame and the warped previous frame are not consistent.

## Claims

### Claim 1

A device for processing frames with aliasing artifacts, comprising:
a motion estimation circuit configured to estimate a motion value between a current frame and a previous frame;
a warping circuit coupled to the motion estimation circuit that is configured to warp the previous frame based on the motion value such that the warped previous frame is aligned with the current frame and to determine whether the current frame and the warped previous frame are consistent; and
a temporal decision circuit coupled to the warping circuit that is configured to generate an output frame, the output frame including either the current frame and the warped previous frame when the current frame and the warped previous frame are consistent, or the current frame when the current frame and the warped previous frame are not consistent.

### Claim 2 (depends on 1)

The device of claim 1, further comprising:
a frame processor coupled to the temporal decision circuit that is configured to process the output frame.

### Claim 3 (depends on 2)

The device of claim 2, wherein the frame processor comprises:
a super-resolution (SR) and anti-aliasing (AA) engine configured to enhance resolution and remove aliasing artifacts of the output frame to generate a first high-resolution frame with aliasing artifacts and a second high-resolution frame with aliasing artifacts removed.

### Claim 4 (depends on 3)

The device of claim 3, wherein the frame processor further comprises an attention reference frame generator coupled to the SR and AA engine that is configured to generate an attention reference frame based on the first high-resolution frame and the second high-resolution frame.

### Claim 5 (depends on 4)

The device of claim 4, wherein the frame processor further comprises an artificial intelligent (AI) neural network (NN) coupled to the attention reference frame generator that is configured to remove aliasing artifacts of a frame based on the attention reference frame.

### Claim 6 (depends on 1)

The device of claim 1, further comprising:
a frame fusion engine coupled to the temporal decision circuit that is configured to fuse the current frame and the warped previous frame.

### Claim 7 (depends on 6)

The device of claim 6, wherein the frame fusion engine fuses the current frame and the warped previous frame by concatenating the warped previous frame to the current frame in a channel-wise manner.

### Claim 8 (depends on 1)

The device of claim 1, wherein the motion estimation circuit estimates the motion value between the current frame and the previous frame using a sum of absolute difference (SAD) method.

### Claim 9 (depends on 8)

The device of claim 8, wherein the warping circuit warps the previous frame based on the motion value when the previous frame after being warped the motion value matches the current frame.

### Claim 10 (depends on 1)

The device of claim 1, wherein the warping circuit warps the previous frame based on the motion value when the motion value is less than a motion threshold.

### Claim 11 (depends on 10)

The device of claim 10, wherein the temporal decision circuit is further coupled to the motion estimation circuit, the temporal decision circuit generating the output frame that includes the current frame when the motion value is not less than the motion threshold.

### Claim 12 (depends on 1)

The device of claim 1, wherein the motion estimation circuit is a neural network (NN).

### Claim 13

A frame processor for processing frames with aliasing artifacts, comprising:
a super-resolution (SR) and anti-aliasing (AA) engine configured to enhance resolution and remove aliasing artifacts of a frame to generate a first high-resolution frame with aliasing artifacts and a second high-resolution frame with aliasing artifacts removed; and
an attention reference frame generator coupled to the SR engine and the AA engine that is configured to generate an attention reference frame based on the first high-resolution frame and the second high-resolution frame.

### Claim 14 (depends on 13)

The frame processor of claim 13, further comprising an artificial intelligence (AI) neural network (NN) coupled to the attention reference frame generator that is configured to remove aliasing artifacts of a frame based on the attention reference frame.

### Claim 15 (depends on 13)

The frame processor of claim 13, wherein the SR and AA engine includes an SR engine that is an AI SR engine, the first and second high-resolution frames having their resolution enhanced by the AI SR engine.

### Claim 16 (depends on 13)

The frame processor of claim 13, wherein the SR and AA engine includes an AI engine that is an AI AA engine, the second high-resolution frame having the aliasing artifacts removed by the AI AA engine.

### Claim 17

A method for processing frames with aliasing artifacts, comprising:
estimating a motion value between a current frame and a previous frame;
warping the previous frame based on the motion value such that the warped previous frame is aligned with the current frame; and
generating an output frame, the output frame including either the current frame and the warped previous frame when the current frame and the warped previous frame are consistent or the current frame when the current frame and the warped previous frame are not consistent.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
enhancing resolution and removing aliasing artifacts of the output frame to generate a first high-resolution frame with aliasing artifacts and a second high-resolution frame with aliasing artifacts removed.

### Claim 19 (depends on 18)

The method of claim 18, further comprising generating an attention reference frame based on the first high-resolution frame and the second high-resolution frame.

### Claim 20 (depends on 19)

The method of claim 19, further comprising removing aliasing artifacts of a frame based on the attention reference frame.
