# US 2024/0177268 A1 · Electronic device performing scaling using artificial intelligence model and method for operating the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177268 A1
- Document ID: US20240177268A1
- Status: Published application
- Kind: A1
- Application number: 18/518,787
- Filed: 2023-11-24
- Published: 2024-05-30
- Earliest claimed: 2022-11-24
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Taewon DO; Yongtae KIM; Hoonjae LEE; Gautam TUSHAR; Hyeyoung JUN
- Web page: https://patentgrep.com/patent/US20240177268A1

## CPC

- G06T 3/4046: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/098: Distributed learning, e.g. federated learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- H04N 5/926: by pulse code modulation (H04N5/919 takes precedence)
- H04N 7/14: Systems for two-way working ({H04N7/12, } H04N7/173 take precedence)
- H04N 21/414: Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
- H04N 21/462: Content or additional data management e.g. creating a master electronic programme guide from data received from the Internet and a Head-end or controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An electronic device is provided. The electronic device includes memory, a camera module, a communication module, and at least one processor operatively connected to the memory, the camera module, and the communication module. The memory, when executed by the at least one processor, cause the electronic device to establish a call connection with a network based on the communication module, identify a first image captured based on the camera module, identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device, identify a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model, and transmit the second image through the call connection.

## Claims

### Claim 1

An electronic device comprising:
memory;
a camera module;
a communication module; and
at least one processor operatively connected to the memory, the camera module and the communication module,
wherein the memory, when executed by the at least one processor, cause the electronic device to:
establish a call connection with a network based on the communication module,
identify a first image captured based on the camera module,
identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device,
identify a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model, and
transmit the second image through the call connection based on the communication module.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image:
identify a first bit per pixel (BPP), obtained by dividing the first bitrate by a product of a first framerate associated with the first image and a resolution associated with the first image, as the first information.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image:
identify the first BPP as the first information associated with the first bitrate based on at least one first condition being met.

### Claim 4 (depends on 3)

The electronic device of claim 3, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image:
identify a second BPP obtained by dividing an average of sizes of a designated number of pre-encoded images by the resolution, as the first information, based on at least one second condition different from the at least one first condition being met or the at least one first condition being not met.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of transmitting the second image:
generate a bitstream by encoding the second image, and
transmit the bitstream through the call connection.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein the memory, when executed by the at least one processor, cause the electronic device to, identify the communication environment based on at least one of a one-way delay, a perceived bitrate, a packet loss rate, or a bandwidth.

### Claim 7 (depends on 1)

The electronic device claim 1,
wherein the artificial intelligence model for down-scaling includes, a first portion extracting a feature of the first image, a second portion extracting a feature of the first information, a multiplier cross-multiplying the feature of the first image and the feature of the first information, a third portion for enhancing a result of the cross-multiplying by the multiplier and configuring a residual image, a fourth portion for down-scaling the first image, and an adder for adding an output result of the third portion and an output result of the fourth portion, and
wherein a result of adding by the adder is provided as the second image.

### Claim 8 (depends on 7)

The electronic device of claim 7,
wherein the artificial intelligence model for down-scaling is a ResNet,
wherein the first portion includes at least one convolution layer,
wherein the second portion is a DenseNet,
wherein the third portion includes at least one convolution layer, and
wherein the fourth portion is a Bicubic down scaler.

### Claim 9 (depends on 1)

The electronic device of claim 1, wherein the memory, when executed by the at least one processor, cause the electronic device to,
identify training data including a first image, which is a high-resolution image, and first information associated with a bitrate,
identify a second image, which is a low-resolution image, output from a first AI model for down-scaling, based on inputting the first image and the first information to the first AI model,
identify a third image, which is a high-resolution image, output from a second AI model for up-scaling, based on inputting the second image and the first information to the second AI model,
identify a fourth image by down-scaling the first image,
identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and
train at least a portion of the first AI model and the second AI model based on the total loss.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein the memory, when executed by the at least one processor, cause the electronic device to,
identify a fifth image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model,
identify a sixth image by encoding the fifth image and decoding a result of the encoding,
identify a seventh image, which is a high-resolution image, output from the second AI model, based on inputting the sixth image and the first information to the second AI model,
identify an eighth image obtained by enhancing the first image,
identify a total loss based on the seventh image and the eighth image, and
train the second AI model based on the total loss.

### Claim 11

An electronic device comprising:
memory;
a display module;
a communication module; and
at least one processor operatively connected to the memory, the display module and the communication module,
wherein the memory, when executed by the at least one processor, cause the electronic device to:
establish a call connection with a network based on the communication module,
receive a first image through the call connection based on the communication module,
identify first information associated with a first bitrate corresponding to the first image, based on a communication environment between the network and the electronic device,
identify a second image corresponding to the first image output from an artificial intelligence model for up-scaling, trained to receive information associated with a low-resolution image and a bitrate as an input value to output a high-resolution image, by inputting the first image and the first information to the artificial intelligence model, and
control the display module to display at least a portion of the second image.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image:
identify a first bit per pixel (BPP), obtained by dividing the first bitrate by a product of a first frame rate associated with the first image and a resolution associated with the first image, as the first information.

### Claim 13 (depends on 12)

The electronic device of claim 12, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image:
identify the first BPP as the first information associated with the first bitrate based on at least one first condition being met.

### Claim 14 (depends on 13)

The electronic device of claim 13, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of identifying the first information associated with the first bitrate corresponding to the first image:
identify a second BPP obtained by dividing an average of sizes of a designated number of pre-encoded images by the resolution, as the first information, based on at least one second condition different from the at least one first condition being met or the at least one first condition being not met.

### Claim 15 (depends on 11)

The electronic device of claim 11, wherein the memory, when executed by the at least one processor, cause the electronic device to, as at least part of receiving the first image:
receive a bitstream through the call connection, and
identify the first image by decoding the bitstream.

### Claim 16 (depends on 11)

The electronic device of claim 11, wherein the memory, when executed by the at least one processor, cause the electronic device to:
identify the communication environment based on at least one of a one way delay, a perceived bitrate, a packet loss rate, or a bandwidth.

### Claim 17 (depends on 11)

The electronic device of claim 11, wherein the memory, when executed by the at least one processor, cause the electronic device to:
identify training data including a first image, which is a high-resolution image, and first information associated with a bitrate,
identify a second image, which is a low-resolution image, output from a first AI model for down-scaling, based on inputting the first image and the first information to the first AI model,
identify a third image, which is a high-resolution image, output from a second AI model for up-scaling, based on inputting the second image and the first information to the second AI model,
identify a fourth image by down-scaling the first image,
identify a total loss based on a first loss corresponding to the first image and the third image and a second loss corresponding to the second image and the fourth image, and
train at least a portion of the first AI model and the second AI model based on the total loss.

### Claim 18 (depends on 17)

The electronic device of claim 17, wherein the memory, when executed by the at least one processor, cause the electronic device to:
identify a fifth image, which is a low-resolution image, output from the first AI model, based on inputting the first image and the first information to the first AI model,
identify a sixth image by encoding the fifth image and decoding a result of the encoding,
identify a seventh image, which is a high-resolution image, output from the second AI model, based on inputting the sixth image and the first information to the second AI model,
identify an eighth image obtained by enhancing the first image,
identify a total loss based on the seventh image and the eighth image, and
train the second AI model based on the total loss.

### Claim 19 (depends on 18)

The electronic device of claim 18, wherein a second loss corresponding to the second image and the fourth image is a loss between images obtained by enhancing the second image and the fourth image.

### Claim 20

One or more non-transitory computer-readable storage media storing at least one computer-readable instruction that, when executed by at least one processor of an electronic device, configures the electronic device to perform operations, the operations comprising:
establishing a call connection with a network based on a communication module of the electronic device;
identifying a first image captured based on a camera module of the electronic device;
identifying first information associated with a first bit rate corresponding to the first image, based on a communication environment between the network and the electronic device;
identifying a second image corresponding to the first image output from an artificial intelligence model for down-scaling, trained to receive information associated with a high-resolution image and a bitrate as an input value to output a low-resolution image, by inputting the first image and the first information to the artificial intelligence model; and
transmitting the second image through the call connection based on the communication module.
