# US 12,003,859 B2 · Brightness adjustment method, and apparatus thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,003,859 B2
- Document ID: US12003859B2
- Status: Granted
- Kind: B2
- Application number: 17/953,857
- Filed: 2022-09-27
- Published: 2023-01-19
- Granted: 2024-06-04
- Earliest claimed: 2021-07-16
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Sawan Dineshbhai Kaulic; Sai Teja Kancharlapalli
- Web page: https://patentgrep.com/patent/US12003859B2

## CPC

- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- H04N 23/70: Circuitry for compensating brightness variation in the scene
- G06N 3/08: Learning methods
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 7/194: involving foreground-background segmentation
- H04N 5/57: Control of contrast or brightness
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 20/00: Machine learning
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of adjusting brightness of a foreground video displayed over a background in a user device is provided. The method includes determining enhancement factors for background sections in a background frame of the background, determining an enhancement mask for each foreground sub-frame of the foreground video based on enhancement factor of corresponding background section and determining enhanced brightness values for each foreground pixel in the foreground video based on the enhancement mask. The method of the disclosure provides real-time, clear and distinct viewing of an enhanced foreground video in varying brightness conditions of the background sections.

## Claims

### Claim 1

A user device for adjusting brightness of a foreground video displayed over a background on a display, the user device comprising:
a camera configured to capture a background frame of the background;
at least one processor configured to:
receive and process the foreground video and the background,
divide the background frame of the background into a plurality of background sub-frames and a foreground video frame of the foreground video into a plurality of foreground sub-frames,
determine an average brightness value of each foreground sub-frame of the plurality of foreground sub-frames and an average brightness value of each background sub-frame of the plurality of background sub-frames,
cluster similar background sub-frames to form a plurality of background sections based on the average brightness value of each background sub-frame,
determine a luminance factor for each background sub-frame of the plurality of background sub-frames,
determine an effective brightness value for each background section of the plurality of background sections based on the luminance factor and the average brightness value of each background sub-frame forming the background section,
determine an enhancement factor for each background section of the plurality of background sections based on the effective brightness value of the background section and an average of the average brightness value of each foreground sub-frame corresponding to each background sub-frame forming the background section,
generate an enhancement mask for each foreground sub-frame of the plurality of foreground sub-frames based on the enhancement factor for the background section corresponding to the foreground sub-frame, and
adjust the average brightness value of each foreground sub-frame of the plurality of foreground sub-frames based on the corresponding enhancement mask to generate an enhanced foreground video frame; and
a display that displays the enhanced foreground video frame over the background on the display of the user device.

### Claim 2 (depends on 1)

The user device of claim 1, wherein the at least one processor is further configured to identify the similar background sub-frames based on the average brightness values of the background sub-frames and a location of each background sub-frame within the background frame.

### Claim 3 (depends on 2)

The user device of claim 2, wherein the at least one processor is further configured to use at least one of artificial intelligence based heuristic techniques or deep learning based image segmentation to identify the similar background sub-frames.

### Claim 4 (depends on 1)

The user device of claim 1, further comprising a memory that stores the background, the foreground video, and the enhanced foreground video frames.

### Claim 5 (depends on 1)

The user device of claim 1, wherein the at least one processor is further configured to determine a dynamic threshold distance for each background sub-frame based on distances of the background sub-frame from nearest and farthest background sub-frames in the same background section of the plurality of background sections, and determine the luminance factor based on the dynamic threshold distance.

### Claim 6 (depends on 5)

The user device of claim 5, wherein the at least one processor is further configured to:
determine a distance DiF between the background sub-frame i and the farthest background sub-frame in the same background section,
determine a distance DiC between the background sub-frame i and the nearest background sub-frame in the same background section, determine the dynamic threshold distance Ti for the background sub-frame, the dynamic threshold distance Ti being an average of the distances DiF and DiC:
T
⁢
i
=
D
i
F
+
D
i
C
2
,
determine a number of background sub-frames of the same background section within the dynamic threshold distance Ti from the background sub-frame i as the luminance factor wi for the background sub-frame i.

### Claim 7 (depends on 1)

The user device of claim 1,
wherein an effective brightness value WCsection for a background section is determined as follows:
WC
section
=
∑
i
=
1
n
x
i
*
w
i
w
i
=
x
1
*
w
1
+
x
2
*
w
2
+
…
+
x
n
*
w
n
w
1
+
w
2
+
…
+
w
n
,
where:
n is a total number of background sub-frames in the background section,
xi is an average brightness value of a background sub-frame i, and
wi is a luminance factor of the background sub-frame i.

### Claim 8 (depends on 1)

The user device of claim 1, wherein the enhancement factor for a background section EFsection is determined as follows:
EF
section
=
WC
section
Average
⁢
of
⁢
average
⁢
brightness
⁢
value
⁢
of
⁢
each
⁢
foreground
sub
⁢
‐
⁢
frame
⁢
corresponding
⁢
to
⁢
each
⁢
background
sub
⁢
‐
⁢
frame
⁢
in
⁢
the
⁢
background
⁢
section
.

### Claim 9 (depends on 1)

The user device of claim 1, wherein the at least one processor is further configured to generate the adjusted brightness value of each foreground sub-frame of the plurality of foreground sub-frames based on the average brightness value of each foreground sub-frame of the plurality of foreground sub-frames and the enhancement factor of the corresponding background section in the enhancement mask for the foreground sub-frame.

### Claim 10 (depends on 9)

The user device of claim 9, wherein the at least processor is further configured to interpolate the adjusted brightness value of each foreground sub-frame of the plurality of foreground sub-frames to obtain interpolated brightness values for each foreground pixel of a plurality of foreground pixels in the foreground sub-frame.

### Claim 11 (depends on 10)

The user device of claim 10,
wherein the at least one processor is configured to adjust a V value of each foreground pixel of the plurality of foreground pixels, and
wherein the foreground pixel is represented in a Hue-Saturation-Value (HSV) format.

### Claim 12 (depends on 1)

The user device of claim 1, wherein the user device is an Augmented Reality (AR) device configured to display foreground video superimposed over the background to a user of the user device.

### Claim 13 (depends on 1)

The user device of claim 1, wherein the foreground video is translucent.

### Claim 14 (depends on 1)

The user device of claim 1, wherein the background is a live stream of a user environment obtained by the user device in real-time.

### Claim 15

A method of adjusting brightness of a foreground video displayed over a background on a display of a user device, the method comprising:
capturing, by a camera, a background frame of the background;
dividing, by at least one processor, the background frame of the background into a plurality of background sub-frames and a foreground video frame of the foreground video into a plurality of foreground sub-frames;
determining an average brightness value of each foreground sub-frame of the plurality of foreground sub-frames and an average brightness value of each background sub-frame of the plurality of background sub-frames;
clustering similar background sub-frames for forming a plurality of background sections based on the average brightness value of each background sub-frame;
determining a luminance factor for each background sub-frame of the plurality of background sub-frames;
determining an effective brightness value for each background section of the plurality of background sections based on the luminance factor and the average brightness value of each background sub-frame forming the background section;
determining an enhancement factor for each background section of the plurality of background sections based on the effective brightness value of the background section and an average of the average brightness value of each foreground sub-frame corresponding to each background sub-frame forming the background section;
generating an enhancement mask for each foreground sub-frame of the plurality of foreground sub-frames based on the enhancement factor for the background section corresponding to the foreground sub-frame;
adjusting the average brightness value of each foreground sub-frame of the plurality of foreground sub-frames based on the corresponding enhancement mask for generating an enhanced foreground video frame; and
displaying the enhanced foreground video frame over the background on the display of the user device.

### Claim 16 (depends on 15)

The method of claim 15, further comprising identifying, the similar background sub-frames based on the average brightness value of each background sub-frame of the plurality of background sub-frames and a location of each background sub-frame within the background frame.

### Claim 17 (depends on 16)

The method of claim 16, further comprising using at least one of artificial intelligence based heuristic techniques or deep learning based image segmentation for identifying the similar background sub-frames.

### Claim 18 (depends on 15)

The method of claim 15, wherein determining the luminance factor for each background sub-frame of the plurality of background sub-frames comprises:
determining a dynamic threshold distance for each background sub-frame based on distances of the background sub-frame from nearest and farthest background sub-frames in a same background section of the plurality of background sections; and
determining the luminance factor for the background sub-frame based on the dynamic threshold distance.

### Claim 19 (depends on 18)

The method of claim 18, wherein determining the luminance factor for the background sub-frame includes:
determining a distance DiF between the background sub-frame i and the farthest background sub-frame in the same background section;
determining a distance DiC between the background sub-frame i and the nearest background sub-frame in the same background section;
determining the dynamic threshold distance Ti for the background sub-frame i, the dynamic threshold distance Ti being an average of the distances DiF and DiC:
T
⁢
i
=
D
i
F
+
D
i
C
2
;
and
determining a number of background sub-frames of the same background section within the dynamic threshold distance Ti from the background sub-frame i as the luminance factor wi for the background sub-frame i.

### Claim 20

A video enhancement server in communication with a user device, the video enhancement server being configured to receive a foreground video and a background from the user device, the video enhancement server comprising:
a memory; and
at least one processor configured to:
divide a background frame of the background into a plurality of background sub-frames and a foreground video frame of the foreground video into a plurality of foreground sub-frames and to determine an average brightness value of each foreground sub-frame of the plurality of foreground sub-frames and an average brightness value of each background sub-frame of the plurality of background sub-frames,
cluster similar background sub-frames to form a plurality of background sections,
determine a luminance factor for each background sub-frame of the plurality of background sub-frames,
determine an effective brightness value for each background section of the plurality of background sections based on the luminance factor and the average brightness value of each background sub-frame forming the background section,
determine an enhancement factor for each background section of the plurality of background sections based on the effective brightness value of the background section and an average of the average brightness value of each foreground sub-frame corresponding to each background sub-frame forming the background section,
generate an enhancement mask for each foreground sub-frame of the plurality of foreground sub-frames based on the enhancement factor for the background section corresponding to the foreground sub-frame, and
adjust the average brightness value of each foreground sub-frame of the plurality of foreground sub-frames based on the corresponding enhancement mask to generate an enhanced foreground video frame.