# US 12,118,693 B2 · Method and electronic device for capturing media using under display camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,693 B2
- Document ID: US12118693B2
- Status: Granted
- Kind: B2
- Application number: 17/950,331
- Filed: 2022-09-22
- Published: 2023-01-26
- Granted: 2024-10-15
- Earliest claimed: 2021-07-21
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Alok Shankarlal Shukla; Manoj Kumar; Saketh Sharma; Woojhon Choi; Wonjoon Do; Jaesung Choi
- Web page: https://patentgrep.com/patent/US12118693B2

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06V 10/82: using neural networks
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/73: Deblurring; Sharpening
- G06T 5/80: Geometric correction
- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/141: Control of illumination
- G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19)
- G06V 10/16: {using multiple overlapping images; Image stitching}
- G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- H04N 5/265: Mixing
- H04N 23/53: of electronic viewfinders, e.g. rotatable or detachable
- H04N 23/617: Upgrading or updating of programs or applications for camera control
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/08: Learning methods
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10144: Varying exposure
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/20208: High dynamic range [HDR] image processing
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An electronic device includes a UDC and a UDC controller configured to determine an optimal number of frames required to be captured for a scene to compensate at least one parameter to optimize an output media using the UDC, obtain a multi-frame fusion media by performing at least one multi-frame fusion on the determined optimal number of frames, perform a light source spread correction on the multi-frame fusion media, and optimize the output media based on the light source spread correction on the multi-frame fusion media.

## Claims

### Claim 1

An electronic device comprising:
an under display camera (UDC); and
a UDC controller configured to:
determine an optimal number of frames required to be captured for a scene to compensate at least one parameter to optimize an output media using the UDC;
obtain a multi-frame fusion media by performing at least one multi-frame fusion on the determined optimal number of frames;
perform a light source spread correction on the multi-frame fusion media; and
optimize the output media based on the light source spread correction on the multi-frame fusion media.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the UDC controller is further configured to determine the optimal number of frames by:
receiving at least one input frame and a metadata information of the scene to be captured using the UDC;
measuring at least one ambient light parameter of the scene;
determining a light transmission loss through a display panel associated with the UDC by analyzing at least one input frame in connection with at least one ambient light parameter;
identifying at least one intrinsic camera parameter associated with the at least one input frame and a type of the scene; and
determining the optimal number of frames required to be captured for the scene to compensate for the light transmission loss and the at least one ambient light parameter based on the at least one intrinsic camera parameter associated with the at least one input frame and the type of the scene.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the at least one intrinsic camera parameter comprises at least one of an exposure time, a gain, a color, temperature and a focus region, and
wherein the type of the scene comprises at least one of an indoor environment, an outdoor environment, a mix lighting condition, a bright day light, and low light condition.

### Claim 4 (depends on 2)

The electronic device of claim 2, wherein the UDC controller is further configured to:
determine at least one display panel characteristic associated with the UDC for capturing each frame among the optimal number of frames;
apply the measured light transmission loss, the at least one ambient light parameter; and the display panel characteristics for capturing each frame among the optimal number of frames; and
automatically modify at least one media capture setting of the UDC by applying the measured light transmission loss, the at least one ambient light parameter and the at least one display panel characteristic.

### Claim 5 (depends on 4)

The electronic device of claim 4, wherein the UDC controller is further configured to determine the at least one display panel characteristic associated with the UDC by:
obtaining at least one intrinsic camera parameter associated with the at least one input frame; and
determining the at least one display panel characteristic associated with the UDC based on the at least one intrinsic camera parameter.

### Claim 6 (depends on 4)

The electronic device of claim 4, wherein the at least one display panel characteristic comprises a diffraction pattern, a point spread function of the display panel, and a modulation transfer function (MTF) of the display panel.

### Claim 7 (depends on 2)

The electronic device of claim 2, wherein the UDC controller is further configured to determine the light transmission loss by:
analyzing at least one preview frame from the UDC, and
estimating the light transmission loss for the display panel based on the analyzing of the at least one preview frame.

### Claim 8 (depends on 1)

The electronic device of claim 1, wherein the UDC controller is further configured to obtain the multi-frame fusion media by:
determining whether a dominant light source is present in the scene;
performing at least one of:
analyzing the scene to determine the multi-frame fusion media to be formed with at least one input frame with multiple exposure values upon determining the dominant light source is present in the scene, or
analyzing the scene to determine the multi-frame fusion media to be formed with the at least one input frame with a same exposure value upon determining the dominant light source is not present in the scene;
estimating a diffraction model on the multi-frame fusion media using an artificial intelligence (AI) model; and
performing an inversion operation to compensate for a diffraction loss in the multi-frame fusion media using the estimated diffraction model.

### Claim 9 (depends on 8)

The electronic device of claim 8, wherein the diffraction model is estimated on the multi-frame fusion media based on transmission characteristics of a display panel and a light transmission loss.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein the UDC controller is further configured to perform the light source spread correction on the multi-frame fusion media by:
analyzing the multi-frame fusion media to determine a type of the scene and localize a light source in the scene;
creating a segmentation mask for the localized light source in the scene;
applying a light spread correction for the localized light source in the scene using the segmentation mask; and
performing a content aware sharpness improvement to enhance the output media based on the applied light spread correction.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the light spread correction is applied for the localized light source in the scene using an artificial intelligence (AI) model, and wherein the AI model is configured to learn a light spread function of a display panel over a period of time.

### Claim 12 (depends on 1)

The electronic device of claim 1, wherein the at least one parameter comprises at least one of a light transmission loss associated with the scene, an ambient light parameter associated with the scene, a display panel characteristic associated with the UDC, an exposure time associated with at least one input frame, a gain associated with the at least one input frame, a temperature associated with the at least one input frame and focus region, and an environment lighting condition.

### Claim 13

An electronic device comprising:
an under display camera (UDC); and
a UDC controller configured to:
receive at least one input frame and a metadata information of a scene to be captured using the UDC;
measure at least one ambient light parameter of the scene;
determine a light transmission loss through a display panel associated with the UDC by analyzing the at least one input frame in connection with at least one measured ambient light parameter;
identify at least one intrinsic camera parameter associated with the at least one input frame and a type of the scene;
determine an optimal number of frames required to be captured for the scene to compensate for the light transmission loss and at least one ambient light parameter based on the at least one intrinsic camera parameter associated with the at least one input frame and the type of the scene;
determine at least one display panel characteristic associated with the UDC;
apply the measured light transmission loss, the at least one ambient light parameter and the display panel characteristics for capturing each frame among the optimal number of frames;
automatically modify at least one media capture setting of the UDC by applying the measured light transmission loss, the at least one ambient light parameter and the at least one display panel characteristic; and
optimize a media capture for the UDC in the electronic device based on the at least one modified media capture setting of the UDC.

### Claim 14

A method for capturing media using an under display camera (UDC) in an electronic device, comprising:
determining, by the electronic device, an optimal number of frames required to be captured for a scene to compensate at least one parameter to optimize an output media using the UDC;
obtaining, by the electronic device, a multi-frame fusion media by performing at least one multi-frame fusion on the determined optimal number of frames;
performing, by the electronic device, a light source spread correction on the multi-frame fusion media; and
optimizing, by the electronic device, the output media based on the light source spread correction on the multi-frame fusion media.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining, by the electronic device, the optimal number of frames comprises:
receiving, by the electronic device, at least one input frame and a metadata information of the scene to be captured using the UDC;
measuring, by the electronic device, at least one ambient light parameter of the scene;
determining, by the electronic device, a light transmission loss through a display panel associated with the UDC by analyzing the at least one input frame in connection with at least one measured ambient light parameter;
identifying, by the electronic device, at least one intrinsic camera parameter associated with the at least one input frame and a type of the scene; and
determining, by the electronic device, the optimal number of frames required to be captured for the scene to compensate for the light transmission loss and the at least one ambient light parameter based on the at least one intrinsic camera parameter associated with the at least one input frame and the type of the scene.