# US 2021/0279951 A1 · Method and apparatus for processing holographic image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279951 A1
- Document ID: US20210279951A1
- Status: Published application
- Kind: A1
- Application number: 17/191,969
- Filed: 2021-03-04
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Min Sung YOON; Soo Myung PARK
- Web page: https://patentgrep.com/patent/US20210279951A1

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- H04N 13/30: Image reproducers (optical systems for producing stereoscopic or other three-dimensional [3D] effects G02B30/00)
- G03H 1/0808: {Methods of numerical synthesis, e.g. coherent ray tracing [CRT], diffraction specific}
- G03H 1/0841: {Encoding method mapping the synthesized field into a restricted set of values representative of the modulator parameters, e.g. detour phase coding}
- G03H 1/16: using Fourier transform ({G03H1/10}, G03H1/12, G03H1/14 take precedence; analogue computers G06G, e.g. G06G7/19)
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/50: Depth or shape recovery
- G03H 1/2294: {Addressing the hologram to an active spatial light modulator}
- G03H 2210/441: Numerical processing applied to the object data other than numerical propagation
- G03H 2226/05: Means for tracking the observer
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G03H: HOLOGRAPHIC PROCESSES OR APPARATUS (holograms, e.g. point holograms, used as ordinary optical elements G02B5/32; producing stereoscopic or other three-dimensional effects G02B30/00; diffraction-grating systems G02B27/44; systems using moiré fringes G02B27/60; optical logic elements G02F3/00; stereo-photography G03B35/00; photosensitive materials or processes for photographic purposes G03C; {stereo-photographic or similar processes G03C9/00}; apparatus for processing exposed photographic materials G03D; analogue computers performing mathematical operations with the aid of optical elements G06E3/00; authentication by radiation, of concealed information carried by holograms or diffraction gratings G06K19/16; holographic storage G11B7/0065, G11C13/04; {stereoscopic or other three dimensional effects in television systems H04N13/00})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of generating hologram data in a holographic display apparatus, including a hologram processing device generating hologram data and a display terminal reconstructing a hologram image in a three-dimensional (3D) space on the basis of the generated hologram data, includes generating depth map image data from input image data by using a deep learning engine for generating a depth map and calculating a complex value hologram on the basis of the depth map image data and the input image data to generate the hologram data.

## Claims

### Claim 1

A method of generating hologram data in a holographic display apparatus including a hologram processing device generating hologram data and a display terminal reconstructing a hologram image in a three-dimensional (3D) space on the basis of the generated hologram data, the method comprising:
generating depth map image data from input image data by using a deep learning engine for generating a depth map; and
calculating a complex value hologram on the basis of the depth map image data and the input image data to generate the hologram data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
before the generating of the depth map image data,
building a database storing, as learning data, input image data for training and depth map image data for training; and
training a deep learning engine for generating the depth map, based on the learning data.

### Claim 3 (depends on 1)

The method of claim 1, wherein the generating of the hologram data comprises calculating the complex value hologram by using a deep learning engine for generating convolutional neural network-based hologram data.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
before the generating of the depth map image data,
building a database storing, as learning data, color image data for training, depth map image data for training, and hologram data for training; and
training a deep learning engine for generating the hologram data, based on the learning data.

### Claim 5 (depends on 1)

The method of claim 1, wherein the generating of the hologram data comprises calculating the complex value hologram by using a fast Fourier transform (FFT)-based computer-generated hologram (CGH) equation.

### Claim 6 (depends on 1)

The method of claim 1, further comprising, after the generating of the depth map image data, encoding the hologram data by using one of amplitude modulation, phase modulation, or complex value modulation on the basis of an optical modulation scheme of a spatial light modulator included in the display terminal.

### Claim 7 (depends on 1)

The method of claim 1, wherein the generating of the depth map image data comprises:
extracting and down-sampling a feature of the input image data; and
performing a concatenation operation on the extracted feature and performing up-sampling on the feature with reference to a size of the input image data to generate the depth map image data.

### Claim 8

A method of reconstructing the hologram image in a holographic display apparatus including a hologram processing device generating hologram data and a display terminal reconstructing a hologram image in a three-dimensional (3D) space on the basis of the generated hologram data, the method comprising:
generating depth map image data from input image data by using a first deep learning engine;
generating the hologram data on the basis of the depth map image data and the input image data by using a second deep learning engine; and
reconstructing the hologram image in a 3D space on the basis of the hologram data.

### Claim 9 (depends on 8)

The method of claim 8, wherein each of the first and second deep learning engines is a convolutional neural network-based artificial neural network.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
before the generating of the depth map image data,
training the first deep learning engine by using learning data including input image data for training and depth map image data for training;
training the second deep learning engine by using learning data including the input image data for training, the depth map image data for training, and hologram data for training; and
integrating the trained first deep learning engine and the trained second deep learning engine in an end-to-end structure.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
recognizing a gesture of a user to obtain gesture information about the user;
generating a shape change command which issues a command to change a shape of the reconstructed hologram image, based on the gesture information;
selecting hologram data corresponding to the shape change command from among pieces of hologram data which are generated in the generating of the hologram data; and
updating hologram data, corresponding to a currently reconstructed hologram image, to the selected hologram data,
wherein the reconstructing of the hologram image comprises reconstructing a hologram image having a shape which differs from a shape of the reconstructed hologram image, based on the updated hologram data.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
recognizing a voice of a user to obtain voice recognition information about the user;
generating a shape change command which issues a command to change a shape of the reconstructed hologram image, based on the voice recognition information;
selecting hologram data corresponding to the shape change command from among pieces of hologram data which are generated in the generating of the hologram data; and
updating hologram data, corresponding to a currently reconstructed hologram image, to the selected hologram data,
wherein the reconstructing of the hologram image comprises reconstructing a hologram image having a shape which differs from a shape of the reconstructed hologram image, based on the updated hologram data.

### Claim 13

A holographic display apparatus comprising:
a hologram processing device configured to generate hologram data; and
a display terminal configured to reconstruct a hologram image in a three-dimensional (3D) space on the basis of the generated hologram data,
wherein the hologram processing device comprises:
a depth map generator configured to generate depth map image data from input image data by using a first deep learning engine;
a hologram generator configured to generate the hologram data on the basis of the depth map image data and the input image data by using a fast Fourier transform (FFT)-based computer-generated hologram (CGH) calculation algorithm or a second deep learning engine; and
a transmitter configured to transmit the hologram data to the display terminal, and
wherein the display terminal comprises:
a receiver configured to receive the hologram data;
a spatial light modulator configured to form a diffraction pattern on the basis of the received hologram data and to modulate light incident on the formed diffraction pattern; and
an optical module configured to irradiate coherent light onto the diffraction pattern so that the hologram image is reconstructed in a 3D space on the basis of light modulated by the spatial light modulator.

### Claim 14 (depends on 13)

The holographic display apparatus of claim 13, wherein the first deep learning engine and the second deep learning engine are integrated in an end-to-end structure.

### Claim 15 (depends on 13)

The holographic display apparatus of claim 13, wherein each of the first deep learning engine and the second deep learning engine is based on a convolutional neural network (CNN).

### Claim 16 (depends on 13)

The holographic display apparatus of claim 13, wherein the first deep learning engine is implemented as a DenseDepth model of a densely connected convolutional networks (DenseNet) type.

### Claim 17 (depends on 13)

The holographic display apparatus of claim 13, wherein the first deep learning engine is implemented to operate in a multi-graphics processing unit (GPU) environment by using a data parallel model conversion function which is supported in a Pytorch library.

### Claim 18 (depends on 13)

The holographic display apparatus of claim 13, further comprising a human interaction device configured to provide an interaction function between the display terminal and a user,
wherein the human interaction device comprises:
a user recognition sensor configured to recognize a gesture or a voice of the user to obtain gesture information or voice recognition information about the user; and
a feedback command generator configured to a feedback command for issuing a command to change a shape of the reconstructed hologram image, based on the gesture information or the voice recognition information,
the hologram processing device further comprises a feedback processing unit configured to select hologram data corresponding to the feedback command from among pieces of hologram data generated by the hologram generator and to update hologram data, corresponding to a currently reconstructed hologram image, to the selected hologram data, and
the spatial light modulator included in the display terminal forms a diffraction pattern on the basis of the updated hologram data and modulates light incident on the formed diffraction pattern.
