# US 2016/0366396 A1 · Interactive content control apparatus and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0366396 A1
- Document ID: US20160366396A1
- Status: Published application
- Kind: A1
- Application number: 14/837,181
- Filed: 2015-08-27
- Published: 2016-12-15
- Earliest claimed: 2015-06-15
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Hang-Kee KIM; Ki-Hong KIM; Hong-Kee KIM; Woo-Jin JEON
- Web page: https://patentgrep.com/patent/US20160366396A1

## CPC

- G06T 7/536: from perspective effects, e.g. by using vanishing points
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 13/20: Three-dimensional [3D] animation
- G06T 15/10: Geometric effects
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An interactive content control apparatus and method includes a depth information acquisition unit for acquiring an image that includes depth information by capturing a static background object and a dynamic foreground object, and for calculating depth maps based on respective depth values of the static background object and the dynamic foreground object from the image, a display unit for projecting image content including a background image and an object image on a surface of the static background object captured by the depth information acquisition unit, and an image mapping unit for recognizing the dynamic foreground object depending on the depth maps based on the depth values calculated by the depth information acquisition unit, and controlling the display unit so that an object image projected on the surface of the static background object is transferred to and projected on a surface of the dynamic foreground object.

## Claims

### Claim 1

An interactive content control apparatus, comprising:
a depth information acquisition unit for acquiring an image that includes depth information by capturing a static background object and a dynamic foreground object, and for calculating depth maps based on respective depth values of the static background object and the dynamic foreground object from the image;
a display unit for projecting image content including both a background image and an object image on a surface of the static background object captured by the depth information acquisition unit; and
an image mapping unit for recognizing the dynamic foreground object depending on the depth maps based on the depth values calculated by the depth information acquisition unit, and controlling the display unit so that an object image projected on the surface of the static background object is transferred to and projected on a surface of the dynamic foreground object.

### Claim 2 (depends on 1)

The interactive content control apparatus of claim 1, wherein the image mapping unit comprises:
an object image bounding volume management unit for setting and managing an object image bounding volume including the object image in a foreground direction of the object image projected on the static background object; and
a foreground/background processing unit for, if, depending on the depth maps based on the depth values calculated by the depth information acquisition unit, the dynamic foreground object is determined to collide with the object image bounding volume, controlling the display unit so that the object image projected on the surface of the static background object is transferred to and projected on the surface of the dynamic foreground object.

### Claim 3 (depends on 2)

The interactive content control apparatus of claim 2, wherein the foreground/background processing unit comprises:
a foreground object recognition unit for recognizing a location and movement direction of a dynamic foreground object present in the foreground direction of the static background object, depending on the depth maps based on the depth values calculated by the depth information acquisition unit;
a foreground object bounding volume setting unit for setting a foreground object bounding volume including the dynamic foreground object recognized by the foreground object recognition unit;
a bounding volume collision determination unit for, if the object image bounding volume and the foreground object bounding volume overlap each other by a preset threshold or more, and if the movement direction of the dynamic foreground object is the foreground direction, determining that the dynamic foreground object collides with the object image bounding volume; and
a foreground/background switching unit for, if it is determined by the bounding volume collision determination unit that the dynamic foreground object collides with the object image bounding volume, controlling the display unit so that the object image projected on the surface of the static background object is transferred to and projected on the surface of the dynamic foreground object.

### Claim 4 (depends on 3)

The interactive content control apparatus of claim 3, wherein the foreground object recognition unit is configured to previously acquire a depth map based on a depth value of the static background object obtained when no dynamic foreground object is present, and is configured to, if depth values that differ from the depth value of the static background object by a preset threshold or more are calculated by the depth information acquisition unit, classify depth values, grouped for neighboring depth values, among the depth values, as individual blobs, and recognize a location and a movement direction of the dynamic foreground object only when a size of each blob falls within a preset range.

### Claim 5 (depends on 4)

The interactive content control apparatus of claim 4, wherein the foreground object recognition unit sets a width corresponding to a preset range in an outer region of the depth map based on the depth value of the static background object obtained when no dynamic foreground object is present, and excludes a blob if the blob extends into the width.

### Claim 6 (depends on 5)

The interactive content control apparatus of claim 5, wherein the foreground/background switching unit is configured to, if it is recognized by the foreground object recognition unit that the dynamic foreground object deviates from the foreground direction of the static background object, control the display unit so that the object image transferred to and projected on the surface of the dynamic foreground object is projected on the surface of the static background object.

### Claim 7 (depends on 2)

The interactive content control apparatus of claim 2, wherein the image mapping unit further comprises an event processing unit configured to, when the dynamic foreground object is moved, change an animation effect of the object image transferred to and projected on the surface of the dynamic foreground object, or change content of the background image projected on the surface of the static background object, depending on the depth maps based on the depth values calculated by the depth information acquisition unit.

### Claim 8 (depends on 7)

The interactive content control apparatus of claim 7, wherein the event processing unit comprises:
a foreground object tracking unit for tracking a location and a movement direction of the dynamic foreground object present in the foreground direction of the static background object, depending on the depth maps based on the depth values calculated by the depth information acquisition unit; and
an object image animation processing unit for, as the dynamic foreground object tracked by the foreground object tracking unit is moved, changing the animation effect of the object image transferred to and projected on the surface of the dynamic foreground object.

### Claim 9 (depends on 8)

The interactive content control apparatus of claim 8, wherein the event processing unit further comprises a background image content processing unit for, as the dynamic foreground object tracked by the foreground object tracking unit is moved, changing the content of the background image projected on the surface of the static background object.

### Claim 10 (depends on 2)

The interactive content control apparatus of claim 2, wherein the image mapping unit further comprises a preprocessing unit for calculating an extrinsic parameter of the depth information acquisition unit and an intrinsic parameter of the display unit, generating relative location and direction information, and performing calibration between the depth information acquisition unit and the display unit.

### Claim 11

An interactive content control method, comprising:
acquiring, by a depth information acquisition unit, an image that includes depth information by capturing a static background object, and calculating a depth map based on a depth value of the static background object from the image;
projecting, by an image mapping unit, image content including both a background image and an object image on a surface of the static background object depending on the depth map by controlling a display unit, the depth map being based on the depth value of the static background object calculated by a depth information acquisition unit;
acquiring, by the depth information acquisition unit, an image that includes depth information by capturing a dynamic foreground object, and calculating a depth map based on a depth value of the dynamic foreground object from the image; and
recognizing, by the image mapping unit, the dynamic foreground object depending on the depth map based on the depth value of the dynamic foreground object calculated by the depth information acquisition unit, and transferring and projecting the object image projected on the surface of the static background object to and on a surface of the dynamic foreground object by controlling the display unit.

### Claim 12 (depends on 11)

The interactive content control method of claim 11, wherein transferring and projecting the object image projected on the surface of the static background object to and on the surface of the dynamic foreground object comprises:
setting and managing, by an object image bounding volume management unit, an object image bounding volume including the object image in a foreground direction of the object image projected on the static background object; and
transferring and projecting the object image projected on the surface of the static background object to and on the surface of the dynamic foreground object by controlling the display unit if, depending on the depth maps based on the depth values calculated by the depth information acquisition unit, the dynamic foreground object is determined to collide with the object image bounding volume.

### Claim 13 (depends on 12)

The interactive content control method of claim 12, wherein transferring and projecting the object image projected on the surface of the static background object to and on the surface of the dynamic foreground object if the dynamic foreground object is determined to collide with the object image bounding volume, comprises:
recognizing, by a foreground object recognition unit, a location and a movement direction of the dynamic foreground object present in the foreground direction of the static background object depending on the depth maps based on the depth values calculated by the depth information acquisition unit;
setting, by a foreground object bounding volume setting unit, a foreground object bounding volume including the dynamic foreground object recognized by the foreground object recognition unit;
if the object image bounding volume and the foreground object bounding volume overlap each other by a preset threshold or more, and the movement direction of the dynamic foreground object is the foreground direction, determining that the dynamic foreground object collides with the object image bounding volume; and
if it is determined by the bounding volume collision determination unit that the dynamic foreground object collides with the object image bounding volume, transferring and projecting, by a foreground/background switching unit, the object image projected on the surface of the static background object to and on the surface of the dynamic foreground object, by controlling the display unit.

### Claim 14 (depends on 13)

The interactive content control method of claim 13, wherein recognizing the location and movement direction of the dynamic foreground object present in the foreground direction of the static background object comprises:
previously acquiring a depth map based on a depth value of the static background object obtained when no dynamic foreground object is present; and
if depth values that differ from the depth value of the static background object by a preset threshold or more are calculated by the depth information acquisition unit, classifying depth values, grouped for neighboring depth values, among the depth values, as individual blobs, and recognizing a location and a movement direction of the dynamic foreground object only when a size of each blob falls within a preset range.

### Claim 15 (depends on 14)

The interactive content control method of claim 14, wherein recognizing the location and movement direction of the dynamic foreground object present in the foreground direction of the static background object further comprises:
setting a width corresponding to a preset range in an outer region of the depth map based on the depth value of the static background object obtained when no dynamic foreground object is present, and excluding a blob if the blob extends into the width.

### Claim 16 (depends on 15)

The interactive content control method of claim 15, wherein transferring and projecting the object image projected on the surface of the static background object to and on the surface of the dynamic foreground object if it is determined by the bounding volume collision determination unit that the dynamic foreground object collides with the object image bounding volume, comprises:
if it is recognized by the foreground object recognition unit that the dynamic foreground object deviates from the foreground direction of the static background object upon recognizing the location and movement direction of the dynamic foreground object, controlling the display unit so that the object image transferred to and projected on the surface of the dynamic foreground object is projected on the surface of the static background object.

### Claim 17 (depends on 11)

The interactive content control method of claim 11, further comprising changing, by an event processing unit, an animation effect of the object image transferred to and projected on the surface of the dynamic foreground object, or changing content of the background image projected on the surface of the static background object when the dynamic foreground object is moved, depending on the depth maps based on the depth values calculated by the depth information acquisition unit.

### Claim 18 (depends on 17)

The interactive content control method of claim 17, wherein changing the animation effect of the object image transferred to and projected on the surface of the dynamic foreground object, or changing the content of the background image projected on the surface of the, static background object comprises:
tracking, by a foreground object tracking unit, a location and a movement direction of the dynamic foreground object present in the foreground direction of the static background object, depending on the depth maps based on the depth values calculated by the depth information acquisition unit; and
as the dynamic foreground object tracked by the foreground object tracking unit is moved, changing, by an object image animation processing unit, the animation effect of the object image transferred to and projected on the surface of the dynamic foreground object.

### Claim 19 (depends on 18)

The interactive content control method, of claim 18, wherein changing the animation effect of the object image transferred to and projected on the surface of the dynamic foreground object, or changing the content of the background image projected on the surface of the static background object further comprises:
as the dynamic foreground object tracked by the foreground object tracking unit is moved, changing, by a background image content processing unit, the content of the background image projected on the surface of the static background object.

### Claim 20 (depends on 11)

The interactive content control method of claim 11, further comprising, before acquiring the image including depth information by capturing the static background object, and calculating the depth map based on the depth value of the static background object from the image:
calculating, by a preprocessing unit, an extrinsic parameter of the depth information acquisition unit and an intrinsic parameter of the display unit, generating relative location and direction information, and performing calibration between the depth information acquisition unit and the display unit.
