# US 10,121,273 B2 · Real-time reconstruction of the human body and automated avatar synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,121,273 B2
- Document ID: US10121273B2
- Status: Granted
- Kind: B2
- Application number: 14/904,891
- Filed: 2014-08-08
- Published: 2016-06-09
- Granted: 2018-11-06
- Earliest claimed: 2013-08-08
- Assignees: University of Florida Research Foundation, Inc.
- Applicants: University of Florida Research Foundation, Incorporated
- Inventors: Angelos Barmpoutis
- Web page: https://patentgrep.com/patent/US10121273B2

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 7/149: involving deformable models, e.g. active contour models
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and Techniques for real-time 3D reconstruction of the human body are described. Avatars (the rendered 3D reconstruction of the human body) can be generated from real-time captured RGB-D images of a person. Avatars can be synthesized from the RGB-D data received from a single RGB-D camera by performing body segmentation (into cylindrical-type objects) and dynamic robust data filtering on sequential frames of the captured data. Cylindrical-type objects of the body, including arms, legs, and torso are parameterized using tensor splines; and positive-definite constrains are imposed to the estimated tensor splines using a Riemannian metric defined on the space of positive-definite tensor splines. These generated avatars have an articulated body with separately translatable and rotatable arms, legs, and other limbs or cylindrical features.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving discrete depth frames Di,j from a depth camera;
calculating quadratic meshes given by Xi,j=(i−ic)Di,jcd−1, Yi,j=(j−jc)Di,jcd−1, and Zi,j=Di,j, where ic,jc denote coordinates of a central pixel in a depth frame, and cd is a focal length of the depth camera;
detecting a presence of a particular skeletal geometry from a sequence of data frames by fitting each frame of the discrete depth frames to a skeletal model with parameters defined from
S={al∈,bl∈,Rl∈SO(3):l∈}
where is a set of indices of line segments defined by end-points al and bl, and its orientation in the 3D space is given by the rotation matrix Rl;
segmenting the quadratic mesh into body regions by using the parameters of the skeletal model S; and
fitting a positive-definite tensor spline of a spline model
f
d
⁡
(
x
,
s
)
=
∑
i
=
1
m
⁢
∑
j
=
0
n
⁢
w
i
,
j
⁢
N
j
,
k
+
1
⁡
(
s
)
⁢
f
d
,
i
*
⁡
(
x
)
to a point-set Pl of points that belong to a particular body region of the body regions.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
filtering the point-set before fitting the positive-definite tensor spline to the point-set, wherein filtering the point-set comprises:
calculating
h
⁡
(
f
,
x
,
s
;
𝒫
)
=
∑
p
∈
𝒫
⁢
N
⁡
(
f
;

x
p

,
σ
f
2
)
⁢
N
⁡
(
s
;
s
l
⁡
(
p
)
⁡
(
p
)
,
σ
s
2
)
⁢
V
⁡
(
x
;
x
p

x
p

,
κ
)
where ƒ∈, x∈S1, s∈, functions N( ) and V( ) denote a Normal and a von Mises probability density functions respectively, parameters σƒ2, σs2 and κ are variances and concentration of the probability density functions; and
computing a data estimate
d
i
,
j
=
f
i
,
j
,
argmax
k
=
1
:
K
h
⁢
i
,
j
,
k
.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein fitting the positive-definite tensor spline to the point-set comprises: reconstructing a 3D shape of a human body by computing an energy function
E
⁡
(
w
l
)
=
∑
i
=
1
M
⁢
∑
j
=
1
N
⁢
(
f
l
⁡
(
x
i
,
s
j
)
-
d
i
,
j
)
2
.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein segmenting the quadratic mesh into body regions by using the parameters of the skeletal model S comprises:
calculating, for every vertex p=[Xi,j Yi,j Di,j]−T in the quadratic mesh, the index l of the closest line segment in the skeletal model as
l(p)=argminl∈∥al+sl(p)(bl−al)−p∥
where al, bl∈ are vertices/joints that define a particular line segment in the skeletal model, and sl(p) is the projection of p onto the lth line segment given by:
s
l
⁡
(
p
)
=
max
⁢
{
min
⁢
{
(
b
l
-
a
l
)
T
⁢
(
p
-
a
)

b
l
-
a
l

2
,
1
}
,
0
}
.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
after estimating the coefficient vectors wl ∀l∈, rendering a human avatar in any arbitrary posture given in the form of S.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the rendering of the human avatar comprises:
scaling each tensor-spline model by a magnitude of ∥al-bl∥ along a longitudinal axis, translating its center to a point (al+bl)/2, and rotating by Rl.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving red-green-blue (RGB) video frames from a video camera;
associating the RGB video frames with the quadratic meshes to generate a colored quadratic mesh by using a UV texture mapping given by coordinates Ui,j=Xi,j′Zi,j′−1cv, Vi,j=Yi,j′Zi,j′−1 cv, where the coordinates of the vector [X′ Y′ Z′]T are related to [X Y Z]T via a rigid transformation with rotation and translation, and cv is a focal length of the video camera;
fitting the RGB video frames to the skeletal model with parameters defined from S={al ∈,bl∈, Rl∈SO(3:l∈}; and
segmenting the colored quadratic mesh into the body regions by using the parameters of the skeletal model S.