# US 12,112,416 B2 · Full-body integrated motion capture method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,416 B2 - Document ID: US12112416B2 - Status: Granted - Kind: B2 - Application number: 17/847,316 - Filed: 2022-06-23 - Published: 2023-12-14 - Granted: 2024-10-08 - Earliest claimed: 2022-06-09 - Assignees: 5Motion Inc. - Applicants: 5Motion Inc. - Inventors: Young Ho Seo; Moon Seok Oh - Web page: https://patentgrep.com/patent/US12112416B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06T 7/248: {involving reference images or patches} - G06T 7/251: {involving models} - G06T 7/292: Multi-camera tracking - G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Proposed is a full-body integrated motion capture method capable of integrally detecting joints for full-body motion capture by extracting body joints of a body from a low-resolution multiview video, detecting face and hand regions through the body joints, and precisely extracting detailed joints for a face and a hand from a high-resolution video of the face and hand regions. The full-body integrated motion capture method includes: (a) receiving a multiview color-depth video and a high-resolution video; (b) extracting a body skeleton from the low-resolution video; (c) detecting face and hand regions by using joints of a neck and a wrist of the body skeleton; (d) extracting a detailed joint from a region of the high-resolution video corresponding to the face and hand regions; and (e) generating motion data by integrating a body joint and the detailed joint. ## Claims ### Claim 1 A full-body integrated motion capture method comprising: (a) receiving a multiview color-depth video and a high-resolution video; (b) extracting a body skeleton from the multiview color-depth video; (c) detecting face and hand regions by using joints of a neck and a wrist of the body skeleton; (d) extracting a joint of a predetermined detail level from a region of the high-resolution video corresponding to the face and hand regions; and (e) generating motion data by integrating a body joint and the joint of the predetermined detail level, wherein the step (b) includes: (b1) generating a point cloud of a three-dimensional object; (b2) acquiring a projection image obtained by projecting the point cloud of the three-dimensional object onto at least three projection planes that are different from each other; (b3) acquiring a two-dimensional skeleton image in each direction for each of at least three projection images by using a neural network, in which a two-dimensional joint and a bone configured to connect joints to each other are displayed in the two-dimensional skeleton image; (b4) restoring the two-dimensional skeleton image in each direction, and detecting a three-dimensional internal joint by using a restored three-dimensional surface joint in each direction; and (b5) generating a three-dimensional body skeleton including the detected three-dimensional internal joint. ### Claim 2 (depends on 1) The full-body integrated motion capture method of claim 1, wherein, in the step (b2), the projection image in each direction is acquired by setting the projection plane in each direction such that directions of normal vectors of N projection planes divide 360 degrees into N equal parts about a vertical axis, and projecting the point cloud of the three-dimensional object onto the projection plane in each direction. ### Claim 3 (depends on 2) The full-body integrated motion capture method of claim 2, wherein, in the step (b2), the projection image is acquired by obtaining an axis-aligned bounding box (AABB) for the three-dimensional object, and performing the projection onto four surfaces of the AABB, which are left, right, front, and rear surfaces of the AABB. ### Claim 4 (depends on 1) The full-body integrated motion capture method of claim 1, wherein, in the step (b2), the projection image for the three-dimensional object is acquired by projecting the point cloud of the three-dimensional object onto a horizontal plane, obtaining a front direction vector on a two-dimensional plane onto which the point cloud is projected, and rotating the three-dimensional object such that a front surface of the three-dimensional object is arranged on a vertical axis by using the front direction vector. ### Claim 5 (depends on 4) The full-body integrated motion capture method of claim 4, wherein, in the step (b2), the front direction vector is obtained through principal component analysis, in which a covariance matrix is obtained for the two-dimensional plane onto which the point cloud is projected, an eigenvector for the obtained matrix is obtained, and a vector having a smallest eigenvalue is set as the front direction vector. ### Claim 6 (depends on 1) The full-body integrated motion capture method of claim 1, wherein the neural network includes an OpenPose network. ### Claim 7 (depends on 1) The full-body integrated motion capture method of claim 1, wherein, in the step (b4), the two-dimensional skeleton image in each direction is restored into three-dimensional coordinates, and an intersection region of a straight line that is perpendicular to the projection plane in the restored surface joint of the three-dimensional coordinates in each direction is determined as the three-dimensional internal joint. ### Claim 8 (depends on 1) The full-body integrated motion capture method of claim 1, wherein, in the step (b5), for each internal joint of the generated three-dimensional skeleton, a plane that bisects an angle formed by at least two bones connected to the internal joint (hereafter referred to as “division plane”) is obtained, points within a predetermined reference value range in the division plane are clustered, a center of a cluster obtained through the clustering is detected, and coordinates of the internal joint are corrected by using a center point. ### Claim 9 (depends on 8) The full-body integrated motion capture method of claim 8, wherein, in the step (b5), the clustering is performed by using a density-based spatial clustering of applications with noise (DBSCAN) algorithm, and the center point is detected by performing circle fitting. ### Claim 10 (depends on 1) A non-transitory computer-readable recording medium recorded with a program for performing a full-body integrated motion capture method according to claim 1.