# US 8,786,680 B2 · Motion capture from body mounted cameras > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,786,680 B2 - Document ID: US8786680B2 - Status: Granted - Kind: B2 - Application number: 13/165,619 - Filed: 2011-06-21 - Published: 2012-12-27 - Granted: 2014-07-22 - Earliest claimed: 2011-06-21 - Assignees: Disney Enterprises, Inc.; Carnegie Mellon University - Applicants: Takaaki Shiratori; Hyun Soo Park; Leonid Sigal; Yaser Sheikh; Jessica K. Hodgins - Inventors: Takaaki Shiratori; Hyun Soo Park; Leonid Sigal; Yaser Sheikh; Jessica K. Hodgins - Web page: https://patentgrep.com/patent/US8786680B2 ## CPC - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Body-mounted cameras are used to accurately reconstruct the motion of a subject. Outward-looking cameras are attached to the limbs of the subject, and the joint angles and root pose that define the subject's configuration are estimated through a non-linear optimization, which can incorporate image matching error and temporal continuity of motion. Instrumentation of the environment is not required, allowing for motion capture over extended areas and in outdoor settings. ## Claims ### Claim 1 A method for capturing motion of a subject having a body with a plurality of articulating members movable about a plurality of joints, the method comprising: attaching a plurality of cameras to the body of the subject, wherein at least some of the cameras are attached to the articulating members and oriented toward an external environment surrounding the subject; recording an image sequence from each of the cameras while the subject performs an action; analyzing, using a computer system, the image sequences recorded by each of the cameras, including images of the external environment from the cameras attached to the articulating members, to determine correlations between the images; and based at least in part on the analysis of the image sequences, including the images of the external environment from the cameras attached to the articulating members, computing, using the computer system, a sequence of articulated poses of the body of the subject, the sequence of articulated poses being representative of the action. ### Claim 2 (depends on 1) The method of claim 1 further comprising: rendering a computer-generated animation of an actor performing the action based at least in part on the computed sequence of articulated poses of the body of the subject. ### Claim 3 (depends on 1) The method of claim 1 wherein the external environment includes a plurality of objects that remain stationary while the subject performs the action. ### Claim 4 (depends on 3) The method of claim 3 wherein analyzing the image sequences includes generating a representation of three-dimensional scene geometry based on correlating visual features across multiple ones of the images in the recorded image sequences. ### Claim 5 (depends on 1) The method of claim 1 wherein analyzing the image sequences includes: obtaining a model of three-dimensional scene geometry for the external environment from a source other than the recorded image sequences; and using the model of the three-dimensional scene geometry to facilitate determination of correlations between the images. ### Claim 6 (depends on 1) The method of claim 1 wherein at least one of the cameras is a visible-light camera with a field of view subtending an angle of at least 150 degrees. ### Claim 7 (depends on 1) The method of claim 1 wherein each of the cameras has a field of view such that the fields of view of at least some of the cameras overlap when the subject's body is in a resting position. ### Claim 8 (depends on 1) The method of claim 1 wherein the action performed by the subject includes performing a series of training motions during which different joints of the body of the subject are moved through a full range of motion. ### Claim 9 (depends on 1) The method of claim 1 further comprising: recording a plurality of reference images of the external environment, wherein analyzing the image sequences recorded by each of the cameras includes analyzing the reference images. ### Claim 10 (depends on 9) The method of claim 9 wherein at least one of the plurality of reference images is recorded using a camera that is not attached to the body of the subject. ### Claim 11 (depends on 1) The method of claim 1 wherein attaching the plurality of cameras includes attaching at least one camera to an article of clothing to be worn by the subject. ### Claim 12 (depends on 1) The method of claim 1 wherein attaching the plurality of cameras includes attaching at least one camera to the subject using a strap. ### Claim 13 (depends on 1) The method of claim 1 wherein computing the sequence of articulated poses includes defining a root for the body and determining a position of at least one of the plurality of joints relative to the root. ### Claim 14 (depends on 13) The method of claim 13 wherein computing the sequence of articulated poses further includes determining an absolute position and orientation for the root within the external environment. ### Claim 15 A system for capturing motion of a subject having a body with a plurality of articulating members, the system comprising: a data input port configured to receive a plurality of image sequences captured by a plurality of cameras worn by a subject while performing an action, wherein at least some of the cameras are attached to the articulating members of the body of the subject and oriented toward an external environment surrounding the subject; and a processor configured to: analyze the image sequences received from each of the plurality of cameras, including images of the external environment received from the cameras attached to the articulating members of the body, to determine correlations between the images; and compute, based at least in part on the analysis of the image sequences, including the images of the external environment, a sequence of articulated poses of the body of the subject, the sequence of articulated poses being representative of the action. ### Claim 16 (depends on 15) The system of claim 15 wherein the processor is further configured such that computing the sequence of articulated poses includes: applying a structure-from-motion algorithm to the image sequences to determine a geometric representation of a three-dimensional scene and an initial estimate of camera poses; and applying a reprojection optimization to refine the initial estimate of the camera poses. ### Claim 17 (depends on 16) The system of claim 16 wherein the processor is further configured such that computing the sequence of articulated poses further includes: computing a skeleton based on images from the image sequence associated with a range-of-motion exercise performed by the subject. ### Claim 18 A system for capturing motion of a subject having a body with a plurality of articulating members, the system comprising: a plurality of cameras adapted to be wearable by a subject while performing an activity, the plurality of cameras including at least two cameras wearable on the articulating members of the body of the subject and orientable toward an external environment surrounding the subject; a storage medium configured to receive and store a plurality of image sequences captured by the plurality of cameras, the image sequences including images of the external environment captured while the subject performed the activity; and a processor coupled to the storage medium and configured to: analyze the image sequences including the images of the external environment to determine correlations between the images; and compute, based at least in part on the analysis of the image sequences, including the images of the external environment, a sequence of articulated poses of the body of the subject, the sequence of articulated poses being representative of the action. ### Claim 19 (depends on 18) The system of claim 18 wherein the cameras are operable to capture images at a rate of at least 60 frames per second. ### Claim 20 (depends on 18) The system of claim 18 wherein at least one of the cameras is a visible-light camera with a field of view subtending an angle of at least 150 degrees.