# US 10,310,622 B2 · Method and device for correcting motion information collected by NUI device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,622 B2 - Document ID: US10310622B2 - Status: Granted - Kind: B2 - Application number: 15/425,049 - Filed: 2017-02-06 - Published: 2017-05-25 - Granted: 2019-06-04 - Earliest claimed: 2014-08-06 - Assignees: Starship Vending-Machine Corp. - Applicants: Starship Vending-Machine Corp. - Inventors: Ji-yong Kwon - Web page: https://patentgrep.com/patent/US10310622B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 5/20: using local operators - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20024: Filtering details - G06T 2207/20201: Motion blur correction - G06T 2207/30196: Human being; Person - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present invention provides a method for stabilizing an instable estimated rotation angle included in motion information generated from a plurality of image frames captured by an NUI device and correcting motion information for a fast speed motion lost during a process of stabilization based on the captured plurality of image frames. Accordingly, an instable motion among motions restored based on motion information can be stabilized, and the accuracy of the motion can be maintained. ## Claims ### Claim 1 A method for correcting motion information generated from a plurality of image frames in which a motion of a user has been captured, the method comprising: (a) generating motion information including locations of one or more joints, namely axes of motion based on a motion of a user, and estimated rotation angles, namely rotation angles of the respective joints calculated based on the locations of the one or more joints; (b) applying a stabilization filter, having a time window within a predetermined range based on time, to the estimated rotation angles included in the motion information; and (c) correcting the estimated rotation angles, to which the stabilization filter has been applied, based on the locations of the joints included in the motion information; wherein step (b) comprises removing an estimated rotation angle, for which a variation in an estimated rotation angle exceeds a predetermined range between the plurality of image frames, from the estimated rotation angles of the one or more joints included in the motion information; and wherein removing the estimated rotation angle comprises: calculating a kernel of the stabilization filter to be applied to the estimated rotation angles included in the motion information; and applying the stabilization filter having the calculated kernel to the joints of the plurality of image frames. ### Claim 2 (depends on 1) The method of claim 1, wherein calculating the kernel of the stabilization filter comprises calculating the kernel of the stabilization filter based on time distances between a reference image frame to which the stabilization filter is applied and the image frames included in the time window and estimated rotation angles and tracked rotation angles of the joints in the image frames included in the time window. ### Claim 3 (depends on 2) The method of claim 2, wherein calculating the kernel of the stabilization filter comprises: calculating differences in time between the reference frame to which the stabilization filter is applied and the image frames included in the time window; and calculating a difference between a variation in an estimated rotation angle and an variation in an tracked rotation angle for each of the joints in the image frames included in the time window. ### Claim 4 (depends on 3) The method of claim 3, wherein calculating the difference between the variation in the estimated rotation angle and the variation in the tracked rotation angle comprises: calculating the variation in the estimated rotation angle of the joint between successive image frames included in the time window; calculating a tracked rotation angle based on a connection vector connecting a location of the joint and a location of a child joint having a connection relationship with the joint; and calculating the variation in the tracked rotation angle of the joint in the successive image frames included in the time window. ### Claim 5 (depends on 1) The method of claim 1, wherein step (c) comprises: calculating the locations of the joints based on the estimated rotation angles of the joints to which the stabilization filter has been applied; and correcting the estimated rotation angles of the joints to which the stabilization filter has been applied so that the calculated locations of the joints become similar to the locations of the joints in the image frames. ### Claim 6 (depends on 5) The method of claim 5, wherein calculating the locations of the joints comprises: calculating a location of a child joint determined based on an estimated rotation angle of a parent joint, namely any one of the joints to which the stabilization filter has been applied; and correcting the estimated rotation angle of the parent joint to which the stabilization filter has been applied so that a connection vector connecting the calculated location of the child joint and the location of the parent joint becomes consistent with a connection vector connecting a location of the child joint and a location of the parent joint in the image frames. ### Claim 7 A device for correcting motion information generated from a plurality of image frames in which a motion of a user has been captured, the device comprising: a processor configured to: generate motion information including locations of one or more joints, namely axes of motion based on a motion of a user, and estimated rotation angles, namely rotation angles of the respective joints calculated based on the locations of the one or more joints; calculate a kernel of a stabilization filter to be applied to the estimated rotation angles included in the motion information; apply the stabilization filter, having a time window within a predetermined range based on time, to the estimated rotation angles included in the motion information; and correct the estimated rotation angles, to which the stabilization filter has been applied by the processor, based on the locations of the joints included in the motion information; wherein the processor: removes an estimated rotation angle, for which a variation in an estimated rotation angle exceeds a predetermined range between the plurality of image frames, from the estimated rotation angles of the one or more joints included in the motion information; and applies the stabilization filter having the calculated kernel to the joints of the plurality of image frames. ### Claim 8 (depends on 7) The device of claim 7, wherein the processor calculates the kernel of the stabilization filter based on time distances between a reference image frame to which the stabilization filter is applied and the image frames included in the time window and estimated rotation angles and tracked rotation angles of the joints in the image frames included in the time window. ### Claim 9 (depends on 8) The device of claim 8, wherein the processor: calculates differences in time between the reference frame to which the stabilization filter is applied and the image frames included in the time window; and calculates a difference between a variation in an estimated rotation angle and an variation in an tracked rotation angle for each of the joints in the image frames included in the time window. ### Claim 10 (depends on 9) The device of claim 9, wherein the processor: calculates the variation in the estimated rotation angle of the joint between successive image frames included in the time window, calculates a tracked rotation angle based on a connection vector connecting a location of the joint and a location of a child joint having a connection relationship with the joint, and calculates the variation in the tracked rotation angle of the joint in the successive image frames included in the time window. ### Claim 11 (depends on 7) The device of claim 7, wherein the processor: calculates the locations of the joints based on the estimated rotation angles of the joints to which the stabilization filter has been applied, and corrects the estimated rotation angles of the joints to which the stabilization filter has been applied so that the calculated locations of the joints become similar to the locations of the joints in the image frames. ### Claim 12 (depends on 11) The device of claim 11, wherein the processor: calculates a location of a child joint determined based on an estimated rotation angle of a parent joint, namely any one of the joints to which the stabilization filter has been applied, and corrects the estimated rotation angle of the parent joint to which the stabilization filter has been applied so that a connection vector connecting the calculated location of the child joint and the location of the parent joint becomes consistent with a connection vector connecting a location of the child joint and a location of the parent joint in the image frames. ### Claim 13 (depends on 1) A non-transitory computer-readable storage medium having stored thereon a program for performing the method set forth in claim 1. ### Claim 14 (depends on 1) A computer program stored in a non-transitory medium to perform the method set forth in claim 1.