# US 11,443,141 B2 · Using video tracking technology to create machine learning datasets for tasks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,141 B2 - Document ID: US11443141B2 - Status: Granted - Kind: B2 - Application number: 16/802,893 - Filed: 2020-02-27 - Published: 2021-09-02 - Granted: 2022-09-13 - Earliest claimed: 2020-02-27 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Justin Eyster; Al Chakra; Aniruddh Jhavar; Patrick Morrison; Gagandeep Rajpal - Web page: https://patentgrep.com/patent/US11443141B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06V 10/7747: {Organisation of the process, e.g. bagging or boosting} - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/08: Learning methods - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method, system and computer-usable medium are disclosed for tracking selected points in a series of images to determine motions made by a subject to perform an action to train a system, such as a machine or robot. A series of images are received depicting incremental steps of the subject performing the action. Selected points that are useful to track the subject performing the action are identified. Datasets of points used to train a model are mapped, and the model is trained using the mapped datasets of points. ## Claims ### Claim 1 A method for tracking selected points in a series of video images of captured video to determine motions made by a subject to perform an action to train a system to perform the action comprising: receiving the series of video images of the captured video, wherein successive video images depict incremental steps of the subject performing the action; identifying selected key points associated with the subject useful for tracking the subject performing the action, wherein the key points associated with the subject are tracked over time and adjusted based on a sampling frequency of the captured video; mapping datasets of points to be used to train a model; using physical sensors for live reinforcement learning; and training the model utilizing the mapped datasets of points and live reinforcement training. ### Claim 2 (depends on 1) The computer implemented method of claim 1, wherein the datasets of points comprise a first dataset of points as to the subject and a second data set of points of an object related to action. ### Claim 3 (depends on 1) The computer implemented method of claim 1 further comprising receiving relative position of points in the series of video images using virtual sensors. ### Claim 4 (depends on 3) The computer implemented method of claim 3, wherein the virtual sensors provide the datasets of points used to train the model. ### Claim 5 (depends on 1) The computer implemented method of claim 1, wherein the series of video images are from video and the system that performs the action is a machine or robot. ### Claim 6 (depends on 1) The computer implemented method of claim 1 further comprising performing a live reinforcement training of the system to refine the action. ### Claim 7 (depends on 1) The computer implemented method of claim 1 further comprising implementing physical sensors to capture data not available in the video images and related to performing the action. ### Claim 8 A system comprising: a processor; a data bus coupled to the processor; and a computer-usable medium embodying computer program code, the computer-usable medium being coupled to the data bus, the computer program code used for tracking selected points in a series of video images of capture video to determine motions made by a subject to perform an action to train a system to perform the action comprising instructions executable by the processor and configured for: receiving the series of video images of the captured video, wherein successive video images depict incremental steps of the subject performing the action; identifying selected key points associated with the subject useful for tracking the subject performing the action, wherein the key points associated with the subject are tracked over time and adjusted based on a sampling frequency of the captured video; mapping datasets of points to be used to train a model; using physical sensors for live reinforcement learning; and training the model utilizing the mapped datasets of points and live reinforcement training. ### Claim 9 (depends on 8) The system of claim 8, wherein the datasets of points comprise a first dataset of points as to the subject and a second data set of points of an object related to action. ### Claim 10 (depends on 8) The system of claim 8 further comprising receiving relative position of points in the series of video images using virtual sensors. ### Claim 11 (depends on 10) The system of claim 10, wherein the virtual sensors provide the datasets of points used to train the model. ### Claim 12 (depends on 8) The system of claim 8, wherein the series of video images are from video and the system that performs the action is a machine or robot. ### Claim 13 (depends on 8) The system of claim 8 further comprising performing a live reinforcement training of the system to refine the action. ### Claim 14 (depends on 8) The system of claim 8 further comprising implementing physical sensors to capture data not available in the video images and related to performing the action. ### Claim 15 A non-transitory, computer-readable storage medium embodying computer program code for tracking selected points in a series of video images of captured video to determine motions made by a subject to perform an action to train a system to perform the action, the computer program code comprising computer executable instructions configured for: receiving the series of video images of the captured video, wherein successive video images depict incremental steps of the subject performing the action; identifying selected key points associated with the subject useful for tracking the subject performing the action, wherein the key points associated with the subject are tracked over time and adjusted based on a sampling frequency of the captured video; mapping datasets of points to be used to train a model; using physical sensors for live reinforcement learning; and training the model utilizing the mapped datasets of points and live reinforcement training. ### Claim 16 (depends on 15) The non-transitory, computer-readable storage medium of claim 15, wherein the datasets of points comprise a first dataset of points as to the subject and a second data set of points of an object related to action. ### Claim 17 (depends on 15) The non-transitory, computer-readable storage medium of claim 15 further comprising receiving relative position of points in the series of video images using virtual sensors. ### Claim 18 The non-transitory, computer-readable storage medium of 17, wherein the virtual sensors provide the datasets of points used to train the model. ### Claim 19 The non-transitory, computer-readable storage medium of 15, wherein the series of video images are from video and the system that performs the action is a machine or robot. ### Claim 20 The non-transitory, computer-readable storage medium of 15 further comprising performing a live reinforcement training of the system to refine the action.