# US 2024/0189710 A1 · Method of using physical movement for virtual gameplay control and associated apparatus > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0189710 A1 - Document ID: US20240189710A1 - Status: Published application - Kind: A1 - Application number: 18/533,255 - Filed: 2023-12-08 - Published: 2024-06-13 - Earliest claimed: 2022-12-08 - Assignees: 2354561 Alberta Ltd. - Applicants: 2354561 Alberta Ltd. - Inventors: Geoffrey Samuel FROST; Stephen KIM; Nabil BABU; Adam COLLVER; Andre DAVIAU - Web page: https://patentgrep.com/patent/US20240189710A1 ## CPC - A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/816: Athletics, e.g. track-and-field sports - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR ## Abstract A method and apparatus for using physical movement, e.g. associated with a physical therapy, for virtual gameplay control. A selection of an exercise is received from a user interface. After the selection, data associated with the exercise can be read from memory. The data includes a plurality of joints to detect and expected positions of each of the joints. A plurality of frames of a performance of a user performing the exercise are received, from sensors such as a video camera. Each of the joints within each of the frames can be detected, and a change in positions of joints can be determined. Subsequently, an indication of success that the exercise was performed correctly is sent to the user interface. A character in the game then takes an action, thus in this manner, the user can reliably control their character's in-game actions. ## Claims ### Claim 1 A method for assessing the performance of an exercise, the method comprising: receiving, from a user interface, a selection of an exercise; reading, from a non-transitory memory, data associated with the exercise, the data including indications of a plurality of joints to detect and a plurality of expected positions of each of the plurality of joints; receiving, from a video source, a plurality of frames of a performance of a user performing the exercise; detecting each of the plurality of joints within each of the plurality of frames; determining a change in positions of each of the plurality of joints over a timeframe corresponding to the plurality of frames; determining that the performance was a success based on comparing the changes in positions of each of the plurality of joints to the plurality of expected positions of each of the plurality of joints over the timeframe; and sending, to the user interface, an indication of success that the exercise was performed correctly. ### Claim 2 (depends on 1) The method of claim 1 wherein the plurality of expected positions of each of the plurality of joints includes relative positions between the plurality of expected positions of two of the plurality of joints. ### Claim 3 (depends on 2) The method of claim 2 wherein the comparing of the changes in positions of each of the plurality of joints to the plurality of expected positions of each of the plurality of joints over the timeframe includes comparing relative starting positions of each of the plurality of joints to relative ending positions of each of the plurality of joints. ### Claim 4 (depends on 1) The method of claim 1 further comprising: displaying, on the user interface, in response to sending the indication of success, a movement of a character of a computer game. ### Claim 5 (depends on 4) The method of claim 4 wherein the movement of the character comprises moving the character between horizontal lanes displayed in the computer game. ### Claim 6 (depends on 5) The method of claim 5 further comprising: displaying, on the user interface, objects within the lanes of the computer game that prompt the user to perform the exercise in order to move the character into a lane or out of a lane through a successful performance of the exercise. ### Claim 7 (depends on 1) The method of claim 1 wherein the plurality of joints to detect are selected from the group including a wrist, an elbow, and a shoulder. ### Claim 8 (depends on 1) The method of claim 1 wherein movement of each of the plurality of joints include a flexion movement or a rotational movement. ### Claim 9 (depends on 1) The method of claim 1, wherein movement of each of the plurality of joints include an elbow flexion or a shoulder rotation. ### Claim 10 (depends on 1) The method of claim 1, wherein determining that the performance was a success is based on information preloaded into the non-transitory memory. ### Claim 11 A computing system comprising: a processor; and a non-transitory memory for storing instructions that when executed by the processor cause the computing system to be configured to: receive, from a user interface, a selection of an exercise; read, from the non-transitory memory, data associated with the exercise, the data including indications of a plurality of joints to detect and a plurality of expected positions of each of the plurality of joints; receive, from a video source, a plurality of frames of a performance of a user performing the exercise; detect each of the plurality of joints within each of the plurality of frames; determine a change in positions of each of the plurality of joints over a timeframe corresponding to the plurality of frames; determine that the performance was a success based on comparing the changes in positions of each of the plurality of joints to the plurality of expected positions of each of the plurality of joints over the timeframe; and send, to the user interface, an indication of success that the exercise was performed correctly. ### Claim 12 (depends on 11) The computing system of claim 11, wherein the plurality of expected positions of each of the plurality of joints includes relative positions between the plurality of expected positions of two of the plurality of joints. ### Claim 13 (depends on 12) The computing system of claim 12 wherein the comparing of the changes in positions of each of the plurality of joints to the plurality of expected positions of each of the plurality of joints over the timeframe includes comparing relative starting positions of each of the plurality of joints to relative ending positions of each of the plurality of joints. ### Claim 14 (depends on 11) The computing system of claim 11 further configured to: display, on the user interface, in response to sending the indication of success, a movement of a character of a computer game. ### Claim 15 (depends on 4) The method of claim 4 wherein the movement of the character comprises moving the character between horizontal lanes displayed in the computer game. ### Claim 16 (depends on 5) The computing system of claim 5 further configured to: display, on the user interface, objects within the lanes of the computer game that prompt the user to perform the exercise in order to move the character into a lane or out of a lane through a successful performance of the exercise. ### Claim 17 (depends on 11) The computing system of claim 11 wherein the plurality of joints to detect are selected from the group including a wrist, an elbow, and a shoulder. ### Claim 18 (depends on 1) The method of claim 1 wherein movement of each of the plurality of joints include a flexion movement or a rotational movement. ### Claim 19 (depends on 1) The method of claim 1, wherein movement of each of the plurality of joints include an elbow flexion or a shoulder rotation. ### Claim 20 (depends on 1) A non-transitory computer readable medium having statements and instructions recorded thereon which, when executed by a computer, cause the computer to perform the method of claim 1.