# US 2023/0274580 A1 · Method and system of image processing for action classification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0274580 A1 - Document ID: US20230274580A1 - Status: Published application - Kind: A1 - Application number: 18/014,722 - Filed: 2020-08-14 - Published: 2023-08-31 - Earliest claimed: 2020-08-14 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Anbang YAO; Shandong WANG; Ming LU; Yuqing HOU; Yangyuxuan KANG; Yurong CHEN - Web page: https://patentgrep.com/patent/US20230274580A1 ## CPC - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/82: using neural networks - G06T 2207/20044: Skeletonization; Medial axis transform - G06T 2207/30196: Human being; Person - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method and system of image processing for action classification uses fine-grained motion-attributes. ## Claims ### Claim at position 1 1-25. (canceled) ### Claim 26 At least one non-transitory machine-readable medium comprising a plurality of instructions that in response to being executed on a computing device, cause the computing device to operate by: obtaining image data of frames of a video sequence with content showing at least one person in motion; determining action attribute values each as an output of a same neural network receiving input data associated with the image data, wherein different output attribute value combinations indicate different actions or none of the actions; and determining whether any of the actions are being performed by the at least one person in the content by using the attribute values. ### Claim 27 (depends on 26) The medium of claim 26, wherein the actions are different named motions by an athlete performing a sport wherein the named motions are identified in relation to the sport. ### Claim 28 (depends on 26) The medium of claim 26, wherein one or more of the attributes is a measure of the speed that a body part of the at least one person moves between at least two frames. ### Claim 29 (depends on 26) The medium of claim 26, wherein one or more of the attributes are one or more positions of points on the at least one person relative to the position of at least one other point. ### Claim 30 (depends on 26) The medium of claim 26, wherein one or more of the attributes represents an angle between body parts of the at least one person or between a body part and a reference line. ### Claim 31 (depends on 26) The medium of claim 26, wherein the instructions cause the computing device to operate by using the attribute values as input to a prediction neural network that outputs the likelihoods of multiple actions. ### Claim 32 (depends on 31) The medium of claim 31, wherein the instructions cause the computing device to operate by determining whether to use a frame as being relevant to recognition of an action depending on the likelihoods. ### Claim 33 A computer-implemented system comprising: at least one memory to store image data of frames of a video sequence with content showing at least one person in motion; and at least one processor communicatively coupled to the memory and being arranged to operate by: determining action attribute values each as an output of a same neural network receiving input data associated with the image data, wherein different output attribute value combinations of different output states indicate different actions or none of the actions, and determining whether any of the actions are being performed by the at least one person in the content by using the attribute values. ### Claim 34 (depends on 33) The system of claim 33, wherein the at least one processor is arranged to operate by using the image data to generate feature vectors, forming initial action prediction scores per frame by using the feature vectors, and using the attribute values to determine which frames to use to recognize the actions. ### Claim 35 (depends on 34) The system of claim 34, wherein the determining whether any of the actions are being performed comprises using a neural network that receives the feature vectors, initial scores, and the attribute values as input to generate likelihoods of which action is being performed. ### Claim 36 (depends on 35) The system of claim 35, wherein the attribute values are generated by using feature vectors formed by using a lower resolution version of the image data than the resolution of the image data used to form the feature vectors to form the initial scores. ### Claim 37 (depends on 33) The system of claim 33, wherein the determining whether any of the actions are being performed comprises comparing a likelihood of each action output from a neural network and for a frame to at least one criterium, and only using the frame for further action-related processing when at least one of the likelihoods passes the criterium. ### Claim 38 (depends on 37) The system of claim 37, wherein the at least one processor being arranged to discontinue use of a frame with no likelihoods meet the at least one criterium. ### Claim 39 At least one non-transitory machine-readable medium comprising a plurality of instructions that in response to being executed on a computing device, causes the computing device to operate by: obtaining image data of frames of a video sequence with content showing at least one person in motion; determining an action prediction initial score of individual frames of the video sequence; determining action attribute values each as an output of a same neural network receiving input data associated with the image data, wherein different output attribute value combinations indicate different actions or none of the actions; using the action attribute values to determine which frames of the video sequence are relevant frames that indicate an action is occurring; and providing the initial scores only of the relevant frames to perform further action-related processing. ### Claim 40 (depends on 39) The medium of claim 39, wherein the instructions cause the computing device to operate by: inputting the action attribute values, the initial scores, and feature vectors used to form the initial scores and of a single frame into an agent neural network that generates a probability an individual action is occurring in the frame, wherein the single frame is designated a relevant frame when at least one probability of the frame meets at least one criterium. ### Claim 41 (depends on 40) The medium of claim 40, wherein the agent neural network is a long short-term memory (LSTM) neural network with at least a first layer that is a fully connected layer, subsequently a second layer that is a rectifier linear unit (ReLU) layer, a third layer that is a fully connected layer, and a fourth layer that uses a sigmoid activation function to form the probabilities. ### Claim 42 (depends on 40) The medium of claim 40, wherein feature vectors used to form the action attribute values are based on image data with a lower resolution than the resolution of the image data used to form the feature vectors used to form the initial scores. ### Claim 43 (depends on 40) The medium of claim 40, wherein the instructions cause the computing device to operate by providing access to all of the probabilities of a relevant frame and for applications performing the action-related processing. ### Claim 44 (depends on 40) The medium of claim 40, wherein the instructions cause the computing device to operate by providing only the highest probability of an action among all of the probabilities of a relevant frame to applications performing the action-related processing. ### Claim 45 A method of image processing comprising: obtaining image data of frames of a video sequence with content showing at least one person in motion; determining an action prediction initial score of individual frames of the video sequence based on image data of a first resolution; determining action attribute values each as an output of a same neural network receiving input data associated with a version of the image data in a second resolution different than the first resolution, wherein different output attribute value combinations indicate different actions or none of the actions; and using the action attribute values to determine which of the frames are relevant to an action and should be provided for further action-related processing. ### Claim 46 (depends on 45) The method of claim 45, comprising inputting the action attribute values of a single frame into an agent neural network that outputs probabilities, wherein each probability is of a different action, wherein the single frame is designated a relevant frame when at least one probability of the frame meets at least one criterium. ### Claim 47 (depends on 46) The method of claim 46, wherein a plurality of initial scores of the same action and of multiple frames forming a segment of the video sequence are combined to form a single representative action score to be provided for the further action-related processing. ### Claim 48 (depends on 47) The method of claim 47, wherein a plurality of the probabilities of the same action and of multiple frames forming a segment of the video sequence are combined to form a single representative probability to be provided for the further action-related processing. ### Claim 49 (depends on 45) The method of claim 45, wherein one or more of the attributes is a measure of the speed that a body part of the at least one person moves between at least two frames. ### Claim 50 (depends on 45) The method of claim 45, wherein one or more of the attributes represents an angle between body parts of the at least one person or between a body part and a reference line.