# US 12,008,167 B2 · Action recognition method and device for target object, and electronic apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,167 B2
- Document ID: US12008167B2
- Status: Granted
- Kind: B2
- Application number: 17/260,072
- Filed: 2020-02-26
- Published: 2021-09-09
- Granted: 2024-06-11
- Earliest claimed: 2019-04-29
- Assignees: Beijing Bytedance Network Technology Co., Ltd.
- Applicants: Beijing Bytedance Network Technology Co., Ltd.
- Inventors: Guozhong Luo
- Web page: https://patentgrep.com/patent/US12008167B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 5/92: based on global image properties
- G06V 10/56: relating to colour
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04N 9/77: Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase (circuits for matrixing H04N9/67)
- H04N 23/611: where the recognised objects include parts of the human body
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An action recognition method and device for a target object, and an electronic apparatus is disclosed. The action recognition method for a target object comprises: obtaining an original image from an image source, the original image comprising a target object, recognizing the target object from the original image; detecting a plurality of key points of the target object; determining, on the basis of the detected key points, visibility attributes of the plurality of key points, wherein the visibility attributes are used to indicate whether or not the key points are occluded; and recognizing an action of the target object according to a combined values of the visibility attributes of the plurality of key points. The method uses visibility of key points to determine actions of a target object, thereby resolving the technical issue in the prior art in which complex actions cannot be accurately recognized.

## Claims

### Claim 1

A method for recognizing an action of a target object, comprising:
acquiring an original image from an image source, wherein the original image comprises a target object;
recognizing the target object from the original image;
detecting a plurality of key points of the target object;
determining visibility attributes of the plurality of key points based on the detected key points, wherein a visibility attribute of a key point indicates whether the key point is covered; and
recognizing an action of the target object based on a combined value of the visibility attributes of the plurality of key points,
wherein the recognizing the target object from the original image comprises:
acquiring a current image frame of the original image;
extracting image features from the image frame to form a feature image;
generating a plurality of rectangular boxes in the feature image;
determining, for each of the plurality of rectangular boxes, the number of image features of the target object that are contained in the rectangular box; and
outputting one of the plurality of rectangle boxes that contains a maximum number of image features of the target object as a circumscribing box of the target object.

### Claim 2 (depends on 1)

The method for recognizing an action of a target object according to claim 1, wherein after the recognizing an action of the target object based on a combined value of the visibility attributes of the plurality of key points, the method further comprises:
determining whether the action of the target object is a first action; and
performing, if the action of the target object is the first action, a first processing corresponding to the first action on the original image.

### Claim 3 (depends on 2)

The method for recognizing an action of a target object according to claim 2, wherein the performing, if the action of the target object is the first action, a first processing corresponding to the first action on the original image comprises:
acquiring, if the action of the target object is the first action, the first processing corresponding to the first action and a parameter of the first processing; and
performing the first processing on the original image based on the parameter of the first processing to obtain a processed image.

### Claim 4 (depends on 3)

The method for recognizing an action of a target object according to claim 3, wherein
the first processing is reducing image brightness;
the parameter of the first processing is a percentage by which the image brightness is reduced; and
the performing the first processing on the original image based on the parameter of the first processing to obtain a processed image comprises:
acquiring, for each pixel of the original image, three color components of the pixel respectively in three channels in an RGB color space; and
reducing, for each of the three color components of each pixel, the color component by the percentage to obtain processed three color components of the pixel.

### Claim 5 (depends on 1)

The method for recognizing an action of a target object according to claim 1, wherein the detecting a plurality of key points of the target object comprises:
inputting the recognized target object to a key point detecting model; and
outputting, by using the key point detecting model, coordinates of each of the plurality of key points on the original image.

### Claim 6 (depends on 1)

The method for recognizing an action of a target object according to claim 1, wherein the determining visibility attributes of the plurality of key points based on the detected key points, wherein a visibility attribute of a key point indicates whether the key point is covered, comprises:
inputting the target object of which a plurality of key points are detected to a visibility determining model;
outputting, by using the visibility determining model, a visibility probability of each of the plurality of key points; and
comparing, for each of the plurality of key points, the visibility probability of the key point with a first threshold and outputting a value for the visibility attribute of the key point.

### Claim 7 (depends on 1)

The method for recognizing an action of a target object according to claim 1, wherein the recognizing an action of the target object based on a combined value of the visibility attributes of the plurality of key points comprises:
acquiring a reference value for the combined value of the visibility attributes of the plurality of key points, wherein the reference value corresponds to an action;
comparing the combined value of the visibility attributes with the reference value; and
outputting, if the combined value matches the reference value, the action corresponding to the reference value as the recognized action of the target object.

### Claim 8 (depends on 1)

The method for recognizing an action of a target object according to claim 1, wherein the recognizing an action of the target object based on a combined value of the visibility attributes of the plurality of key points comprises:
acquiring a variation reference value for combined values of the visibility attributes of the plurality of key points, wherein the variation reference value for combined values corresponds to an action;
acquiring two combined values of the visibility attributes of the plurality of key points respectively in two consecutive image frames;
calculating a variation value of the combined values;
comparing the variation value of the two combined values with the variation reference value for combined values; and
outputting, if the variation value of the two combined values matches the variation reference value for combined values, the action corresponding to the variation reference value as the recognized action of the target object.

### Claim 9

A device for recognizing an action of a target object, comprising:
a memory configured to store computer readable instructions; and
a processor configured to execute the computer readable instructions causing the device to:
acquire an original image from an image source, wherein the original image comprises a target object;
recognize the target object from the original image;
detect a plurality of key points of the target object;
determine visibility attributes of the plurality of key points based on the detected key points wherein a visibility attribute of a key point indicates whether the key point is covered; and
recognize an action of the target object based on a combined value of the visibility attributes of the plurality of key points,
wherein the recognizing the target object from the original image comprises:
acquiring a current image frame of the original image;
extracting image features from the image frame to form a feature image;
generating a plurality of rectangular boxes in the feature image;
determining, for each of the plurality of rectangular boxes, the number of image features of the target object that are contained in the rectangular box; and
outputting one of the plurality of rectangle boxes that contains a maximum number of image features of the target object as a circumscribing box of the target object.

### Claim 10

A non-transient computer readable storage media configured to store computer readable instructions that when being executed by a computer, cause the computer to perform the operations of:
acquiring an original image from an image source, wherein the original image comprises a target object;
recognizing the target object from the original image;
detecting a plurality of key points of the target object;
determining visibility attributes of the plurality of key points based on the detected key points, wherein a visibility attribute of a key point indicates whether the key point is covered; and
recognizing an action of the target object based on a combined value of the visibility attributes of the plurality of key points,
wherein the recognizing the target object from the original image comprises:
acquiring a current image frame of the original image;
extracting image features from the image frame to form a feature image;
generating a plurality of rectangular boxes in the feature image;
determining, for each of the plurality of rectangular boxes, the number of image features of the target object that are contained in the rectangular box; and
outputting one of the plurality of rectangle boxes that contains a maximum number of image features of the target object as a circumscribing box of the target object.