# US 10,311,294 B1 · Motion recognition via a two-dimensional symbol having multiple ideograms contained therein

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,294 B1
- Document ID: US10311294B1
- Status: Granted
- Kind: B1
- Application number: 16/290,869
- Filed: 2019-03-02
- Published: 2019-06-27
- Granted: 2019-06-04
- Earliest claimed: 2017-08-03
- Assignees: Gyrfalcon Technology Inc.
- Applicants: Gyrfalcon Technology Inc.
- Inventors: Lin Yang; Patrick Z. Dong; Baohua Sun
- Web page: https://patentgrep.com/patent/US10311294B1

## CPC

- G06N 3/08: Learning methods
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 1/005: {Robust watermarking, e.g. average attack or collusion attack resistant}
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 10/955: {using specific electronic processors}
- G06V 30/19173: {Classification techniques}
- G06V 40/107: {Static hand or arm}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods of recognizing motions of an object in a video clip or an image sequence are disclosed. A plurality of frames are selected out of a video clip or an image sequence of interest. A text category is associated with each frame by applying an image classification technique with a trained deep-learning model for a set of categories containing various poses of an object within each frame. A “super-character” is formed by embedding respective text categories of the frames as corresponding ideograms in a 2-D symbol having multiple ideograms contained therein. Particular motion of the object is recognized by obtaining the meaning of the “super-character” with image classification of the 2-D symbol via a trained convolutional neural networks model for various motions of the object derived from specific sequential combinations of text categories. Ideograms may contain imagery data instead of text categories, e.g., detailed images or reduced-size images.

## Claims

### Claim 1

A method of recognizing motions of an object in a video clip or an image sequence comprising:
selecting a plurality of frames out of a video clip or an image sequence of interest;
obtaining a reduced-size image in each frame by pooling imagery data of said each frame to a desired size such that an object of interest including the object's pose is recognized in the reduced-size image;
forming a super-character by embedding respective reduced-size images of the plurality of frames as corresponding ideograms in a two-dimensional (2-D) symbol having multiple ideograms contained therein and the super-character representing a meaning formed from a specific combination of the multiple ideograms; and
recognizing a particular motion of the object by obtaining the meaning of the super-character with image classification of the 2-D symbol via a trained convolutional neural networks model for various motions of the object derived from specific sequential combinations of respective reduced-size images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the 2-D symbol being a matrix of N×N pixels of K-bit data and the matrix being divided into M×M sub-matrices with each of the sub-matrices containing (N/M)×(N/M) pixels, said each of the sub-matrices representing one ideogram, where K, N and M are positive integers, and N is a multiple of M.

### Claim 3 (depends on 2)

The method of claim 2, wherein K is 5, N is 224, M is 4, M×M is 16 and N/M is 56.

### Claim 4 (depends on 2)

The method of claim 2, wherein K is 5, N is 224, M is 8, M×M is 64 and N/M is 28.

### Claim 5 (depends on 1)

The method of claim 1, wherein the trained convolutional neural networks model comprises bi-valued 3×3 filter kernels in a Cellular Neural Networks or Cellular Nonlinear Networks (CNN) based integrated circuit.

### Claim 6 (depends on 5)

The method of claim 5, wherein the trained convolutional neural networks model is achieved with following operations:
(a) obtaining a convolutional neural networks model by training the convolutional neural networks model based on image classification of a labeled dataset, which contains a number of multi-layer 2-D symbols, the convolutional neural networks model including multiple ordered filter groups, each filter in the multiple ordered filter groups containing a standard 3×3 filter kernel;
(b) modifying the convolutional neural networks model by converting the respective standard 3×3 filter kernels to corresponding bi-valued 3×3 filter kernels of a currently-processed filter group in the multiple ordered filter groups based on a set of kernel conversion schemes;
(c) retraining the modified convolutional neural networks model until a desired convergence criterion is met; and
(d) repeating (b)-(c) for another filter group until all of the multiple ordered filter groups are converted to the bi-valued 3×3 filter kernels.