# US 11,307,865 B2 · Data processing apparatus and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,865 B2
- Document ID: US11307865B2
- Status: Granted
- Kind: B2
- Application number: 16/698,997
- Filed: 2019-11-28
- Published: 2020-04-02
- Granted: 2022-04-19
- Earliest claimed: 2017-09-06
- Assignees: Shanghai Cambricon Information Technology Co., Ltd.
- Applicants: Shanghai Cambricon Information Technology Co., Ltd.
- Inventors: Tianshi Chen; Haoyuan He; Shuai Hu
- Web page: https://patentgrep.com/patent/US11307865B2

## CPC

- G06V 40/168: {Feature extraction; Face representation}
- G06F 1/3296: by lowering the supply or operating voltage
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3004: {to perform operations on memory}
- G06F 9/30047: {Prefetch instructions; cache control instructions}
- G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution}
- G06F 9/3893: {controlled in tandem, e.g. multiplier-accumulator}
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/505: {considering the load}
- G06F 9/546: {Message passing systems or structures, e.g. queues}
- G06F 11/0751: {Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06)}
- G06F 11/1004: {to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00)}
- G06F 11/1044: {with specific ECC/EDC distribution}
- G06F 11/186: {Passive fault masking when reading multiple copies of the same data}
- G06F 11/3024: {where the computing system component is a central processing unit [CPU]}
- G06F 11/3058: {Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations (thermal management in cooling arrangements of a computing system G06F1/206)}
- G06F 11/3062: {where the monitored property is the power consumption (power management in a computing system G06F1/3203)}
- G06F 12/0875: with dedicated cache, e.g. instruction or stack
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 18/24: Classification techniques
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 40/174: {Facial expression recognition}
- G06F 1/324: by lowering clock frequency
- G06F 7/57: Arithmetic logic units [ALU], i.e. arrangements or devices for performing two or more of the operations covered by groups G06F7/483 – G06F7/556 or for performing logical operations {(G06F7/49, G06F7/491 take precedence)}
- G06F 9/3001: {Arithmetic instructions}
- G06F 11/1612: {where the redundant component is persistent storage}
- G06F 11/167: {Error detection by comparing the memory output}
- G06F 11/3051: {Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs (verification or detection of system hardware configuration G06F11/2247)}
- G06F 2201/81: Threshold
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06F 2212/452: Instruction code
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/047: Probabilistic or stochastic networks
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

The disclosure provides a data processing device and method. The data processing device may include: a task configuration information storage unit and a task queue configuration unit. The task configuration information storage unit is configured to store configuration information of tasks. The task queue configuration unit is configured to configure a task queue according to the configuration information stored in the task configuration information storage unit. According to the disclosure, a task queue may be configured according to the configuration information.

## Claims

### Claim 1

An information processing device comprising
a storage unit configured to store data and an instruction; and
a data processing unit connected with the storage unit and configured to:
receive the data and the instruction sent by the storage unit,
perform extraction and computational processing on key features included in the data, and
generate a multidimensional vector according to a computational processing result,
wherein the multidimensional vector is an emotion vector, and each element included in the multidimensional vector represents an emotion on the face, wherein the emotion includes anger, delight, pain, depression, sleepiness, or doubt,
wherein a value of each element of the emotion vector is a number between zero and one, and the value represents a probability of appearance of an emotion corresponding to the element, or the value of each element of the emotion vector is a non-infinite number greater than or equal to zero, and represents an intensity of the emotion corresponding to the element, or a value of an element of the emotion vector is one, values of the other elements are zero, and the emotion vector represents a strongest emotion.

### Claim 2 (depends on 1)

The information processing device of claim 1, wherein the key features include a facial action, a facial expression, and a corresponding position thereof.

### Claim 3 (depends on 1)

The information processing device of claim 1, wherein the input data includes one or more images, and the data processing unit generates a multidimensional vector for each image according to the computational processing result.

### Claim 4 (depends on 1)

The information processing device of claim 1 further comprising a conversion module configured to convert the multidimensional vector into an output, wherein the output includes a control instruction, data, a tag, or a picture.

### Claim 5 (depends on 1)

The information processing device of claim 1,
wherein the storage unit is configured to input one or more images, wherein each image corresponds to a real emotion vector, and
wherein the data processing unit is configured to:
receive the one or more images,
calculate a predicted emotion vector in a format the same as an input, and
update a parameter of the information processing device according to a
comparison result of the predicted emotion vector and the real emotion vector.

### Claim 6 (depends on 5)

The information processing device of claim 5, wherein the data processing unit includes: an operation unit configured to execute corresponding computations on the data according to an instruction stored in the storage unit,
wherein the operation unit includes:
a multiplier configured to multiply data input therein to obtain a multiplication output, and/or one or more adders configured to add data input to obtain output data, the multiple adders forming an adder tree configured to add the data input step by step to obtain the output data, and an activation function unit configured to execute computation through an activation function to obtain activation output data, the activation function including sigmoid, tanh, rectified linear unit (RELU), or softmax.

### Claim 7 (depends on 6)

The information processing device of claim 6, wherein the storage unit is configured to store the data and the instruction, wherein the data includes a neuron, a weight, the image, and the vector, the information processing device further includes:
a preprocessing unit configured to preprocess original input data, i.e., one or more images, to obtain image data consistent with a preset parameter and data format of an artificial neural network, preprocessing including segmentation, Gaussian filtering, binarization, regularization, and normalization,
an instruction cache unit configured to cache instructions,
a weight cache unit configured to cache weight data,
an input neuron cache unit configured to cache an input neuron input into the operation unit, an output neuron cache unit configured to cache an output neuron output by the operation unit,
a control unit configured to read the instruction from the instruction cache unit, decode the instruction into an operation unit instruction, and input the operation unit instruction to the operation unit, and
a DMA (Direct Memory Access) configured to read and write data or instructions in the storage unit, the instruction cache unit, the weight cache unit, the input neuron cache unit, and the output neuron cache unit.

### Claim 8

An information processing method, comprising:
receiving and storing, by a storage unit, data and an instruction; and
receiving, by a data processing unit, the data and the instruction sent by the storage unit,
performing, by the data processing unit, extraction and computational processing on key features included in the data, and
generating, by the data processing unit, a multidimensional vector according to a computational processing result,
wherein the multidimensional vector is an emotion vector, each element included in the multidimensional vector represents an emotion on the face, and the emotion includes anger, delight, pain, depression, sleepiness, or doubt, and
wherein a value of each element of the emotion vector is a number between zero and one, and the value represents a probability of appearance of an emotion corresponding to the element, or the value of each element of the emotion vector is a non-infinite number greater than or equal to zero, and the value represents an intensity of the emotion corresponding to the element, or a value of an element of the emotion vector is one, values of the other elements are zero, and the emotion vector represents a strongest emotion.

### Claim 9 (depends on 8)

The information processing method of claim 8, wherein the key features include a facial action, a facial expression, and corresponding positions thereof.

### Claim 10 (depends on 8)

The information processing method of claim 8, wherein the data includes one or more images, and the data processing unit generates a multidimensional vector for each image according to the computational processing result.

### Claim 11 (depends on 8)

The information processing method of claim 8, further comprising converting, by a conversion module, the multidimensional vector into an output, wherein the output includes a control instruction, data, a tag, or a picture.

### Claim 12 (depends on 11)

The information processing method of claim 11, comprising
inputting one or more images into the storage unit, wherein each image corresponds to a real emotion vector, and
receiving, by the data processing unit, the one or more images sent by the storage unit,
calculating, by the data processing unit, a predicted emotion vector, in a format the same as an input,
comparing, by the data processing unit, the predicted emotion vector with the real emotion vector, and
updating, by the data processing unit, a parameter of the information processing method according to a comparison result.

### Claim 13 (depends on 12)

The information processing method of claim 12, further comprising:
preprocessing, by a preprocessing unit, the one or more images, to obtain image data consistent with an input layer scale of a bottom layer of an artificial neural network used by a chip, wherein the preprocessing includes segmentation, Gaussian filtering, binarization, regularization, and normalization,
transmitting, by the input data, into the storage unit through the preprocessing unit or directly transmitting into the storage unit, transmitting, by the instruction, into the storage unit, wherein the data includes a neuron, a weight, the image, and the vector,
receiving, by a DMA, the data sent by the storage unit and transmitting, by the DMA, into a corresponding instruction cache unit, input neuron cache unit and weight cache unit in batches, reading and decoding, by the control unit, the instruction from the instruction cache unit,
receiving, by the operation unit, the decoded instruction and executing, by the operation unit, corresponding computation according to the instruction,
executing, by the operation unit, neural network operation until computation for all the data is completed to obtain a final computational result, wherein the final computational result is obtained by an output neuron of a last layer of the neural network and each neuron of a final output layer of the neural network corresponding to an element of the emotion vector,
transmitting, by the final computational result, into an output neuron cache unit, and
transmitting, by the output neuron cache unit, the final computational result into the DMA, and returning, by the DMA, the final computational result to the storage unit.

### Claim 14 (depends on 8)

The information processing method of claim 8, wherein an operation unit of the data processing unit executes corresponding computation on the data according to the instruction stored in the storage unit:
executing a multiplication computation to multiply an input neuron and weight data, executing an addition computation to add a multiplication computational result step by step through an adder tree to obtain a weighted sum, and selectively adding an offset to the weighted sum, and
executing an activation function computation to obtain an output neuron, wherein the activation function is a sigmoid function, a tanh function, an RELU function, or a softmax function.

### Claim 15

An electronic device, comprising:
an information processing device that includes:
a storage unit configured to store data and an instruction; and
a data processing unit connected with the storage unit and configured to:
receive the data and the instruction sent by the storage unit,
perform extraction and computational processing on key features included in the data, and
generate a multidimensional vector according to a computational processing result,
wherein the multidimensional vector is an emotion vector, and each element included in the multidimensional vector represents an emotion on the face, wherein the emotion includes anger, delight, pain, depression, sleepiness, or doubt,
wherein a value of each element of the emotion vector is a number between zero and one, and the value represents a probability of appearance of an emotion corresponding to the element, or the value of each element of the emotion vector is a non-infinite number greater than or equal to zero, and represents an intensity of the emotion corresponding to the element, or a value of an element of the emotion vector is one, values of the other elements are zero, and the emotion vector represents a strongest emotion.

### Claim 16 (depends on 15)

The electronic device of claim 15, wherein the input data includes one or more images, and the data processing unit generates a multidimensional vector for each image according to the computational processing result, and wherein the image includes a static picture, pictures forming a video, or a video, wherein the static picture, the pictures forming the video, or the video include image of one or more parts of a face.