# US 11,907,679 B2 · Arithmetic operation device using a machine learning model, arithmetic operation method using a machine learning model, and training method of the machine learning model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,679 B2 - Document ID: US11907679B2 - Status: Granted - Kind: B2 - Application number: 16/818,823 - Filed: 2020-03-13 - Published: 2021-03-25 - Granted: 2024-02-20 - Earliest claimed: 2019-09-19 - Assignees: Kioxia Corporation - Applicants: Kioxia Corporation - Inventors: Kengo Nakata; Asuka Maki; Daisuke Miyashita - Web page: https://patentgrep.com/patent/US11907679B2 ## CPC - G06F 7/48: using non-contact-making devices, e.g. tube, solid state device; using unspecified devices - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06N 5/04: Inference or reasoning models - G06N 20/20: Ensemble learning - G06N 3/048: Activation functions - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract An arithmetic operation device is provided that removes a part of parameters of a predetermined number of parameters from a first machine learning model which includes the predetermined number of parameters and is trained so as to output second data corresponding to input first data, determines the number of bits of a weight parameter according to required performance related to an inference to generate a second machine learning model, and acquires data output from the second machine learning model so as to correspond to the input first data with a smaller computational complexity than the first machine learning model. ## Claims ### Claim 1 An arithmetic operation device, comprising: a memory configured to store a first machine learning model that includes a predetermined number of parameters and is trained so as to output second data corresponding to input first data; processing circuitry configured to execute an arithmetic operation using the first machine learning model stored in the memory; and an interface connected to the processing circuitry, wherein the processing circuitry is further configured to receive the first data and a required performance related to an inference through the interface, the required performance being provided from outside the arithmetic operation device and including an accuracy of inference using a machine learning model and a computational complexity based on a product of a number of multiply-accumulate computations and a number of bits of weight parameters, remove a part of parameters of the predetermined number of parameters from the first machine learning model and decrease the number of bits of weight parameters according to the required performance to generate a second machine learning model, wherein a second accuracy of inference using the second machine learning model is lower than a first accuracy of inference using the first machine learning model but satisfies the received required performance, input the first data into the second machine learning model to acquire third data output from the second machine learning model with a smaller computational complexity than the first machine learning model, and output the third data through the interface. ### Claim 2 (depends on 1) The arithmetic operation device according to claim 1, wherein the first data are image data, and data output corresponding to the input first data are an inference result related to a classification of the image data in the first machine learning model and the second machine learning model. ### Claim 3 An arithmetic operation method performed by an arithmetic operation device, the arithmetic operation device including a memory configured to store a first machine learning model that includes a predetermined number of parameters and is trained so as to output second data corresponding to input first data, processing circuitry configured to execute the arithmetic operation method, and an interface connected to the processing circuitry, the method comprising: receiving the first data and a required performance related to an inference through the interface, the required performance being provided from outside the arithmetic operation device and including an accuracy of inference using a machine learning model and a computational complexity based on a product of a number of multiply-accumulate computations and a number of bits of weight parameters; removing a part of parameters of the predetermined number of parameters from the first machine learning model and decreasing the number of bits of weight parameters according to the required performance to generate a second machine learning model, wherein a second accuracy of inference using the second machine learning model is lower than a first accuracy of inference using the first machine learning model but satisfies the received required performance; inputting the first data into the second machine learning model to acquire third data output from the second machine learning model with a smaller computational complexity than the first machine learning model; and outputting the third data through the interface. ### Claim 4 (depends on 1) The arithmetic operation device according to claim 1, wherein the parameters include at least one of a neuron, a weight parameter, and an intermediate layer. ### Claim 5 (depends on 4) The arithmetic operation device according to claim 4, wherein the removed part of the parameters includes at least one neuron, at least one weight parameter, or at least one intermediate layer. ### Claim 6 (depends on 1) The arithmetic operation device according to claim 1, wherein the first machine learning model is a machine learning model trained while removing the part of the parameters of the predetermined number of parameters from a machine learning model having the predetermined number of parameters for each training data. ### Claim 7 (depends on 6) The arithmetic operation device according to claim 6, wherein the part of the parameters of the predetermined number of parameters are randomly removed from the machine learning model having the predetermined number of parameters. ### Claim 8 (depends on 6) The arithmetic operation device according to claim 6, wherein the first machine learning model is generated by re-training the trained machine learning model having the predetermined number of parameters. ### Claim 9 (depends on 1) The arithmetic operation device according to claim 1, wherein the part of the parameters of the predetermined number of parameters is removed based on a relationship between inference accuracy and a computational complexity related to the first machine learning model. ### Claim 10 (depends on 9) The arithmetic operation device according to claim 9, wherein the memory is further configured to store the parameters of the first machine learning model and the relationship between the inference accuracy and the computational complexity related to the first machine learning model. ### Claim 11 (depends on 10) The arithmetic operation device according to claim 10, wherein the relationship between the inference accuracy and the computational complexity related to the first machine learning model is stored in association with a parameter that is removed during the inference among the predetermined number of parameters. ### Claim 12 (depends on 1) The arithmetic operation device according to claim 1, wherein the first machine learning model is a machine learning model trained while adding a parameter to a third machine learning model having a smaller number of parameters than the predetermined number for each training data. ### Claim 13 (depends on 12) The arithmetic operation device according to claim 12, wherein the parameter is randomly added to the third machine learning model. ### Claim 14 (depends on 12) The arithmetic operation device according to claim 12, wherein the third machine learning model is generated by removing the part of the parameters of the predetermined number of parameters from a trained machine learning model having the predetermined number of parameters, and the added parameter is one of parameters removed when the third machine learning model is generated. ### Claim 15 (depends on 12) The arithmetic operation device according to claim 12, wherein the third machine learning model is a trained machine learning model, and the first machine learning model is generated by re-training the third machine learning model. ### Claim 16 (depends on 12) The arithmetic operation device according to claim 12, wherein the first machine learning model is trained by updating the added parameter. ### Claim 17 (depends on 16) The arithmetic operation device according to claim 16, wherein the first machine learning model is trained by updating a parameter of a normalization layer of the third machine learning model in addition to the added parameter. ### Claim 18 (depends on 3) The arithmetic operation method according to claim 3, wherein the first machine learning model is a machine learning model trained while adding a parameter to a third machine learning model having a smaller number of parameters than the predetermined number for each training data. ### Claim 19 A training method of a machine learning model having a predetermined number of parameters, the training method being performed by a training device, the training device including a memory storing the machine learning model, processing circuitry configured to execute the training the machine learning model using the memory, and an interface connected to the processing circuitry, the method comprising: receiving training data through the interface; training the predetermined number of parameters for the received training data by using the machine learning model to generate a first machine learning model having a plurality of characteristics, wherein the training includes training a parameter other than a part of parameters of the predetermined number of parameters while removing the part of the parameters multiple times; performing a plurality of inferences on the first machine learning model having the predetermined number of parameters generated by the training performed multiple times under a plurality of conditions in which another part of parameters different from each other are removed among the predetermined number of parameters; calculating a relationship between a computational complexity and an accuracy of inference using the first machine learning model for the conditions based on results of the plurality of inferences, the computational complexity being based on a product of a number of multiply-accumulate computations and a number of bits of weight parameters; generating characteristic data indicating the relationship between the computational complexity and the accuracy of inference related to the first machine learning model, wherein the characteristic data allows specifying the computational complexity from the accuracy of inference; and outputting through the interface, the first machine learning model and the characteristic data. ### Claim 20 (depends on 1) The arithmetic operation device according to claim 1, wherein the first machine learning model and the second machine learning model are convolutional neural network models.