# US 2020/0401896 A1 · Generating larger neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401896 A1 - Document ID: US20200401896A1 - Status: Published application - Kind: A1 - Application number: 16/915,502 - Filed: 2020-06-29 - Published: 2020-12-24 - Earliest claimed: 2015-11-12 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Ian Goodfellow; Tianqi Chen; Jonathon Shlens - Web page: https://patentgrep.com/patent/US20200401896A1 ## CPC - G06N 3/045: Combinations of networks - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The specification describes methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a larger neural network from a smaller neural network. One of the described methods includes obtaining data specifying an original neural network and generating a larger neural network from the original neural network. The larger neural network has a larger neural network structure than the original neural network structure. The values of the parameters of the original neural network units and the additional neural network units are initialized so that the larger neural network generates the same outputs from the same inputs as the original neural network, and the larger neural network is trained to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim 21 A method performed by one or more computers, the method comprising: identifying an original neural network structure for an original neural network, the original neural network being configured to generate neural network outputs from neural network inputs, the original neural network structure comprising a plurality of original neural network units, each original neural network unit having respective parameters, and each of the parameters of each of the original neural network units having a respective original value; generating a larger neural network having a larger neural network structure, the larger neural network structure comprising: (i) the plurality of original neural network units, and (ii) a plurality of additional neural network units not in the original neural network structure, each additional neural network unit having respective parameters; initializing values of the parameters of the original neural network units and the additional neural network units by setting the values of the parameters of the original neural network units and the additional neural network units to values that result in the larger neural network generating, for any particular neural network input, the same neural network output for the particular neural network input as would be generated by the original neural network by processing the particular neural network input in accordance with the original parameter values for the original neural network units; and training the larger neural network to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values. ### Claim 22 (depends on 21) The method of claim 21, further comprising: training the original neural network to determine the original values of the parameters of the original neural network. ### Claim 23 (depends on 22) The method of claim 22, wherein the original neural network structure comprises a first original neural network layer having a first number of original units, and wherein generating the larger neural network comprises: adding a plurality of additional neural network units to the first original neural network layer to generate a larger neural network layer. ### Claim 24 (depends on 23) The method of claim 23, wherein initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the original neural network units in the larger neural network layer to the respective original values for the parameters; and for each additional neural network unit in the larger neural network layer: selecting an original neural network unit in the original neural network layer, and initializing the values of the parameters of the additional neural network unit to be the same as the respective original values for the selected original neural network unit. ### Claim 25 (depends on 24) The method of claim 24, wherein selecting an original neural network unit in the larger neural network layer comprises: randomly selecting an original neural network unit from the original neural network units in the original neural network layer. ### Claim 26 (depends on 24) The method of claim 24, wherein: in the original neural network structure, a second original neural network layer is configured to receive as input outputs generated by the first original neural network layer; in the larger neural network structure, the second original neural network layer is configured to receive as input outputs generated by the larger neural network layer; and initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the original neural network units in the second original neural network layer so that, for a given neural network input, the second neural network layer generates the same output in both the original neural network structure and the larger neural network structure. ### Claim 27 (depends on 26) The method of claim 26, wherein the original neural network structure comprises a third original neural network layer configured to receive a third original layer input and generate a third original layer output from the third layer input, and wherein generating the larger neural network comprises: replacing the third original neural network layer with a first additional neural network layer having additional neural network units and a second additional neural network layer having additional neural network units, wherein: the first additional neural network layer is configured to receive the third original layer input and generate a first additional layer output from the third original layer input, and the second additional neural network layer is configured to receive the first additional layer output and generate a second additional layer output from the first additional layer output. ### Claim 28 (depends on 27) The method of claim 27, wherein initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the additional neural network units in the first additional neural network layer and in the second additional neural network layer so that, for the same neural network input, the second additional layer output is the same as the third original layer output. ### Claim 29 (depends on 27) The method of claim 27, wherein initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the additional neural network units in the first additional neural network layer using the respective original values for the parameters of the original neural network units in the third original neural network layer. ### Claim 30 A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: identifying an original neural network structure for an original neural network, the original neural network being configured to generate neural network outputs from neural network inputs, the original neural network structure comprising a plurality of original neural network units, each original neural network unit having respective parameters, and each of the parameters of each of the original neural network units having a respective original value; generating a larger neural network having a larger neural network structure, the larger neural network structure comprising: (i) the plurality of original neural network units, and (ii) a plurality of additional neural network units not in the original neural network structure, each additional neural network unit having respective parameters; initializing values of the parameters of the original neural network units and the additional neural network units by setting the values of the parameters of the original neural network units and the additional neural network units to values that result in the larger neural network generating, for any particular neural network input, the same neural network output for the particular neural network input as would be generated by the original neural network by processing the particular neural network input in accordance with the original parameter values for the original neural network units; and training the larger neural network to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values. ### Claim 31 (depends on 30) The system of claim 30, further comprising: training the original neural network to determine the original values of the parameters of the original neural network. ### Claim 32 (depends on 31) The system of claim 31, wherein the original neural network structure comprises a first original neural network layer having a first number of original units, and wherein generating the larger neural network comprises: adding a plurality of additional neural network units to the first original neural network layer to generate a larger neural network layer. ### Claim 33 (depends on 32) The system of claim 32, wherein initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the original neural network units in the larger neural network layer to the respective original values for the parameters; and for each additional neural network unit in the larger neural network layer: selecting an original neural network unit in the original neural network layer, and initializing the values of the parameters of the additional neural network unit to be the same as the respective original values for the selected original neural network unit. ### Claim 34 (depends on 33) The system of claim 33, wherein selecting an original neural network unit in the larger neural network layer comprises: randomly selecting an original neural network unit from the original neural network units in the original neural network layer. ### Claim 35 (depends on 33) The system of claim 33, wherein: in the original neural network structure, a second original neural network layer is configured to receive as input outputs generated by the first original neural network layer; in the larger neural network structure, the second original neural network layer is configured to receive as input outputs generated by the larger neural network layer; and initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the original neural network units in the second original neural network layer so that, for a given neural network input, the second neural network layer generates the same output in both the original neural network structure and the larger neural network structure. ### Claim 36 (depends on 35) The system of claim 35, wherein the original neural network structure comprises a third original neural network layer configured to receive a third original layer input and generate a third original layer output from the third layer input, and wherein generating the larger neural network comprises replacing the third original neural network layer with a first additional neural network layer having additional neural network units and a second additional neural network layer having additional neural network units, wherein: the first additional neural network layer is configured to receive the third original layer input and generate a first additional layer output from the third original layer input, and the second additional neural network layer is configured to receive the first additional layer output and generate a second additional layer output from the first additional layer output. ### Claim 37 A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: identifying an original neural network structure for an original neural network, the original neural network being configured to generate neural network outputs from neural network inputs, the original neural network structure comprising a plurality of original neural network units, each original neural network unit having respective parameters, and each of the parameters of each of the original neural network units having a respective original value; generating a larger neural network having a larger neural network structure, the larger neural network structure comprising: (i) the plurality of original neural network units, and (ii) a plurality of additional neural network units not in the original neural network structure, each additional neural network unit having respective parameters; initializing values of the parameters of the original neural network units and the additional neural network units by setting the values of the parameters of the original neural network units and the additional neural network units to values that result in the larger neural network generating, for any particular neural network input, the same neural network output for the particular neural network input as would be generated by the original neural network by processing the particular neural network input in accordance with the original parameter values for the original neural network units; and training the larger neural network to determine trained values of the parameters of the original neural network units and the additional neural network units from the initialized values. ### Claim 38 (depends on 37) The computer storage medium of claim 37, further comprising: training the original neural network to determine the original values of the parameters of the original neural network. ### Claim 39 (depends on 38) The computer storage medium of claim 38, wherein the original neural network structure comprises a first original neural network layer having a first number of original units, and wherein generating the larger neural network comprises: adding a plurality of additional neural network units to the first original neural network layer to generate a larger neural network layer. ### Claim 40 (depends on 39) The computer storage medium of claim 39, wherein initializing values of the parameters of the original neural network units and the additional neural network units so that the larger neural network generates the same neural network outputs from the same neural network inputs as the original neural network comprises: initializing the values of the parameters of the original neural network units in the larger neural network layer to the respective original values for the parameters; and for each additional neural network unit in the larger neural network layer: selecting an original neural network unit in the original neural network layer, and initializing the values of the parameters of the additional neural network unit to be the same as the respective original values for the selected original neural network unit.