# US 2019/0377985 A1 · Image classification neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0377985 A1 - Document ID: US20190377985A1 - Status: Published application - Kind: A1 - Application number: 16/550,731 - Filed: 2019-08-26 - Published: 2019-12-12 - Earliest claimed: 2016-02-18 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Vincent O. Vanhoucke; Christian Szegedy; Sergey Ioffe - Web page: https://patentgrep.com/patent/US20190377985A1 ## CPC - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06F 18/24: Classification techniques - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/048: Activation functions - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A neural network system that includes: multiple subnetworks that includes: a first subnetwork including multiple first modules, each first module including: a pass-through convolutional layer configured to process the subnetwork input for the first subnetwork to generate a pass-through output; an average pooling stack of neural network layers that collectively processes the subnetwork input for the first subnetwork to generate an average pooling output; a first stack of convolutional neural network layers configured to collectively process the subnetwork input for the first subnetwork to generate a first stack output; a second stack of convolutional neural network layers that are configured to collectively process the subnetwork input for the first subnetwork to generate a second stack output; and a concatenation layer configured to concatenate the pass-through output, the average pooling output, the first stack output, and the second stack output to generate a first module output for the first module. ## Claims ### Claim at position 1 -16. (canceled) ### Claim 17 A neural network system implemented by one or more computers, wherein the neural network system is configured to receive an image and to generate a classification output for the input image, and wherein the neural network system comprises: a plurality of subnetworks arranged in a stack on top of each other, wherein each subnetwork is configured to process a subnetwork input to generate a subnetwork output and to provide the subnetwork output as input to another subnetwork above the subnetwork in the stack, and wherein the plurality of subnetworks includes: a first subnetwork comprising a plurality of first modules, each first module comprising: a first pass-through convolutional layer configured to process the subnetwork input for the first subnetwork to generate a first pass-through output; a first average pooling stack of neural network layers, wherein the layers in the first average pooling stack are configured to collectively process the subnetwork input for the first subnetwork to generate a first average pooling output; a first stack of convolutional neural network layers, wherein the layers in the first stack are configured to collectively process the subnetwork input for the first subnetwork to generate a first stack output; a second stack of convolutional neural network layers, wherein the second stack comprises a 1×1 convolutional layer immediately followed by a 3×3 convolutional layer immediately followed by a 3×3 convolutional layer, and wherein the layers in the second stack are configured to collectively process the subnetwork input for the first subnetwork to generate a second stack output; and a first concatenation layer configured to concatenate the first pass-through output, the first average pooling output, the first stack output, and the second stack output to generate a first module output for the first module; and a third subnetwork comprising a plurality of third modules, each third module comprising: a third pass-through convolutional layer configured to process the subnetwork input for the third subnetwork to generate a third pass-through output; an third average pooling stack of neural network layers, wherein the layers in the third average pooling stack are configured to collectively process the subnetwork input for the third subnetwork to generate an third average pooling output; a first group of convolutional neural network layers, wherein the first group including a 1×1 convolutional layer, a 1×3 convolutional layer, and a 3×1 convolutional layer, wherein the layers in the first group are configured to collectively process the subnetwork input for the third subnetwork to generate a first group output; a second group of convolutional neural network layers, wherein the layers in the second group are configured to collectively process the subnetwork input for the third subnetwork to generate a second group output; and a third concatenation layer configured to concatenate the pass-through output, the average pooling output, the first group output, and the second group output to generate a third module output for the third module. ### Claim 18 (depends on 17) The neural network system of claim 17, wherein the first subnetwork includes four first modules. ### Claim 19 (depends on 17) The neural network system of claim 17, wherein the third subnetwork includes three third modules. ### Claim 20 (depends on 17) The neural network system of claim 17, wherein at least one of the first pass-through convolutional layer or the third pass-through convolutional layer is a 1×1 convolutional layer. ### Claim 21 (depends on 17) The neural network system of claim 17, wherein the first average pooling stack includes an average pooling layer followed by a 1×1 convolutional layer. ### Claim 22 (depends on 17) The neural network system of claim 17, wherein the first stack includes a 1×1 convolutional layer followed by a 3×3 convolutional layer. ### Claim 23 (depends on 17) The neural network system of claim 17, wherein the first subnetwork is configured to combine the first module outputs generated by the plurality of first subnetworks to generate a first subnetwork output for the first subnetwork. ### Claim 24 (depends on 17) The neural network system of claim 17, wherein the first subnetwork receives an input that is 35×35×384 and each first module generates an output that is 35×35×384. ### Claim 25 (depends on 17) The neural network system of claim 17, wherein the third average pooling stack includes an average pooling layer followed by a 1×1 convolutional layer. ### Claim 26 (depends on 17) The neural network system of claim 17, wherein: the 1×1 convolutional layer in the first group is configured to process the subnetwork input for the third subnetwork to generate a first intermediate output; the 1×3 convolutional layer in the first group is configured to process the first intermediate output to generate a second intermediate output; the 3×1 convolutional layer in the first group is configured to process the first intermediate output to generate a third intermediate output; and wherein the first group further includes a first group concatenation layer configured to concatenate the second intermediate output and the third intermediate output to generate the first group output. ### Claim 27 (depends on 17) The neural network system of claim 17, wherein the second group includes: a fifth stack of convolutional layers that is configured to process the subnetwork input for the third subnetwork to generate a fifth stack output; a 1×3 convolutional layer configured to process the fifth stack output to generate a fourth intermediate output; a 3×1 convolutional layer configured to process the fifth stack output to generate a fifth intermediate output; and a second group concatenation layer configured to concatenate the fourth intermediate output and the fifth intermediate output to generate the second group output. ### Claim 28 (depends on 22) The neural network system of claim 22, wherein the fifth stack includes a 1×1 convolutional layer followed by a 1×3 convolutional layer followed by a 3×1 convolutional layer. ### Claim 29 (depends on 17) The neural network system of claim 17, wherein the third subnetwork is configured to combine the third module outputs generated by the plurality of third modules to generate a third subnetwork output for the third subnetwork. ### Claim 30 (depends on 17) The neural network system of claim 17, wherein the third subnetwork receives an input that is 8×8×1536 and each third module generates an output that is 8×8×1536. ### Claim 31 One or more non-transitory storage media encoded with instructions that when executed by one or more computers cause the one or more computers to implement a neural network system that is configured to receive an image and to generate a classification output for the input image, wherein the neural network system comprises: a plurality of subnetworks arranged in a stack on top of each other, wherein each subnetwork is configured to process a subnetwork input to generate a subnetwork output and to provide the subnetwork output as input to another subnetwork above the subnetwork in the stack, and wherein the plurality of subnetworks includes: a first subnetwork comprising a plurality of first modules, each first module comprising: a first pass-through convolutional layer configured to process the subnetwork input for the first subnetwork to generate a first pass-through output; a first average pooling stack of neural network layers, wherein the layers in the first average pooling stack are configured to collectively process the subnetwork input for the first subnetwork to generate a first average pooling output; a first stack of convolutional neural network layers, wherein the layers in the first stack are configured to collectively process the subnetwork input for the first subnetwork to generate a first stack output; a second stack of convolutional neural network layers, wherein the second stack comprises a 1×1 convolutional layer immediately followed by a 3×3 convolutional layer immediately followed by a 3×3 convolutional layer, and wherein the layers in the second stack are configured to collectively process the subnetwork input for the first subnetwork to generate a second stack output; and a first concatenation layer configured to concatenate the first pass-through output, the first average pooling output, the first stack output, and the second stack output to generate a first module output for the first module; and a third subnetwork comprising a plurality of third modules, each third module comprising: a third pass-through convolutional layer configured to process the subnetwork input for the third subnetwork to generate a third pass-through output; an third average pooling stack of neural network layers, wherein the layers in the third average pooling stack are configured to collectively process the subnetwork input for the third subnetwork to generate an third average pooling output; a first group of convolutional neural network layers, wherein the first group including a 1×1 convolutional layer, a 1×3 convolutional layer, and a 3×1 convolutional layer, wherein the layers in the first group are configured to collectively process the subnetwork input for the third subnetwork to generate a first group output; a second group of convolutional neural network layers, wherein the layers in the second group are configured to collectively process the subnetwork input for the third subnetwork to generate a second group output; and a third concatenation layer configured to concatenate the pass-through output, the average pooling output, the first group output, and the second group output to generate a third module output for the third module. ### Claim 32 (depends on 31) The one or more non-transitory storage media of claim 31, wherein the first subnetwork includes four first modules and the third subnetwork includes three first modules. ### Claim 33 (depends on 31) The one or more non-transitory storage media of claim 31, wherein at least one of the first pass-through convolutional layer or the third pass-through convolutional layer is a 1×1 convolutional layer. ### Claim 34 (depends on 31) The one or more non-transitory storage media of claim 31, wherein the second group includes: a fifth stack of convolutional layers that is configured to process the subnetwork input for the third subnetwork to generate a fifth stack output; a 1×3 convolutional layer configured to process the fifth stack output to generate a fourth intermediate output; a 3×1 convolutional layer configured to process the fifth stack output to generate a fifth intermediate output; and a second group concatenation layer configured to concatenate the fourth intermediate output and the fifth intermediate output to generate the second group output. ### Claim 35 (depends on 31) The one or more non-transitory storage media of claim 31, wherein the fifth stack includes a 1×1 convolutional layer followed by a 1×3 convolutional layer followed by a 3×1 convolutional layer. ### Claim 36 A computer-implemented method comprising: processing an input image using a neural network system to generate a classification output for the input image, wherein the neural network system comprises a plurality of subnetworks arranged in a stack on top of each other, wherein each subnetwork is configured to process a subnetwork input to generate a subnetwork output and to provide the subnetwork output as input to another subnetwork above the subnetwork in the stack, and wherein the plurality of subnetworks includes: a first subnetwork comprising a plurality of first modules, each first module comprising: a first pass-through convolutional layer configured to process the subnetwork input for the first subnetwork to generate a first pass-through output; a first average pooling stack of neural network layers, wherein the layers in the first average pooling stack are configured to collectively process the subnetwork input for the first subnetwork to generate a first average pooling output; a first stack of convolutional neural network layers, wherein the layers in the first stack are configured to collectively process the subnetwork input for the first subnetwork to generate a first stack output; a second stack of convolutional neural network layers, wherein the second stack comprises a 1×1 convolutional layer immediately followed by a 3×3 convolutional layer immediately followed by a 3×3 convolutional layer, and wherein the layers in the second stack are configured to collectively process the subnetwork input for the first subnetwork to generate a second stack output; and a first concatenation layer configured to concatenate the first pass-through output, the first average pooling output, the first stack output, and the second stack output to generate a first module output for the first module; and a third subnetwork comprising a plurality of third modules, each third module comprising: a third pass-through convolutional layer configured to process the subnetwork input for the third subnetwork to generate a third pass-through output; an third average pooling stack of neural network layers, wherein the layers in the third average pooling stack are configured to collectively process the subnetwork input for the third subnetwork to generate an third average pooling output; a first group of convolutional neural network layers, wherein the first group including a 1×1 convolutional layer, a 1×3 convolutional layer, and a 3×1 convolutional layer, wherein the layers in the first group are configured to collectively process the subnetwork input for the third subnetwork to generate a first group output; a second group of convolutional neural network layers, wherein the layers in the second group are configured to collectively process the subnetwork input for the third subnetwork to generate a second group output; and a third concatenation layer configured to concatenate the pass-through output, the average pooling output, the first group output, and the second group output to generate a third module output for the third module.