# US 2021/0241111 A1 · Shaping a neural network architecture utilizing learnable sampling layers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241111 A1 - Document ID: US20210241111A1 - Status: Published application - Kind: A1 - Application number: 16/782,793 - Filed: 2020-02-05 - Published: 2021-08-05 - Earliest claimed: 2020-02-05 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Shikun Liu; Zhe Lin; Yilin Wang; Jianming Zhang; Federico Perazzi - Web page: https://patentgrep.com/patent/US20210241111A1 ## CPC - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The present disclosure relates to shaping the architecture of a neural network. For example, the disclosed systems can provide a neural network shaping mechanism for at least one sampling layer of a neural network. The neural network shaping mechanism can include a learnable scaling factor between a sampling rate of the at least one sampling layer and an additional sampling function. The disclosed systems can learn the scaling factor based on a dataset while jointly learning the network weights of the neural network. Based on the learned scaling factor, the disclosed systems can shape the architecture of the neural network by modifying the sampling rate of the at least one sampling layer. ## Claims ### Claim 1 A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to: initialize a neural network comprising a plurality of sampling layers and a plurality of network weights; provide a neural network shaping mechanism for at least one sampling layer of the neural network, the neural network shaping mechanism comprising a learnable scaling factor between a sampling rate of the at least one sampling layer and an additional sampling function based on input to the at least one sampling layer; jointly learn the scaling factor and the plurality of network weights based on a dataset; and shape an architecture of the neural network by modifying the sampling rate of the at least one sampling layer of the neural network based on the learned scaling factor. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the additional sampling function comprises an identity function. ### Claim 3 (depends on 1) The non-transitory computer-readable medium of claim 1, further comprising instructions that, when executed by the at least one processor, cause the computing device to: generate a first feature map comprising a first size using the sampling rate of the at least one sampling layer based on the input to the at least one sampling layer, wherein the input to the at least one sampling layer corresponds to the dataset; generate a second feature map comprising a second size using the additional sampling function based on the input to the at least one sampling layer, wherein the second size is different from the first size, and perform linear interpolation between the first feature map and the second feature map based on the scaling factor. ### Claim 4 (depends on 3) The non-transitory computer-readable medium of claim 3, further comprising instructions that, when executed by the at least one processor, cause the computing device to determine, using an interpolation layer of the neural network shaping mechanism, a third feature map comprising a third size based on the linear interpolation between the first feature map and the second feature map. ### Claim 5 (depends on 4) The non-transitory computer-readable medium of claim 4, wherein the third size of the third feature map is different from the first size of the first feature map and the second size of the second feature map. ### Claim 6 (depends on 4) The non-transitory computer-readable medium of claim 4, further comprising instructions that, when executed by the at least one processor, cause the computing device to provide, within the interpolation layer of the neural network shaping mechanism, a scaling function comprising the scaling factor. ### Claim 7 (depends on 6) The non-transitory computer-readable medium of claim 6, wherein the scaling function comprises a continuous monotonic function. ### Claim 8 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein modifying the sampling rate of the at least one sampling layer of the neural network based on the learned scaling factor comprises combining parameters associated with the at least one sampling layer and parameters associated with the additional sampling function via element-wise addition based on the learned scaling factor. ### Claim 9 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the instructions, when executed by the at least one processor, cause the computing device to provide the neural network shaping mechanism for the at least one sampling layer of the neural network by providing a plurality of neural network shaping mechanisms for the plurality of sampling layers of the neural network, the plurality of neural network shaping mechanisms comprising learnable scaling factors between sampling rates of the plurality of sampling layers and additional sampling functions based on input to the plurality of sampling layers. ### Claim 10 A system comprising: at least one memory device comprising a neural network comprising a plurality of sampling layers and a plurality of network weights; and at least one server device configured to cause the system to: provide a neural network shaping mechanism for at least one sampling layer of the neural network, the neural network shaping mechanism comprising a learnable scaling factor between a sampling rate of the at least one sampling layer and an additional sampling function; generate output feature maps based on input to the at least one sampling layer using the sampling rate of the at least one sampling layer and the additional sampling function; learn the scaling factor based on an interpolation between the output feature maps while jointly learning the plurality of network weights; and modify the sampling rate of the at least one sampling layer based on the learned scaling factor. ### Claim 11 (depends on 10) The system of claim 10, wherein the at least one server device is further to cause the system to learn the scaling factor while jointly learning the plurality of network weights based on the output feature maps by iteratively modifying the scaling factor and the plurality of network weights based on a loss function. ### Claim 12 (depends on 10) The system of claim 10, wherein the at least one server device is configured to cause the system to: determine a first sampling factor for the at least one sampling layer and a second sampling factor for the additional sampling function; and generate the output feature maps further based on the first sampling factor and the second sampling factor. ### Claim 13 (depends on 12) The system of claim 12, wherein the at least one server device is configured to cause the system to determine that the at least one sampling layer comprises either an up-sampling layer or a down-sampling layer based on the first sampling factor. ### Claim 14 (depends on 12) The system of claim 12, wherein the at least one server device is configured to cause the system to determine that the additional sampling function comprises an identity function based on the second sampling factor. ### Claim 15 (depends on 10) The system of claim 10, wherein the at least one server device is configured to cause the system to: generate, using an interpolation layer of the neural network shaping mechanism, interpolated feature maps based on the output feature maps; and learn the scaling factor and the plurality of network weights based on the output feature maps by learning the scaling factor and the plurality of network weights based on the interpolated feature maps. ### Claim 16 (depends on 15) The system of claim 15, wherein the at least one server device is configured to cause the system to provide, within the interpolation layer of the neural network shaping mechanism, a scaling function comprising the scaling factor. ### Claim 17 A computer-implemented method for morphing neural network architectures comprising: initializing a neural network comprising a plurality of sampling layers and a plurality of network weights; providing one or more neural network shaping mechanisms between sampling layers of the neural network; performing a step for learning scaling factors for the one or more neural network shaping mechanisms; and shaping an architecture of the neural network based on the learned scaling factors. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, wherein each neural network shaping mechanism from the one or more neural network shaping mechanisms comprise a learnable scaling factor between a sampling rate of a sampling layer and an additional sampling function based on input to the sampling layer. ### Claim 19 (depends on 17) The computer-implemented method of claim 17, wherein at least one sampling layer from the sampling layers of the neural network comprises an up-sampling layer. ### Claim 20 (depends on 17) The computer-implemented method of claim 17, wherein at least one sampling layer from the sampling layers of the neural network comprises a pooling layer.