# US 11,443,182 B2 · Privacy enhancing deep learning cloud service using a trusted execution environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,182 B2 - Document ID: US11443182B2 - Status: Granted - Kind: B2 - Application number: 16/016,752 - Filed: 2018-06-25 - Published: 2019-12-26 - Granted: 2022-09-13 - Earliest claimed: 2018-06-25 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Zhongshu Gu; Heqing Huang; Jialong Zhang; Dong Su; Dimitrios Pendarakis; Ian M. Molloy - Web page: https://patentgrep.com/patent/US11443182B2 ## CPC - G06N 3/08: Learning methods - G06F 21/53: by executing in a restricted environment, e.g. sandbox or secure virtual machine - G06F 21/602: {Providing cryptographic facilities or services} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/063: using electronic means - G06N 3/09: Supervised learning - G06F 2221/034: Test or assess a computer or a system - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Mechanisms are provided to implement an enhanced privacy deep learning system framework (hereafter “framework”). The framework receives, from a client computing device, an encrypted first subnet model of a neural network, where the first subnet model is one partition of multiple partitions of the neural network. The framework loads the encrypted first subnet model into a trusted execution environment (TEE) of the framework, decrypts the first subnet model, within the TEE, and executes the first subnet model within the TEE. The framework receives encrypted input data from the client computing device, loads the encrypted input data into the TEE, decrypts the input data, and processes the input data in the TEE using the first subnet model executing within the TEE. ## Claims ### Claim 1 A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions that are executed by the at least one processor to configure the at least one processor to implement an enhanced privacy deep learning system framework, the method comprising: receiving, by the enhanced privacy deep learning system framework, from a client computing device, an encrypted first subnet model of a neural network, wherein the first subnet model is one partition of multiple partitions of the neural network; loading, by the enhanced privacy deep learning system framework, the encrypted first subnet model into a trusted execution environment of the enhanced privacy deep learning system framework; decrypting, by the enhanced privacy deep learning system framework, the first subnet model within the trusted execution environment and executing the first subnet model within the trusted execution environment; receiving, by the enhanced privacy deep learning system framework, encrypted input data from the client computing device; loading, by the enhanced privacy deep learning system framework, the encrypted input data into the trusted execution environment; and decrypting and processing, by the enhanced privacy deep learning system framework, the input data in the trusted execution environment using the first subnet model executing within the trusted execution environment, wherein the first subnet model is a FrontNet subnet model comprising an input layer of the neural network and one or more intermediate layers of the neural network model, and wherein the neural network comprises a second subnet model that is a BackNet subnet model comprising an output layer of the neural network that outputs result data, and one or more intermediate layers of the neural network model; and outputting the result data to a deep learning system to perform a classification operation to classify the encrypted input data into one of a plurality of predefined classes. ### Claim 2 (depends on 1) The method of claim 1, wherein a partition point in the neural network indicating a last intermediate layer to be included in the FrontNet subnet model is selected as an intermediate layer whose intermediate representation output does not contain sensitive information corresponding to an input to the neural network, and wherein subsequent intermediate layers and the output layer of the neural network are included in the BackNet subnet model. ### Claim 3 (depends on 1) The method of claim 1, wherein the neural network is partitioned automatically using an automated partitioning tool that identifies an optimal partition point in the neural network at which to partition the neural network, wherein the optimal partition point identifies an intermediate layer at which to partition the neural network into the first subnet model and the second subnet model. ### Claim 4 (depends on 1) The method of claim 1, wherein the processing of the input data in the trusted execution environment using the first subnet model executing within the trusted execution environment generates one or more intermediate representations of processing of the input data, and wherein the method further comprises: inputting the one or more intermediate representations into the second subnet model of the neural network; processing, by the second subnet model, the one or more intermediate representations to generate result data; and outputting the result data. ### Claim 5 (depends on 1) The method of claim 1, wherein the second subnet model executes outside the trusted execution environment. ### Claim 6 (depends on 4) The method of claim 4, wherein the result data is a N-dimensional real-value vector that represents a probability distribution over N different possible classes, and wherein the method further comprises selecting a top-k classes with corresponding probability values from the N-dimensional real-value vector, to return to the client computing device. ### Claim 7 (depends on 1) The method of claim 1, wherein the input data is an input image and the classification operation classifies the input image into one of a plurality of predefined classes. ### Claim 8 (depends on 1) The method of claim 1, wherein the trusted execution environment prevents access to the decrypted first subnet model and decrypted input data from outside the trusted execution environment. ### Claim 9 A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement an enhanced privacy deep learning system framework that is configured to: receive from a client computing device, an encrypted first subnet model of a neural network, wherein the first subnet model is one partition of multiple partitions of the neural network; load the encrypted first subnet model into a trusted execution environment of the enhanced privacy deep learning system framework; decrypt the first subnet model within the trusted execution environment and execute the first subnet model within the trusted execution environment; receive encrypted input data from the client computing device; load the encrypted input data into the trusted execution environment; and decrypt and process the input data in the trusted execution environment using the first subnet model executing within the trusted execution environment, wherein the first subnet model is a FrontNet subnet model comprising an input layer of the neural network and one or more intermediate layers of the neural network model, and wherein the neural network comprises a second subnet model that is a BackNet subnet model comprising an output layer of the neural network that outputs result data, and one or more intermediate layers of the neural network model; and output the result data to a deep learning system to perform a classification operation to classify the encrypted input data into one of a plurality of predefined classes. ### Claim 10 (depends on 9) The computer program product of claim 9, wherein a partition point in the neural network indicating a last intermediate layer to be included in the FrontNet subnet model is selected as an intermediate layer whose intermediate representation output does not contain sensitive information corresponding to an input to the neural network, and wherein subsequent intermediate layers and the output layer of the neural network are included in the BackNet subnet model. ### Claim 11 (depends on 9) The computer program product of claim 9, wherein the neural network is partitioned automatically using an automated partitioning tool that identifies an optimal partition point in the neural network at which to partition the neural network, wherein the optimal partition point identifies an intermediate layer at which to partition the neural network into the first subnet model and the second subnet model. ### Claim 12 (depends on 9) The computer program product of claim 9, wherein the processing of the input data in the trusted execution environment using the first subnet model executing within the trusted execution environment generates one or more intermediate representations of processing of the input data, and wherein the method further comprises: inputting the one or more intermediate representations into the second subnet model of the neural network; processing, by the second subnet model, the one or more intermediate representations to generate result data; and outputting the result data. ### Claim 13 (depends on 9) The computer program product of claim 9, wherein the second subnet model executes outside the trusted execution environment. ### Claim 14 (depends on 12) The computer program product of claim 12, wherein the result data is a N-dimensional real-value vector that represents a probability distribution over N different possible classes, and wherein the method further comprises selecting a top-k classes with corresponding probability values from the N-dimensional real-value vector, to return to the client computing device. ### Claim 15 (depends on 9) The computer program product of claim 9, wherein the input data is an input image and the classification operation classifies the input image into one of a plurality of predefined classes. ### Claim 16 A system, comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to implement an enhanced privacy deep learning system framework that is configured to: receive from a client computing device, an encrypted first subnet model of a neural network, wherein the first subnet model is one partition of multiple partitions of the neural network; load the encrypted first subnet model into a trusted execution environment of the enhanced privacy deep learning system framework; decrypt the first subnet model within the trusted execution environment and execute the first subnet model within the trusted execution environment; receive encrypted input data from the client computing device; load the encrypted input data into the trusted execution environment; and decrypt and process the input data in the trusted execution environment using the first subnet model executing within the trusted execution environment, wherein the first subnet model is a FrontNet subnet model comprising an input layer of the neural network and one or more intermediate layers of the neural network model, and wherein the neural network comprises a second subnet model that is a BackNet subnet model comprising an output layer of the neural network that outputs result data, and one or more intermediate layers of the neural network model; and output the result data to a deep learning system to perform a classification operation to classify the encrypted input data into one of a plurality of predefined classes. ### Claim 17 (depends on 16) The system of claim 16, wherein a partition point in the neural network indicating a last intermediate layer to be included in the FrontNet subnet model is selected as an intermediate layer whose intermediate representation output does not contain sensitive information corresponding to an input to the neural network, and wherein subsequent intermediate layers and the output layer of the neural network are included in the BackNet subnet model. ### Claim 18 (depends on 16) The system of claim 16, wherein the neural network is partitioned automatically using an automated partitioning tool that identifies an optimal partition point in the neural network at which to partition the neural network, wherein the optimal partition point identifies an intermediate layer at which to partition the neural network into the first subnet model and the second subnet model. ### Claim 19 (depends on 16) The system of claim 16, wherein the second subnet model executes outside the trusted execution environment. ### Claim 20 (depends on 16) The system of claim 16, wherein the input data is an input image and the classification operation classifies the input image into one of a plurality of predefined classes.