# US 10,504,020 B2 · System and method for applying a deep learning neural network to data obtained from one or more sensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,020 B2 - Document ID: US10504020B2 - Status: Granted - Kind: B2 - Application number: 14/736,189 - Filed: 2015-06-10 - Published: 2016-02-04 - Granted: 2019-12-10 - Earliest claimed: 2014-06-10 - Assignees: Sightline Innovation Inc. - Applicants: Sightline Innovation Inc. - Inventors: Wallace Trenholm; Maithili Mavinkurve; Mark Alexiuk; Jason Cassidy - Web page: https://patentgrep.com/patent/US10504020B2 ## CPC - G06F 8/30: Creation or generation of source code - G06F 8/35: model driven - G06F 8/36: Software reuse - G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system} - G06N 3/02: Neural networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/091: Active learning - H04L 41/5041: characterised by the time relationship between creation and deployment of a service - H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks - H04L 67/34: {involving the movement of software or configuration parameters (network booting or remote initial program loading [RIPL] G06F9/4416)} - G06F 2209/5017: Task decomposition - G06F 2209/503: Resource availability - G06F 2209/509: Offload - G06F 2209/541: Client-server - G06F 2209/549: Remote execution - G06F 2218/12: Classification; Matching - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract An application provisioning system and method. A server provides an application provisioning service. A user of a client provides a schema defining an application. The application interacts with peripherals coupled to the client and receives input from sensors coupled to the peripherals. The sensor data is provided to the server for processing, including by neural networks. The application includes a workflow defining a finite state machine that traverses states at least partially based on the response to sensor data. The server may provide dynamic reallocation of compute resources to resolve demand for classifier training job requests; use of jurisdictional certificates to define data usage and sharing; and data fusion. Applications include manufacturing verification, medical diagnosis and treatment, genomics and viral detection. ## Claims ### Claim 1 A system for applying a deep learning neural network to data obtained from one or more sensors on a client computing system, the one or more sensors sensing components of an industrial scene, the system comprising: a network accessible server system linked to the client computing system, the server system configured to provision and send an application to be executable on the client computing system, the application allocating computing tasks to the server system, the server system provisioning the application based on a schema definition comprising metadata received from the client computing system such that the application is customized for the client computing system and causes interaction with the sensors particular to the client computing system; and one or more peripheral devices, at least one of the peripheral devices comprising one or more of the sensors, each peripheral device associated with the scene, each peripheral device connected to the client computing system, the client computing system transmitting the sensor data to the server system, the server system building a one or more trained classification models to extract one or more features using the sensor data sensed by the sensors connected to the client computing system, the one or more classification models applying a neural network based scene parsing and scene matching to a portion of the sensor data transmitted from the client computing system to parse and match the one or more extracted features as feature vectors, training of the one or more classification models considering only sensor data from the one or more sensors connected to the client computing system, the client computing system parsing and matching one or more features from other portions of the sensor data transmitted from the client computing system based on feature vectors computed using the one or more trained classification models received from the server system, the matching using previously known components, the client computing system controlling positioning, motion, or obtainment of sensor information of at least one of the peripheral devices based on the output of the matching from the one or more trained classification models. ### Claim 2 (depends on 1) The system of claim 1, wherein the sensors comprise image capture devices. ### Claim 3 (depends on 1) The system of claim 1, wherein the sensor data is transmitted by the client computing system to the server system through interfaces that provide anonymization, encryption, compression and jurisdictional stamping. ### Claim 4 (depends on 1) The system of claim 1, wherein the sensor data comprises a plurality of data points aggregated from a plurality of sensors. ### Claim 5 (depends on 1) The system of claim 1, wherein results of the classification process are transmitted to a ground truther to evaluate model accuracy to implement an active learning process. ### Claim 6 (depends on 5) The system of claim 5, wherein the system further provides a confidence score to the ground truther. ### Claim 7 (depends on 1) The system of claim 1, wherein the server system applies a plurality of neural networks simultaneously and provides a list of classifiers. ### Claim 8 (depends on 7) The system of claim 7, wherein the classifiers in the list are provided with confidence values. ### Claim 9 A method of applying a deep learning neural network to data obtained from one or more peripheral devices connected to a client computing system, at least one of the peripherals devices comprising one or more sensors, the one or more sensors sensing components of an industrial scene, each peripheral device associated with the scene, the method comprising: linking a network accessible server system to the client computing system; configuring the server system to provision and send an application to be executable on the client computing system, the application allocating computing tasks to the server system, the server system provisioning the application based on a schema definition comprising metadata received from the client computing system such that the application is customized for the client computing system and causes interaction with the sensors particular to the client computing system; receiving sensor data from the one or more peripheral devices connected to the client computing system at the server system; building one or more trained classification models at the server system to extract one or more features using the sensor data sensed by the sensors, the one or more classification models applying a neural network based scene parsing and scene matching to a portion of the sensor data transmitted from the client computing system to parse and match the one or more extracted features as feature vectors, training of the one or more classification models considering only the sensor data from the one or more sensors connected to the client computing system; transmitting the one or more trained classification models to the client computing system to be usable at the client computing system to parse and match one or more features from other portions of the sensor data based on feature vectors computed using the one or more trained classification models, the matching using previously known components; and outputting the matching from the one or more trained classification models to be used by the client computing system to control positioning, motion, or obtainment of sensor information of at least one of the peripheral devices based on the output of the matching. ### Claim 10 (depends on 9) The method of claim 9, wherein the sensors comprise image capture devices. ### Claim 11 (depends on 9) The method of claim 9, wherein the sensor data is transmitted by the client computing system to the server system through interfaces that provide anonymization, encryption, compression and jurisdictional stamping. ### Claim 12 (depends on 9) The method of claim 9, wherein the sensor data comprises a plurality of data points aggregated from a plurality of sensors. ### Claim 13 (depends on 9) The method of claim 9, wherein results of the classification process are transmitted to a ground truther to evaluate model accuracy to implement an active learning process. ### Claim 14 (depends on 13) The method of claim 13, further comprising providing a confidence score to the ground truther. ### Claim 15 (depends on 13) The method of claim 13, wherein the server system applies a plurality of neural networks simultaneously and provides a list of classifiers. ### Claim 16 (depends on 15) The method of claim 15, wherein the classifiers in the list are provided with confidence values.