# US 2023/0274147 A1 · Processing Sensor Data with Multi-Model System on Resource-Constrained Device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0274147 A1 - Document ID: US20230274147A1 - Status: Published application - Kind: A1 - Application number: 18/313,072 - Filed: 2023-05-05 - Published: 2023-08-31 - Earliest claimed: 2020-11-17 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Chun-Te Chu; Claire Jaja; Kara Vaillancourt; Oleg Veryovka - Web page: https://patentgrep.com/patent/US20230274147A1 ## CPC - G06F 1/3212: Monitoring battery levels, e.g. power saving mode being initiated when battery voltage goes below a certain level - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06F 1/3287: by switching off individual functional units in the computer system - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/24133: {Distances to prototypes} - G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/09: Supervised learning - G06N 20/20: Ensemble learning - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and computer-readable media for multi-model processing on resource-constrained devices. A resource-constrained device can determine, based on a battery-life for a battery of the device, whether to process input through a first model or a second model. The first model can be a gating model that is more energy efficient to execute, and the second model can be a main model that is more accurate than the gating model. Depending on the current battery-life and/or other criteria, the system can process, through the gating model, sensor input that can record activity performed by a user of the resource-constrained device. If the gating model predicts an activity performed by the user that is recorded by the sensor data, the device can process the same or additional input through the main model. Overall power consumption can be reduced with a minimum accuracy maintained over processing input only through the main model. ## Claims ### Claim 1 A computing device comprising one or more processors configured to: receive an input; determine whether the device is operating in a resource-constrained mode; and determine, based on at least the determination of whether the device is operating in a resource-constrained mode, whether to process the input using a first machine learning model or a second machine learning model, the first machine learning model trained to generate, from the input, a first output with an average accuracy meeting a minimum accuracy threshold, and the second machine learning model trained to generate, from the input, a second output with an average accuracy exceeding the minimum accuracy threshold. ### Claim 2 (depends on 1) The computing device of claim 1, wherein the one or more processors are further configured to, when the first machine learning model predicts that a primary input recorded an activity, process a secondary input through the second machine learning model to generate the second output predicting whether the secondary input recorded the activity. ### Claim 3 (depends on 1) The computing device of claim 1, wherein the first machine learning model requires fewer operations to process the input than the second machine learning model requires to process the input. ### Claim 4 (depends on 1) The computing device of claim 1, wherein the determination of whether to use the first machine learning model or the second machine learning model is based at least in part on how each machine learning model affects a resource of the device. ### Claim 5 (depends on 2) The computing device of claim 2 further comprising one or more sensors, wherein: the primary input is obtained from a first sensor of the one or more sensors operating at a first power level; and the secondary input is at least partially obtained from the first sensor operating at a second power level higher than the first power level. ### Claim 6 (depends on 1) The computing device of claim 1, wherein the computing device further comprises a display, and wherein the one or more processors are further configured to determine, from the second output, that the second machine learning model predicted that the input recorded an activity, and in response generate and transmit a third output to the display. ### Claim 7 (depends on 1) The computing device of claim 1 further comprising one or more sensors, wherein: the first machine learning model and the second machine learning model are trained to perform a machine learning task, wherein the machine learning task comprises detecting a plurality of types of activities recorded by at least one sensor of the one or more sensors; and the second machine learning model is trained to detect at least one type of the plurality of types with an average accuracy greater than the first machine learning model. ### Claim 8 (depends on 1) The computing device of claim 1, wherein the computing device further comprises a battery, and the one or more processors are further configured to: receive a current battery-life of the battery indicating a change in available energy stored in the battery; and determine to process the input through the first machine learning model: at a first frequency in response to the current battery-life indicating a change in more available energy stored in the battery; and at a second frequency in response to the current battery-life indicating a change in less available energy stored in the battery, wherein the first frequency is greater than the second frequency. ### Claim 9 (depends on 1) The computing device of claim 1, wherein the computing device further comprises a battery, and the one or more processors are further configured to: receive one or more performance thresholds corresponding to an average measure of performance for the first machine learning model under respective one or more conditions and that are equal to or greater than a minimum average threshold; receive a current battery-life of the battery indicating a change in available energy stored in the battery; determine, based at least on the current battery-life, whether to cause the first machine learning model to perform at a level of performance meeting a performance threshold of the one or more performance thresholds; and in response to the determination to cause the first machine learning model to perform at the level of performance, cause the first machine learning model to operate under respective one or more conditions for the performance threshold. ### Claim 10 (depends on 1) The computing device of claim 1, wherein the first and the second machine learning models are neural networks, the second machine learning model having a first plurality of layers, and the first machine learning model having a second plurality of layers that is a subset of the first plurality of layers. ### Claim 11 (depends on 1) The computing device of claim 1, wherein the first machine learning model is of a different model architecture than the second machine learning model. ### Claim 12 A method comprising: determining whether a computing device is operating in a resource-constrained mode; and determining, based on at least the determination of whether the computing device is operating in a resource-constrained mode, whether to process an input through a first machine learning model or a second machine learning model, wherein: the first and second machine learning models are implemented by a computing device; the first machine learning model is trained to generate, from the input, a first output with an average accuracy meeting a minimum accuracy threshold; and the second machine learning model is trained to generate, from the input, a second output with an average accuracy exceeding the minimum accuracy threshold. ### Claim 13 (depends on 12) The method of claim 12, further comprising: processing a secondary input through the second machine learning model when the first machine learning model predicts that a primary input recorded an activity; and generating the second output predicting whether the secondary input recorded the activity. ### Claim 14 (depends on 12) The method of claim 12, wherein the first machine learning model requires fewer operations to process the input than the second machine learning model requires to process the input. ### Claim 15 (depends on 12) The method of claim 12, wherein the determination of whether to use the first machine learning model or the second machine learning model is based at least in part on how each machine learning model affects a resource of a computing device. ### Claim 16 (depends on 13) The method of claim 13, wherein: obtaining the primary input comprises obtaining sensor input from a first sensor of one or more sensors operating at a first power level; and the secondary input is at least partially obtained by the first sensor operating at a second power level higher than the first power level. ### Claim 17 (depends on 12) The method of claim 12, wherein the method further comprises determining, from the second output, that the second machine learning model predicted that the input recorded an activity, and in response generating and transmitting a third output to a display of a computing device. ### Claim 18 (depends on 12) The method of claim 12, wherein determining to process the input through the first machine learning model comprises: receiving a current battery-life of a battery of a computing device, the current battery-life indicating a change in available energy stored in the battery; and determining to process the input through the first machine learning model: at a first frequency in response to the current battery-life indicating a change in more available energy stored in the battery; and at a second frequency in response to the current battery-life indicating a change in less available energy stored in the battery, wherein the first frequency is greater than the second frequency. ### Claim 19 (depends on 12) The method of claim 12, wherein determining to process the input through the first machine learning model comprises: receiving one or more performance thresholds corresponding to an average measure of performance for the first machine learning model under respective one or more conditions and that are equal to or greater than a minimum average threshold; receiving a current battery-life of a battery of a computing device, the current battery-life indicating a change in available energy stored in the battery; determining, based at least on the current battery-life, whether to cause the first machine learning model to perform at a level of performance meeting a performance threshold of the one or more performance thresholds; and in response to determining to cause the first machine learning model to perform at the level of performance, causing the first machine learning model to operate under respective one or more conditions for the performance threshold. ### Claim 20 One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing device, causes the one or more processors to perform operations comprising: determining whether the computing device is operating in a resource-constrained mode; and determining, based on at least the determination of whether the computing device is operating in a resource-constrained mode, whether to process an input through a first machine learning model or a second machine learning model, wherein: the first and second machine learning models are implemented by the computing device; first machine learning model is trained to generate a first output with an average accuracy meeting a minimum accuracy threshold; and the second machine learning model is trained to generate a second output with an average accuracy exceeding the minimum accuracy threshold.