US 12,711,033 B2Granted

Evaluating new feature(s) for client device(s) based on performance measure(s)

AssigneeGoogle Inc.

InventorsDragan Zivkovic, Harry Bleyan, Tamar Lucassen, Akash Agrawal, Françoise Beaufays

Life of this patent

Estimating the term…

Loading patent…

Abstract

Implementations disclosed herein are directed to systems and methods for evaluating new feature(s) for client device(s) based on performance measure(s) of the client device(s) and/or the new feature(s). The new feature(s) can include, for example, machine learning (ML) model(s), non-ML software-enabled functionality, non-ML hardware-enabled functionality, and/or ML or non-ML software application features for a given software application utilized by the client device(s). The client device(s) can generate the performance measure(s) by processing a plurality of testing instances for the new feature(s). The performance measure(s) can include, for example, latency measure(s), memory consumption measure(s), CPU usage measure(s), precision and/or recall measure(s), and/or other measures. In some implementations, the new feature(s) may be activated for use locally at the client device(s) based on the performance measure(s), and optionally at other client device(s) that share the same device characteristics. In other implementations, the new feature(s) may be modified based on the performance measure(s).

Claims18

Dependent claims

A method implemented by one or more processors, the method comprising:

identifying a new client device feature to be rolled out to a plurality of client devices;
selecting, from among the plurality of client devices, a subset of the plurality of client devices to be utilized in testing the new client device feature, wherein selecting the subset of the plurality of client devices to be utilized in testing the new client device feature comprises:
identifying, for each of the plurality of client devices, a corresponding set of device characteristics, the corresponding sets of device characteristics including at least a first set of device characteristics and a second set of device characteristics;
and
selecting, for inclusion in the subset, at least the given client device and an additional client device based on the given client device and the additional client device both having the first set of device characteristics;
causing a given client device, from among the subset of the plurality of client devices, to generate a plurality of performance measures for the new client device feature based on one or more testing instances,
wherein causing the given client device to generate the plurality of performance measures based on a given testing instance, of the one or more testing instances, comprises:
causing the given client device to process testing instance input for the given testing instance to generate output;
and
causing the given client device to generate the plurality of performance measures based on the processing of the testing instance input of the given testing instance and/or based on the output;
determining, based on the plurality of performance measures, whether to activate the new client device feature at the given client device;
and
in response to determining to activate the new client device feature at the given client device:
causing the new client device feature to be activated for use locally at the given client device.

11 more elements

A method implemented by one or more processors, the method comprising:

identifying a new software application feature, for a given software application, to be rolled out to a plurality of client devices that utilize the given software application;
selecting, from among the plurality of client devices, a subset of the plurality of client devices to be utilized in testing the new software application feature, wherein selecting the subset of the plurality of client devices to be utilized in testing the new software application feature comprises:
identifying, for each of the plurality of client devices, a corresponding set of device characteristics, the corresponding sets of device characteristics including at least a first set of device characteristics and a second set of device characteristics;
and
selecting, for inclusion in the subset, at least the given client device and an additional client device based on the given client device and the additional client device both having the first set of device characteristics;
causing a given client device, from among the subset of the plurality of client devices, to generate a plurality of performance measures for the new software application feature based on one or more testing instances,
wherein causing the given client device to generate the plurality of performance measures based on a given testing instance, of the one or more testing instances, comprises:
causing the given client device to process testing instance input for the given testing instance to generate output;
and
causing the given client device to generate the plurality of performance measures based on the processing of the testing instance input of the given testing instance and/or based on the output;
determining, based on the plurality of performance measures, whether to activate the new software application feature for the given software application at the given client device;
and
in response to determining to activate the new software application feature for the given software application at the given client device:
causing the new software application feature to be activated for use locally by the given software application at the given client device.

11 more elements

A method implemented by one or more processors, the method comprising:

receiving, from a developer client device of a developer, an indication of a new feature to be rolled out, wherein the developer is a first-party developer that is associated with an entity that hosts a system for generating the plurality of performance measures, or is a third-party developer that is associated with an additional entity that is in addition to the entity that hosts the system for generating the plurality of performance measures;
selecting, from among a plurality of client devices, a subset of the plurality of client devices to be utilized in testing the new feature;
causing a given client device, from among the subset of the plurality of client devices, to generate a plurality of performance measures for the new feature based on one or more testing instances,
wherein causing the given client device to generate the plurality of performance measures based on a given testing instance, of the one or more testing instances, comprises:
causing the given client device to process testing instance input for the given testing instance to generate output;
and
causing the given client device to generate the plurality of performance measures based on the processing of the testing instance input of the given testing instance and/or based on the output;
generating, based on the plurality of performance measures, a recommendation of whether to activate the new feature or refrain from activating the new feature;
and
causing the recommendation of whether to activate the new feature to be provided for presentation to the developer via the developer client device.

7 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/717,822Filed 11 Apr 2022 · continuation in part
Priority: 29 Mar 2021
Published: US 2022/0308975 A129 Sep 2022
Granted: 18 Aug 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G06F 11/3409
Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of…:…
G06F 8/65
Software deployment: updates
G06F 11/3006
Monitoring arrangements specially adapted to the computing system or computing system component being monitored: where the computing system is…
G06F 11/3419
For performance assessment: by assessing time

Family

Application17/215,588Parent of this continuation-in-part · filed 29 Mar 2021
US 2022/0308975 A1Pre-grant publication · published 29 Sep 2022
US 12,450,518 B2Related patent
US 12,572,439 B2Related patent
US 2022/0309389 A1Published application
US 12,711,033 B2This patent

From the US filing history. Foreign family members aren’t listed.