# US 8,543,645 B1 · Live experiment framework > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,543,645 B1 - Document ID: US8543645B1 - Status: Granted - Kind: B1 - Application number: 13/653,195 - Filed: 2012-10-16 - Granted: 2013-09-24 - Earliest claimed: 2010-01-04 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: Matthew I. Lloyd; Douglas H. Beeferman - Web page: https://patentgrep.com/patent/US8543645B1 ## CPC - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)} - G06F 11/3051: {Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs (verification or detection of system hardware configuration G06F11/2247)} - G10L 15/01: Assessment or evaluation of speech recognition systems - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract This disclosure generally relates to assigning and simultaneously running multiple client-side experiments on client devices. A file includes information regarding experiments that are available, including information regarding “layers,” which are logical, imaginary containers in which each experiment “resides.” Each experiment is associated with one layer. For each experiment, the file includes information regarding a location and size of the experiment within the layer. When the client device takes an action, a software module identifies a value of an identifier associated with the action. Each such identifier is associated with one or more of the layers. The software module can calculate, for each of the associated layers, a location within the layer based on the identifier value. The computer software module can identify, based on the information in the file, each experiment that overlaps with the calculated location within each layer and cause each identified experiment to be activated. ## Claims ### Claim 1 A computer-implemented method comprising: obtaining organizational information related to client-side experiments that are available for deployment on client devices, the organizational information mapping event identifiers to client-side experiments; detecting, by a server, a value of an event identifier associated with a trigger event, the trigger event comprising an action taken by a client device; identifying one or more corresponding client-side experiments that are to be activated at the client device by comparing the event identifier to the organizational information; and causing, by the server, the identified corresponding client-side experiments to be activated at the client device. ### Claim 2 (depends on 1) The method of claim 1, wherein identifying one or more corresponding client-side experiments comprises: calculating a concatenated value by concatenating the value of the event identifier; calculating a hash value by hashing the concatenated value; calculating a mod value based on the hash value; and matching the calculated mod value to the organizational information to determine the client-side experiments that are to be activated. ### Claim 3 (depends on 1) The method of claim 1, further comprising: for each identified client-side experiment, selecting a parameter set based on at least one condition; wherein activating each identified client-side experiment causes the identified client-side experiment to be activated on the client device using the selected parameter set. ### Claim 4 (depends on 3) The method of claim 3, wherein the parameter set is chosen to be a default parameter set or a modified parameter set, and the at least one condition comprises a characteristic of the client device. ### Claim 5 (depends on 1) The method of claim 1, wherein the event that triggers receiving a value of the event identifier comprises the loading of a speech application on the client device. ### Claim 6 (depends on 1) The method of claim 1, wherein the event that triggers receiving a value of the event identifier comprises the submission of a search query via the client device. ### Claim 7 (depends on 1) The method of claim 1, wherein the event that triggers receiving a value of the event identifier comprises the installation of a software application on the client device. ### Claim 8 (depends on 1) The method of claim 1, wherein the organizational information comprises information about a universe of possible values for the event identifiers, each client-side experiment being associated with one or more of the possible values, and wherein the identifying comprises matching the value of the identifier associated with the trigger event with a possible value in the universe of possible values and identifying the client-side experiments that are associated with that possible value. ### Claim 9 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: obtaining organizational information related to client-side experiments that are available for deployment on client devices, the organizational information mapping event identifiers to client-side experiments; detecting, by the one or more computers, a value of an event identifier associated with a trigger event, the trigger event comprising an action taken by a client device; identifying one or more corresponding client-side experiments that are to be activated at the client device by comparing the event identifier to the organizational information; and causing, by the one or more computers, the identified corresponding client-side experiments to be activated at the client device. ### Claim 10 (depends on 9) The system of claim 9, wherein identifying one or more corresponding client-side experiments comprises: calculating a concatenated value by concatenating the value of the event identifier; calculating a hash value by hashing the concatenated value; calculating a mod value based on the hash value; and matching the calculated mod value to the organizational information to determine the client-side experiments that are to be activated. ### Claim 11 (depends on 9) The system of claim 9, wherein the operations further comprise: for each identified client-side experiment, selecting a parameter set based on at least one condition; wherein activating each identified client-side experiment causes the identified client-side experiment to be activated on the client device using the selected parameter set. ### Claim 12 (depends on 11) The system of claim 11, wherein the parameter set is chosen to be a default parameter set or a modified parameter set, and the at least one condition comprises a characteristic of the client device. ### Claim 13 (depends on 9) The system of claim 9, wherein the event that triggers receiving a value of the event identifier comprises the loading of a speech application on the client device. ### Claim 14 (depends on 9) The system of claim 9, wherein the event that triggers receiving a value of the event identifier comprises the submission of a search query via the client device. ### Claim 15 (depends on 9) The system of claim 9, wherein the event that triggers receiving a value of the event identifier comprises the installation of a software application on the client device. ### Claim 16 (depends on 9) The system of claim 9, wherein the organizational information comprises information about a universe of possible values for the event identifiers, each client-side experiment being associated with one or more of the possible values, and wherein the identifying comprises matching the value of the identifier associated with the trigger event with a possible value in the universe of possible values and identifying the client-side experiments that are associated with that possible value. ### Claim 17 A non-transitory computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: obtaining organizational information related to client-side experiments that are available for deployment on client devices, the organizational information mapping event identifiers to client-side experiments; detecting, by a server, a value of an event identifier associated with a trigger event, the trigger event comprising an action taken by a client device; identifying one or more corresponding client-side experiments that are to be activated at the client device by comparing the event identifier to the organizational information; and causing, by the server, the identified corresponding client-side experiments to be activated at the client device. ### Claim 18 (depends on 17) The computer-readable storage device of claim 17, wherein identifying one or more corresponding client-side experiments comprises: calculating a concatenated value by concatenating the value of the event identifier; calculating a hash value by hashing the concatenated value; calculating a mod value based on the hash value; and matching the calculated mod value to the organizational information to determine the client-side experiments that are to be activated. ### Claim 19 (depends on 17) The computer-readable storage device of claim 17, wherein the operations further comprise: for each identified client-side experiment, selecting a parameter set based on at least one condition; wherein activating each identified client-side experiment causes the identified client-side experiment to be activated on the client device using the selected parameter set. ### Claim 20 (depends on 19) The computer-readable storage device of claim 19, wherein the parameter set is chosen to be a default parameter set or a modified parameter set, and the at least one condition comprises a characteristic of the client device.