# US 12,242,515 B2 · System and method for custom label embedding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,242,515 B2 - Document ID: US12242515B2 - Status: Granted - Kind: B2 - Application number: 18/300,032 - Filed: 2023-04-13 - Published: 2024-10-17 - Granted: 2025-03-04 - Earliest claimed: 2023-04-13 - Assignees: User Testing, Inc. - Applicants: User Testing, Inc. - Inventors: Ranjitha Kumar; Kristijan IVancic; Kelsey Elizabeth Turbeville; Ali Hur Zaidi; Jenny Gutierrez Villalobos; Jason Matthew Moss - Web page: https://patentgrep.com/patent/US12242515B2 ## CPC - G06F 16/285: {Clustering or classification} - G06F 16/2237: {Vectors, bitmaps or matrices} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-based method to identify test user intent during evaluation of user interaction with a digital asset is provided comprising: receiving test user interaction data; producing a vector based upon the received data; using a machine learning model to project the vector to a first non-custom label embedding vector space; using the machine learning model to project the vector to a second custom label embedding vector space; determining a starting label associated with the vector based at least in part upon a nearest neighbor search within the first embedding vector space for a label nearest to the vector and a nearest neighbor search within the second embedding vector space for a nearest neighbor to the vector, receiving identification of a custom label; and adjusting the second embedding vector space based upon the custom label. ## Claims ### Claim 1 A computer-based method to identify test user intent during evaluation of user interaction with a digital asset comprising: receiving data indicative of test user interaction with the digital asset; producing a vector representing the received data; using a trained machine learning model to project the vector to a first non-custom label embedding vector space, stored in a storage memory; using the trained machine learning model to project the vector to a second custom label embedding vector space, stored in the storage memory; determining a starting label associated with the vector based at least in part upon a nearest neighbor search within the first non-custom label embedding vector space for a label nearest to the vector and a nearest neighbor search within the second custom label embedding vector space for a label nearest to the vector; receiving user input associating a custom label with the starting label; and adjusting the second custom label embedding vector space based upon the user input. ### Claim 2 (depends on 1) The method of claim 1, wherein determining further includes prioritizing a starting label returned from within the second vector space over a starting label returned from the first vector space. ### Claim 3 (depends on 1) The method of claim 1 further including: causing displaying at a computing device, a visual representation of the received data on a display screen; and causing displaying at the computing device, the determined starting label on the display screen in association with the visual representation. ### Claim 4 (depends on 3) The method of claim 3, wherein the visual representation includes a Sankey diagram. ### Claim 5 (depends on 3) The method of claim 3, wherein the received data includes test user speech recorded during test user interaction with the digital asset; and wherein the visual representation includes a transcript of the recorded test user speech. ### Claim 6 (depends on 1) The method of claim 1, wherein the received data are indicative of a sequence of test user interactions with the digital asset. ### Claim 7 (depends on 1) The method of claim 1, wherein the received data are indicative of values for variable features of the digital asset during the test user interaction with the digital asset. ### Claim 8 (depends on 1) The method of claim 1, wherein the received data are indicative of a sequence of values for variable features of the digital asset during a corresponding sequence of test user interactions with the digital asset. ### Claim 9 (depends on 1) The method of claim 1 further including: receiving user input identifying the starting label. ### Claim 10 (depends on 1) The method of claim 1 further including: receiving user input providing an adjustment command; wherein adjusting includes adjusting the second custom label embedding vector space based upon the adjustment command. ### Claim 11 (depends on 1) The method of claim 1, wherein the machine learning model is trained based at least in part upon multiple non-custom labels located at vector locations within the first embedding vector space. ### Claim 12 (depends on 1) The method of claim 1, wherein the second custom label embedding vector space comprises a copy of at least a portion of the first embedding vector space. ### Claim 13 (depends on 1) The method of claim 1, wherein the machine learning model is trained based at least in part upon multiple non-custom labels located at vector locations within the first non-custom label embedding vector space; wherein the second custom label embedding vector space comprises a copy of at least a portion of the first non-custom label embedding vector space. ### Claim 14 (depends on 1) The method of claim 1, wherein the vector represents an element text; and wherein based at least in part upon both element text data represented by the vector and the custom label being present in the second custom label embedding vector space, adjusting includes: associating the custom label as a nearest neighbor to element text data represented by the vector. ### Claim 15 (depends on 10) The method of claim 10, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector being present in the second custom label embedding vector space and the custom label not being present in the second custom label embedding vector space and receiving the received adjustment command indicating the starting label is too general, adjusting includes: adding the custom label to the second custom label embedding vector space; and moving the element text data within the second custom label embedding vector space to cause the custom label to be a nearest neighbor to element text data represented by the vector to prioritize a nearest neighbor association between the custom label and the element text data over a nearest neighbor association between the starting label and the element text data. ### Claim 16 (depends on 10) The method of claim 10, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector being present in the second custom label embedding vector space and the custom label not being present in the second custom label embedding vector space and the received adjusting command indicating the starting label is incorrect, adjusting includes: adding the custom label to the second custom label embedding vector space; and associating the custom label as a nearest neighbor to the text data within the second custom label embedding vector space; and deprioritizing a nearest neighbor association between the starting label and the element text. ### Claim 17 (depends on 1) The method of claim 1, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the second embedding and the custom label being present in the second custom label embedding vector space, adjusting includes: adding the element text to the second custom label embedding vector space; and associating the element text and the custom label as nearest neighbors within the second custom label embedding vector space. ### Claim 18 (depends on 1) The method of claim 1, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the second custom label embedding vector space and the custom label not being included in the second custom label embedding vector space and the element text including pre-population words within the second custom label embedding vector space, adjusting including: determining a vector location for the element text based upon an average of vector locations of pre-population words included within the element text. ### Claim 19 (depends on 1) The method of claim 1, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the second embedding and the custom label not being present in the second custom label embedding vector space, adjusting includes: creating a reference label vector in the custom embedding vector space; producing an element text vector based upon the reference vector within custom embedding vector space; associating the element text with the element text vector; and receiving user input to associate a custom label to the reference vector. ### Claim 20 (depends on 1) The method of claim 1, wherein adjusting includes moving the vector within the second custom label embedding vector space to cause the custom label to be a nearest neighbor of the vector within the custom label embedding vector space. ### Claim 21 (depends on 1) The method of claim 1, wherein receiving user input associating a custom label with the starting label includes receiving user input associating a new custom label with the starting label. ### Claim 22 A system to identify test user intent during evaluation of user interaction with a digital asset comprising: processor circuitry; and a memory storing instructions which, when executed by the processor circuitry, cause the processing circuitry to perform operations comprising: receiving data indicative of test user interaction with the digital asset; producing a vector representing the received data; using a trained machine learning model to project the vector to a first non-custom label embedding vector space, stored in a storage memory; using the trained machine learning model to project the vector to a second custom label embedding vector space, stored in the storage memory; determining a starting label associated with the vector based at least in part upon a nearest neighbor search within the first non-custom label embedding vector space for a label nearest to the vector and a nearest neighbor search within the second custom label embedding vector space for a label nearest to the vector; receiving user input associating a custom label with the starting label; and adjusting the second custom label embedding vector space based upon the user input. ### Claim 23 (depends on 22) The system of claim 22, wherein determining further includes prioritizing a starting label returned from within the second vector space over a starting label returned from the first vector space. ### Claim 24 (depends on 22) The system of claim 22 further including: causing displaying at a computing device, a visual representation of the received data on a display screen; and causing displaying at the computing device, the determined starting label on the display screen in association with the visual representation. ### Claim 25 (depends on 24) The system of claim 24, wherein the visual representation includes a Sankey diagram. ### Claim 26 (depends on 24) The system of claim 24, wherein the received data includes test user speech recorded during test user interaction with the digital asset; and wherein the visual representation includes a transcript of the recorded test user speech. ### Claim 27 (depends on 1) The system of claim 1, wherein the received data are indicative of a sequence of test user interactions with the digital asset. ### Claim 28 (depends on 22) The system of claim 22, wherein the received data are indicative of values for variable features of the digital asset during the test user interaction with the digital asset. ### Claim 29 (depends on 22) The system of claim 22, wherein the received data are indicative of a sequence of values for variable features of the digital asset during a corresponding sequence of test user interactions with the digital asset. ### Claim 30 (depends on 22) The system of claim 22 further including: receiving user input identifying the starting label. ### Claim 31 (depends on 22) The system of claim 22 further including: receiving user input providing an adjustment command; wherein adjusting includes adjusting the second custom label embedding vector space based upon the adjustment command. ### Claim 32 (depends on 22) The system of claim 22, wherein the machine learning model is trained based at least in part upon multiple non-custom labels located at vector locations within the first non-custom label embedding vector space. ### Claim 33 (depends on 22) The system of claim 22, wherein the second custom label embedding vector space comprises a copy of at least a portion of the first non-custom label embedding vector space. ### Claim 34 (depends on 22) The system of claim 22, wherein the machine learning model is trained based at least in part upon multiple non-custom labels located at vector locations within the first non-custom label embedding vector space; wherein the second custom label embedding vector space comprises a copy of at least a portion of the first non-custom label embedding vector space. ### Claim 35 (depends on 22) The system of claim 22, wherein the vector represents an element text; and wherein based at least in part upon both element text data represented by the vector and the custom label being present in the second custom label embedding vector space, adjusting includes: associating the custom label as a nearest neighbor to element text data represented by the vector. ### Claim 36 (depends on 31) The system of claim 31, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector being present in the second custom label embedding vector space and the custom label not being present in the second custom label embedding vector space and receiving the received adjustment command indicating the starting label is too general, adjusting includes: adding the custom label to the second custom label embedding vector space; and moving the element text data within the second custom label embedding vector space to cause the custom label to be a nearest neighbor to element text data represented by the vector to prioritize a nearest neighbor association between the custom label and the element text data over a nearest neighbor association between the starting label and the element text data. ### Claim 37 (depends on 31) The system of claim 31, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector being present in the second custom label embedding vector space and the custom label not being present in the second custom label embedding vector space and the received adjustment command indicating the starting label is incorrect, adjusting includes: adding the custom label to the second custom label embedding vector space; and associating the custom label as a nearest neighbor to the text data within the second custom label embedding vector space; and deprioritizing a nearest neighbor association between the starting label and the element text. ### Claim 38 (depends on 22) The system of claim 22, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the second embedding and the custom label being present in the second custom label embedding vector space, adjusting includes: adding the element text to the second custom label embedding vector space; and associating the element text and the custom label as nearest neighbors within the second custom label embedding vector space. ### Claim 39 (depends on 22) The system of claim 22, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the second custom label embedding vector space and the custom label not being included in the second custom label embedding vector space and the element text including pre-population words within the second custom label embedding vector space, adjusting including: determining a vector location for the element text based upon an average of vector locations of pre-population words included within the element text. ### Claim 40 (depends on 22) The system of claim 22, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the second embedding and the custom label not being present in the second custom label embedding vector space, adjusting includes: creating a reference label vector in the custom embedding vector space; producing an element text vector based upon the reference vector within custom embedding vector space; associating the element text with the element text vector; and receiving user input to associate a custom label to the reference vector. ### Claim 41 A digital experience analysis system comprising: a storage memory database including a plurality of different respective storage memory regions corresponding to a plurality of different respective groups, wherein different respective storage memory regions associated with a different respective groups, store different respective copies of a global embedding vector space and store different respective custom embedding vector spaces; processor circuitry; and a memory storing instructions which, when executed by the processor circuitry, cause the processor circuitry to perform operations comprising: receiving data identifying a group from among the different respective groups, wherein the data is indicative of test user interaction with a digital asset; producing a vector representing the received data; using a trained machine learning model to project the vector to an instance of a global label embedding vector space at a storage region associated with the identified group; using the trained machine learning model to project the vector to a custom label embedding vector space at a storage region associated with the identified group; determining a starting label associated with the vector based at least in part upon a nearest neighbor search within the instance of the global embedding vector space associated with the identified group for a label nearest to the vector and a nearest neighbor search within the custom embedding vector space associated with the identified group for a label nearest to the vector; receiving user input associating a custom label with the determined starting label; and adjusting the custom embedding vector space associated with the identified group based upon the received user input. ### Claim 42 (depends on 41) The system of claim 41, wherein the machine learning model is trained based at least in part upon multiple non-custom labels located at vector locations within the global embedding vector space. ### Claim 43 (depends on 41) The system of claim 41, wherein determining further includes prioritizing a starting label returned from within the custom vector space associated with the identified group over a starting label returned from the instance of the global vector space associated with the identified group. ### Claim 44 (depends on 41) The system of claim 41 further including: wherein adjusting includes adjusting the custom embedding vector space to associate the custom label as a nearest neighbor with the vector. ### Claim 45 (depends on 41) The system of claim 41 further including: receiving user input providing an adjustment command; wherein adjusting includes adjusting the custom label embedding vector space based upon the adjustment command. ### Claim 46 (depends on 41) The system of claim 41, wherein the vector represents an element text; and wherein based at least in part upon both element text data represented by the vector and the custom label being present in the custom embedding vector space associated with the identified group, adjusting includes: associating the custom label as a nearest neighbor to element text data represented by the vector within the custom embedding vector space associated with the identified group. ### Claim 47 (depends on 45) The system of claim 45, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector being present in the custom embedding vector space associated with the identified group and the custom label not being present in the custom embedding vector space associated with the identified group and receiving the received adjustment command indicating the starting label is too general, adjusting includes: adding the custom label to the custom embedding vector space associated with the identified group; and moving the element text data within the custom label embedding vector space to cause the custom label to be a nearest neighbor to element text data represented by the vector to prioritize a nearest neighbor association between the custom label and the element text data within the custom embedding vector space associated with the identified group over a nearest neighbor association between the starting label and the element text data within the custom embedding vector space associated with the identified group. ### Claim 48 (depends on 45) The system of claim 45, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector being present in the custom embedding vector space associated with the identified group and the custom label not being present in the custom embedding vector space associated with the identified group and the received adjusting command indicating the starting label is incorrect, adjusting includes: adding the custom label to the custom embedding vector space associated with the identified group; associating the custom label as a nearest neighbor to the text data within the custom label embedding vector space within the custom embedding vector space associated with the identified group; and deprioritizing a nearest neighbor association between the starting label and the element text within the custom embedding vector space associated with the identified group. ### Claim 49 (depends on 41) The system of claim 41, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the custom embedding associated with the identified group and the custom label being present in the custom embedding vector space associated with the identified group, adjusting includes: adding the element text to the custom embedding vector space associated with the identified group; and associating the element text and the custom label as nearest neighbors within the custom embedding vector space associated with the identified group. ### Claim 50 (depends on 41) The system of claim 41, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the custom embedding vector space associated with the identified group and the custom label not being included in the custom embedding vector space associated with the identified group and the element text including pre-population words within the custom label embedding vector space, adjusting including: determining a vector location for the element text within the custom embedding vector space associated with the identified group, based upon an average of vector locations of pre-population words included within the element text. ### Claim 51 (depends on 41) The system of claim 41, wherein the vector represents an element text; and wherein based at least in part upon element text data represented by the vector not being present in the global embedding vector space and the custom label not being present in the second custom label embedding vector space, adjusting includes: creating a reference label vector in the custom embedding vector space; producing an element text vector based upon the reference vector within custom embedding vector space; associating the element text with the element text vector; and receiving user input to associate a custom label to the reference vector. ### Claim 52 (depends on 41) The digital experience analysis system of claim 41, wherein adjusting includes moving the vector within the custom label embedding vector space to cause the custom label to be a nearest neighbor of the vector within the custom label embedding vector space. ### Claim 53 (depends on 41) The digital experience analysis system of claim 41, wherein receiving user input associating a custom label with the starting label includes receiving user input associating a new custom label with the starting label. ### Claim 54 A digital experience analysis system comprising: a storage memory database that stores one or more instances of a global label embedding vector space and that includes a plurality of different respective storage memory regions corresponding to a plurality of different respective groups, wherein different respective storage memory regions store different respective custom embedding vector spaces associated with a different respective groups; processor circuitry; and a memory storing instructions which, when executed by the processor circuitry, cause the processor circuitry to perform operations comprising: receiving data identifying a group from among the different respective groups, wherein the data is indicative of test user interaction with a digital asset; producing a vector representing the received data; using a trained machine learning model to project the vector to an instance of the one or more instances of the global label embedding vector space stored within the database; using the trained machine learning model to project the vector to a custom label embedding vector space at a storage region associated with the identified group; determining a starting label associated with the vector based at least in part upon a nearest neighbor search within the instance of the one or more instances of the global label embedding vector space for a label nearest to the vector and a nearest neighbor search within the custom embedding vector space associated with the identified group for a label nearest to the vector; receiving user input associating a custom label with the determined starting label; and adjusting the custom embedding vector space associated with the identified group based upon the received user input. ### Claim 55 (depends on 54) The digital experience analysis system of claim 54, wherein adjusting includes moving the vector within the custom label embedding vector space to cause the custom label to be a nearest neighbor of the vector within the custom label embedding vector space. ### Claim 56 (depends on 22) The system to identify test user intent of claim 22, wherein adjusting includes moving the vector within the custom label embedding vector space to cause the custom label to be a nearest neighbor of the vector within the custom label embedding vector space. ### Claim 57 (depends on 54) The digital experience analysis system of claim 54, wherein receiving user input associating a custom label with the starting label includes receiving user input associating a new custom label with the starting label. ### Claim 58 A computer-implemented method of digital experience analysis comprising: storing one or more instances of a global label embedding vector space in a storage memory; storing multiple different respective custom embedding vector spaces at different respective storage memory regions within the storage memory, the different respective custom embedding spaces associated with different respective groups having different respective language domains; receiving data identifying a group from among the different respective groups, wherein the data is indicative of test user interaction with a digital asset; producing a vector representing the received data; using a trained machine learning model to project the vector to an instance of the one or more instances of the global label embedding vector space stored within the database; using the trained machine learning model to project the vector to a custom label embedding vector space at a storage region associated with the identified group; determining a starting label associated with the vector based at least in part upon a nearest neighbor search within the instance of the one or more instances of the global label embedding vector space for a label nearest to the vector and a nearest neighbor search within the custom embedding vector space associated with the identified group for a label nearest to the vector; receiving user input associating a custom label with the determined starting label; and adjusting the custom embedding vector space associated with the identified group based upon the received user input. ### Claim 59 (depends on 58) The method of claim 58, wherein storing the one or more instances of the global embedding vector space in the storage memory includes storing a different respective instance of the global embedding vector space within the storage memory for each different respective group. ### Claim 60 (depends on 58) The computer-implemented method of digital experience analysis of claim 58, wherein adjusting includes moving the vector within the custom label embedding vector space to cause the custom label to be a nearest neighbor of the vector within the custom label embedding vector space. ### Claim 61 (depends on 58) The computer-implemented method of digital experience analysis of claim 58, wherein receiving user input associating a custom label with the starting label includes receiving user input associating a new custom label with the starting label.