# US 2024/0193167 A1 · Database systems with adaptive automated metadata assignment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0193167 A1
- Document ID: US20240193167A1
- Status: Published application
- Kind: A1
- Application number: 18/064,696
- Filed: 2022-12-12
- Published: 2024-06-13
- Earliest claimed: 2022-12-12
- Assignees: Salesforce, Inc.
- Applicants: Salesforce, Inc.
- Inventors: Zachary Alexander; Yixin Mao
- Web page: https://patentgrep.com/patent/US20240193167A1

## CPC

- G06F 16/24573: {using data annotations, e.g. user-defined metadata}
- G06F 16/244: {Grouping and aggregation}
- G06F 16/285: {Clustering or classification}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assigning structural metadata involves determining a candidate group of semantically similar conversations based on unassigned conversations, determining a clustering performance metric associated with the candidate group based on a relationship between the candidate group and a plurality of existing groups of semantically similar conversations, and when the clustering performance metric is greater than a threshold, automatically assigning one or more unassigned conversations to the candidate group based on the representative utterances associated therewith and automatically updating one or more associated records at a database system to include metadata identifying the candidate group.

## Claims

### Claim 1

A method comprising:
determining, based upon a plurality of unassigned conversations, a new candidate group of semantically similar conversations comprising a subset of the plurality of unassigned conversations, wherein the new candidate group is distinct from a plurality of existing groups of semantically similar conversations, wherein a respective existing group of the plurality of existing groups comprises a distinct subset of conversations assigned to the respective existing group based on a respective representative utterance associated with the respective conversation of the distinct subset of conversations;
determining a clustering performance metric associated with the new candidate group of semantically similar conversations based on a relationship between a reference semantic representation for the new candidate group of semantically similar conversations and respective semantic representations for the plurality of existing groups of semantically similar conversations; and
when the clustering performance metric is greater than a threshold:
automatically assigning one or more conversations of the plurality of unassigned conversations to the new candidate group based on a relationship between a respective representative utterance associated with a respective conversation of the one or more conversations and the reference semantic representation for the new candidate group; and
automatically updating, at a database system, one or more records associated with the one or more conversations to include metadata identifying the new candidate group.

### Claim 2 (depends on 1)

The method of claim 1, further comprising identifying a set of out-of-domain conversations from among the plurality of unassigned conversations using the plurality of existing groups of semantically similar conversations, wherein:
the set of out-of-domain conversations comprises a first subset of the plurality of unassigned conversations; and
determining the new candidate group comprises identifying the new candidate group of semantically similar conversations comprising a second subset of the set of out-of-domain conversations.

### Claim 3 (depends on 2)

The method of claim 2, wherein identifying the new candidate group comprises clustering the second subset of the set of out-of-domain conversations into the new candidate group based on a respective representative utterance associated with the respective out-of-domain conversation of the second subset of the set of out-of-domain conversations.

### Claim 4 (depends on 2)

The method of claim 2, wherein identifying the set of out-of-domain conversations comprises:
assigning a respective unassigned conversation of the plurality of unassigned conversations to a first existing group of the plurality of existing groups based on a respective representative utterance associated with the respective unassigned conversation using a k-nearest neighbors (KNN) algorithm;
determining a KNN consistency metric associated with the respective unassigned conversation based on the first existing group and one or more neighboring conversations assigned to respective ones of the plurality of existing groups; and
adding the respective unassigned conversation to the set of out-of-domain conversations based at least in part on the KNN consistency metric.

### Claim 5 (depends on 2)

The method of claim 2, wherein identifying the set of out-of-domain conversations comprises:
assigning a respective unassigned conversation of the plurality of unassigned conversations to an unclustered group of conversations using a Gaussian mixture model (GMM) fit to the plurality of existing groups; and
adding the respective unassigned conversation to the set of out-of-domain conversations in response to the GMM assigning the respective unassigned conversation to the unclustered group of conversations.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the clustering performance metric comprises determining a silhouette score associated with a set of cluster groups comprising the new candidate group and the plurality of existing groups.

### Claim 7 (depends on 6)

The method of claim 6, wherein automatically assigning the one or more conversations comprises configuring an assignment algorithm to cluster conversations into the set of cluster groups comprising the plurality of existing groups and the new candidate group when the silhouette score associated with the set of cluster groups is greater than zero.

### Claim 8 (depends on 1)

The method of claim 1, further comprising configuring an assignment algorithm to cluster conversations into a set of cluster groups comprising the plurality of existing groups and the new candidate group when the clustering performance metric is greater than a cluster group inclusion threshold, resulting in an updated assignment algorithm, wherein automatically assigning the one or more conversations comprises automatically assigning the one or more conversations to the new candidate group based on the representative utterance associated with the respective conversation of the one or more conversations using the updated assignment algorithm.

### Claim 9 (depends on 8)

The method of claim 8, wherein configuring the assignment algorithm comprises updating a fitted Gaussian mixture model (GMM) to include the new candidate group in addition to the plurality of existing groups.

### Claim 10 (depends on 1)

The method of claim 1, further comprising determining a performance metric associated with the new candidate group after automatically assigning the one or more conversations, wherein the performance metric is influenced by one or more performance metrics associated with the one or more conversations.

### Claim 11

At least one non-transitory machine-readable storage medium that provides instructions that, when executed by at least one processor, are configurable to cause the at least one processor to perform operations comprising:
determining, based upon a plurality of unassigned conversations, a new candidate group of semantically similar conversations comprising a subset of the plurality of unassigned conversations, wherein the new candidate group is distinct from a plurality of existing groups of semantically similar conversations, wherein a respective existing group of the plurality of existing groups comprises a distinct subset of conversations assigned to the respective existing group based on a respective representative utterance associated with the respective conversation of the distinct subset of conversations;
determining a clustering performance metric associated with the new candidate group of semantically similar conversations based on a relationship between a reference semantic representation for the new candidate group of semantically similar conversations and respective semantic representations for the plurality of existing groups of semantically similar conversations; and
when the clustering performance metric is greater than a threshold:
automatically assigning one or more conversations of the plurality of unassigned conversations to the new candidate group based on a relationship between a respective representative utterance associated with a respective conversation of the one or more conversations and the reference semantic representation for the new candidate group; and
automatically updating one or more records associated with the one or more conversations to include metadata identifying the new candidate group.

### Claim 12 (depends on 11)

The at least one non-transitory machine-readable storage medium of claim 11, wherein the instructions are configurable to cause the at least one processor to identify a set of out-of-domain conversations from among the plurality of unassigned conversations using the plurality of existing groups of semantically similar conversations, wherein:
the set of out-of-domain conversations comprises a first subset of the plurality of unassigned conversations; and
determining the new candidate group comprises identifying the new candidate group of semantically similar conversations comprising a second subset of the set of out-of-domain conversations.

### Claim 13 (depends on 12)

The at least one non-transitory machine-readable storage medium of claim 12, wherein the instructions are configurable to cause the at least one processor to identify the new candidate group by clustering the second subset of the set of out-of-domain conversations into the new candidate group based on a respective representative utterance associated with a respective out-of-domain conversation of the second subset of the set of out-of-domain conversations.

### Claim 14 (depends on 12)

The at least one non-transitory machine-readable storage medium of claim 12, wherein the instructions are configurable to cause the at least one processor to identify the set of out-of-domain conversations by:
assigning a respective unassigned conversation of the plurality of unassigned conversations to a first existing group of the plurality of existing groups based on a respective representative utterance associated with the respective unassigned conversation using a k-nearest neighbors (KNN) algorithm;
determining a KNN consistency metric associated with the respective unassigned conversation based on the first existing group and one or more neighboring conversations assigned to respective ones of the plurality of existing groups; and
adding the respective unassigned conversation to the set of out-of-domain conversations based at least in part on the KNN consistency metric.

### Claim 15 (depends on 12)

The at least one non-transitory machine-readable storage medium of claim 12, wherein the instructions are configurable to cause the at least one processor to identify the set of out-of-domain conversations by:
assigning a respective unassigned conversation of the plurality of unassigned conversations to an unclustered group of conversations using a Gaussian mixture model (GMM) fit to the plurality of existing groups; and
adding the respective unassigned conversation to the set of out-of-domain conversations in response to the GMM assigning the respective unassigned conversation to the unclustered group of conversations.

### Claim 16 (depends on 11)

The at least one non-transitory machine-readable storage medium of claim 11, wherein the clustering performance metric comprises a silhouette score.

### Claim 17 (depends on 11)

The at least one non-transitory machine-readable storage medium of claim 11, wherein the clustering performance metric comprises a k-nearest neighbors (KNN) consistency metric.

### Claim 18 (depends on 11)

The at least one non-transitory machine-readable storage medium of claim 11, wherein the instructions are configurable to cause the at least one processor to configure an assignment algorithm to cluster conversations into a set of cluster groups comprising the plurality of existing groups and the new candidate group when the clustering performance metric is greater than a cluster group inclusion threshold, resulting in an updated assignment algorithm, wherein automatically assigning the one or more conversations comprises automatically assigning the one or more conversations to the new candidate group based on the respective representative utterance associated with the respective conversation of the one or more conversations using the updated assignment algorithm.

### Claim 19

A computing system comprising:
at least one non-transitory machine-readable storage medium that stores software; and
at least one processor, coupled to the at least one non-transitory machine-readable storage medium, to execute the software that implements a conversation mining service and that is configurable to perform operations comprising:
determining, based upon a plurality of unassigned conversations, a new candidate group of semantically similar conversations comprising a subset of the plurality of unassigned conversations, wherein the new candidate group is distinct from a plurality of existing groups of semantically similar conversations, wherein a respective existing group of the plurality of existing groups comprises a distinct subset of conversations assigned to the respective existing group based on a respective representative utterance associated with the respective conversation of the distinct subset of conversations;
determining a clustering performance metric associated with the new candidate group of semantically similar conversations based on a relationship between a reference semantic representation for the new candidate group of semantically similar conversations and respective semantic representations for the plurality of existing groups of semantically similar conversations; and
when the clustering performance metric is greater than a threshold:
automatically assigning one or more conversations of the plurality of unassigned conversations to the new candidate group based on a relationship between a respective representative utterance associated with a respective conversation of the one or more conversations and the reference semantic representation for the new candidate group; and
automatically updating one or more records associated with the one or more conversations to include metadata identifying the new candidate group.

### Claim 20 (depends on 19)

The computing system of claim 19, wherein the conversation mining service is configurable to cause the at least one processor to:
identify a set of out-of-domain conversations from among the plurality of unassigned conversations using the plurality of existing groups of semantically similar conversations; and
identify the new candidate group comprising a first subset of semantically similar conversations identified from the set of out-of-domain conversations.
