# US 2023/0058829 A1 · Weakly supervised extraction of attributes from unstructured data to generate training data for machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0058829 A1
- Document ID: US20230058829A1
- Status: Published application
- Kind: A1
- Application number: 17/407,158
- Filed: 2021-08-19
- Published: 2023-02-23
- Earliest claimed: 2021-08-19
- Assignees: Maplebear, Inc.(dba Instacart)
- Applicants: Maplebear Inc. (dba Instacart)
- Inventors: Shih-Ting Lin; Jonathan Newman; Min Xie; Haixun Wang
- Web page: https://patentgrep.com/patent/US20230058829A1

## CPC

- G06F 18/23: Clustering techniques
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06Q 30/0641: {utilising user interfaces specially adapted for shopping}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

An online concierge system receives unstructured data describing items offered for purchase by various warehouses. To generate attributes for products from the unstructured data, the online concierge system extracts candidate values for attributes from the unstructured data through natural language processing. One or more users associate a subset candidate values with corresponding attributes, and the online concierge system clusters the remaining candidate values with the candidate values of the subset associated with attributes. One or more users provide input on the accuracy of the generated clusters. The candidate values are applied as labels to items by the online concierge system, which uses the labeled items as training data for an attribute extraction model to predict values for one or more attributes from unstructured data about an item.

## Claims

### Claim 1

A method comprising:
obtaining unstructured data describing items for display by an online concierge system, the unstructured data including a name of each item;
identifying a set of items each having a common characteristic;
extracting candidate values for attributes as segments from names of each item of the set, each candidate value associated with a frequency with which the segment occurs in the set of items;
identifying a subset of candidate values based on frequency of occurrence in the set of items;
generating a seed set of candidate values for one or more attributes from inputs received from one or more users associating candidate values of the subset with one or more attributes;
generating clusters of candidate values from distances between candidate values not included in the subset and candidate values of the subset associated with one or more attributes, each cluster of candidate value corresponding to an attribute and including candidate values that are potential values for the attribute;
receiving input from one or more users manually reviewing the generated clusters for accuracy;
applying one or more labels to each item of the set, a label applied to an item of the set indicating a candidate value corresponding to an attribute matching a segment extracted from a name of the item of the set;
generating training data including a plurality of examples, each example including an identifier of the item of the set and the one or more labels applied to the item of the set; and
training an attribute extraction model to predict values for one or more attributes of an item from unstructured data describing the item by applying the attribute extraction model to the plurality of examples of the training data.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the subset of candidate values based on frequency of occurrence in the set of items comprises:
selecting candidate values having at least a threshold frequency of occurrence in the set of items.

### Claim 3 (depends on 1)

The method of claim 1, wherein identifying the subset of candidate values based on frequency of occurrence in the set of items comprises:
ranking candidate values based on frequency of occurrence in the set of items; and
selecting candidate values having at least a threshold position in the ranking.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating clusters of candidate values from distances between candidate values not included in the subset and candidate values of the subset associated with one or more attributes comprises:
identifying seed clusters as candidate values of the subset associated each associated with a common attribute;
identifying a candidate value not included in the subset;
determining a distance between the candidate value not included in the subset and each seed cluster; and
generating a cluster including the candidate value not included in the subset and a seed cluster having less than a threshold distance to the candidate value not included in the subset.

### Claim 5 (depends on 4)

The method of claim 4, wherein the distance between the candidate value not included in the subset a seed cluster is based on semantic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 6 (depends on 4)

The method of claim 4, wherein the distance between the candidate value not included in the subset a seed cluster is based on syntactic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating clusters of candidate values from distances between candidate values not included in the subset and candidate values of the subset associated with one or more attributes comprises:
identifying seed clusters as candidate values of the subset associated each associated with a common attribute;
identifying a candidate value not included in the subset;
determining a distance between the candidate value not included in the subset and each seed cluster; and
generating a cluster including the candidate value not included in the subset and a seed cluster having a minimum distance to the candidate value not included in the subset.

### Claim 8 (depends on 7)

The method of claim 7, wherein the distance between the candidate value not included in the subset a seed cluster is based on semantic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 9 (depends on 7)

The method of claim 7, wherein the distance between the candidate value not included in the subset a seed cluster is based on syntactic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
applying the trained attribute extraction model to unstructured data describing additional items;
displaying one or more of the additional items to one or more users using predicted values for one or more of the additional items;
receiving feedback from a user to whom the one or more additional items were displayed; and
updating the training data based on the received feedback.

### Claim 11 (depends on 1)

The method of claim 1, wherein receiving input from one or more users manually reviewing the generated clusters for accuracy comprises:
receiving an input from a user identifying a generated cluster corresponding to an attribute and an indication whether candidate values included in the generated cluster are accurate.

### Claim 12 (depends on 11)

The method of claim 11, wherein the input identifies one or more candidate values to remove from the generated cluster.

### Claim 13

A computer program product comprising a non-transitory computer readable medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
obtain unstructured data describing items for display by an online concierge system, the unstructured data including a name of each item;
identify a set of items each having a common characteristic;
extract candidate values for attributes as segments from names of each item of the set, each candidate value associated with a frequency with which the segment occurs in the set of items;
identify a subset of candidate values based on frequency of occurrence in the set of items;
generate a seed set of candidate values for one or more attributes from inputs received from one or more users associating candidate values of the subset with one or more attributes;
generate clusters of candidate values from distances between candidate values not included in the subset and candidate values of the subset associated with one or more attributes, each cluster of candidate value corresponding to an attribute and including candidate values that are potential values for the attribute;
receive input from one or more users manually reviewing the generated clusters for accuracy;
apply one or more labels to each item of the set, a label applied to an item of the set indicating a candidate value corresponding to an attribute matching a segment extracted from a name of the item of the set;
generate training data including a plurality of examples, each example including an identifier of the item of the set and the one or more labels applied to the item of the set; and
train an attribute extraction model to predict values for one or more attributes of an item from unstructured data describing the item by applying the attribute extraction model to the plurality of examples of the training data.

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein identify the subset of candidate values based on frequency of occurrence in the set of items comprises:
select candidate values having at least a threshold frequency of occurrence in the set of items.

### Claim 15 (depends on 13)

The computer program product of claim 13, wherein identify the subset of candidate values based on frequency of occurrence in the set of items comprises:
rank candidate values based on frequency of occurrence in the set of items; and
select candidate values having at least a threshold position in the ranking.

### Claim 16 (depends on 13)

The computer program product of claim 13, wherein generate clusters of candidate values from distances between candidate values not included in the subset and candidate values of the subset associated with one or more attributes comprises:
identify seed clusters as candidate values of the subset associated each associated with a common attribute;
identify a candidate value not included in the subset;
determine a distance between the candidate value not included in the subset and each seed cluster; and
generate a cluster including the candidate value not included in the subset and a seed cluster having less than a threshold distance to the candidate value not included in the subset.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein the distance between the candidate value not included in the subset a seed cluster is based on semantic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 18 (depends on 16)

The computer program product of claim 16, wherein the distance between the candidate value not included in the subset a seed cluster is based on syntactic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 19 (depends on 13)

The computer program product of claim 13, wherein generate clusters of candidate values from distances between candidate values not included in the subset and candidate values of the subset associated with one or more attributes comprises:
identify seed clusters as candidate values of the subset associated each associated with a common attribute;
identify a candidate value not included in the subset;
determine a distance between the candidate value not included in the subset and each seed cluster; and
generate a cluster including the candidate value not included in the subset and a seed cluster having a minimum distance to the candidate value not included in the subset.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the distance between the candidate value not included in the subset a seed cluster is based on semantic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 21 (depends on 19)

The computer program product of claim 19, wherein the distance between the candidate value not included in the subset a seed cluster is based on syntactic distances between the not included in the subset and candidate values included in the seed cluster.

### Claim 22 (depends on 13)

The computer program product of claim 13, wherein the non-transitory computer readable storage medium further has instructions encoded thereon that, when executed by the processor, causes the processor to:
apply the trained attribute extraction model to unstructured data describing additional items;
display one or more of the additional items to one or more users using predicted values for one or more of the additional items;
receive feedback from a user to whom the one or more additional items were displayed; and
update the training data based on the received feedback.

### Claim 23 (depends on 13)

The computer program product of claim 13, wherein receive input from one or more users manually reviewing the generated clusters for accuracy comprises:
receive an input from a user identifying a generated cluster corresponding to an attribute and an indication whether candidate values included in the generated cluster are accurate.

### Claim 24 (depends on 23)

The computer program product of claim 23, wherein the input identifies one or more candidate values to remove from the generated cluster.
