# US 11,030,183 B2 · Automatic content-based append detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,030,183 B2
- Document ID: US11030183B2
- Status: Granted
- Kind: B2
- Application number: 15/264,377
- Filed: 2016-09-13
- Published: 2017-09-14
- Granted: 2021-06-08
- Earliest claimed: 2016-03-14
- Assignees: DR HoldCo 2, Inc.
- Applicants: DR HoldCo 2, Inc.
- Inventors: Dave Brewster; Victor Tze-Yeuan Tso; Ashley Ping Jin; Quan Chuong Ta; Lakshman Roy Sankar; Whitman Kwok
- Web page: https://patentgrep.com/patent/US11030183B2

## CPC

- G06F 16/2365: {Ensuring data consistency and integrity}
- G06F 16/221: {Column-oriented storage; Management thereof}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Automatic append includes: identifying, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns, a plurality of matching columns and a plurality of non-matching columns. The matching columns comprise one or more columns among the first plurality of columns; and corresponding one or more matching columns among the second plurality of columns. The non-matching columns comprise: one or more columns among the first plurality of columns that do not match with any columns among the second plurality of columns; and one or more columns among the second plurality of columns that do not match with any columns among the first plurality of columns. Automatic append further includes obtaining a user specification of a first one or more non-matching columns to be appended to a second one or more non-matching columns, the first one or more non-matching columns and the second one or more non-matching columns being selected among the plurality of non-matching columns; and appending the first data set and the second data set according to at least the identified plurality of matching columns and the user specification.

## Claims

### Claim 1

A method, comprising:
identifying, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns:
a plurality of matching columns comprising:
one or more columns among the first plurality of columns; and
corresponding one or more matching columns among the second plurality of columns, wherein the one or more columns among the first plurality of columns and the corresponding one or more matching columns among the second plurality of columns have at least some matching content,
wherein identifying the plurality of matching columns comprises performing a clustering technique on an extracted plurality of features of cells in the first plurality of columns of the first data set and an extracted plurality of features of cells in the second plurality of columns of the second data set; and
a plurality of non-matching columns comprising:
one or more columns among the first plurality of columns, that are included in the first data set and that do not match with any columns among the second plurality of columns; and
one or more columns among the second plurality of columns, that are included in the second data set and that do not match with any columns among the first plurality of columns;
obtaining a specification of:
a selection of a first column among non-matching columns of the first data set to be appended to a second column among non-matching columns of the second data set, a selection of a first column among non-matching columns of the second data set to be appended to a second column among non-matching columns of the first data set, or both;
a change to the plurality of matching columns; or
both; and
appending at least a portion of the first data set and at least a portion of the second data set according to the plurality of matching columns and the specification, including:
appending at least some of the first one or more non-matching columns with at least some of the second one or more non-matching columns;
appending matching columns that are not subject to the change to the plurality of matching columns; or
both.

### Claim 2 (depends on 1)

The method of claim 1, wherein the identification of the plurality of matching columns and the plurality of non-matching columns includes:
extracting the plurality of features of cells in the first plurality of columns of the first data set and extracting the plurality of features of cells in the second plurality of columns of the second data set.

### Claim 3 (depends on 1)

The method of claim 1, wherein performing the clustering technique includes performing a K-means based clustering technique.

### Claim 4 (depends on 1)

The method of claim 1, wherein the identification of the plurality of matching columns and the plurality of non-matching columns further includes identifying one or more clusters with matching pairs, and one or more clusters with tied matching columns in which multiple columns from the first or the second data set independently match at least one column from the other data set.

### Claim 5 (depends on 4)

The method of claim 4, further comprising performing a pattern matching operation on the one or more clusters with the tied matching columns to identify one or more additional clusters with matching pairs.

### Claim 6 (depends on 5)

The method of claim 5, wherein the pattern matching operation is implemented as a TOPEI-based pattern matching operation.

### Claim 7 (depends on 5)

The method of claim 5, further comprising performing a title matching operation on one or more remaining clusters with the tied matching columns to identify one or more additional clusters with untied matching columns.

### Claim 8 (depends on 2)

The method of claim 2, wherein the features that are extracted for a column include one or more of: number of spaces in cells of the column, number of punctuations in the cells of the column, average length of values in the cells of the column, variance of values in the cells of the column, total number of words in the cells of the column, average number of words in the cells of the column, and/or number of symbol type transitions in the cells of the column.

### Claim 9 (depends on 1)

The method of claim 1, further comprising outputting the plurality of non-matching columns to be displayed.

### Claim 10 (depends on 1)

The method of claim 1, further comprising causing a selection interface to be provided to a user, and the selection interface being configured for the user to: select the first column among non-matching columns of the first data set to be appended to the second column among non-matching columns of the second data set, select the first column among non-matching columns of the second data set to be appended to the second column among non-matching columns of the first data set, or both.

### Claim 11 (depends on 1)

The method of claim 1, wherein the plurality of matching columns are identified based at least in part on a plurality of N-gram feature vectors.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
determining N-grams of entries in the first data set and in the second data set;
forming a plurality of matrices based at least in part on the N-grams of the entries in the first data set and in the second data set;
determining, based at least in part on the plurality of matrices, a first plurality of N-gram feature vectors corresponding to the first plurality of columns and a second plurality of N-gram feature vectors corresponding to the second plurality of columns; and
comparing one or more vectors in the first plurality of N-gram feature vectors with one or more vectors in the second plurality of N-gram feature vectors to determine the matching columns.

### Claim 13 (depends on 12)

The method of claim 12, wherein the comparing of the one or more vectors in the first plurality of N-gram feature vectors with the one or more vectors in the second plurality of N-gram feature vectors to determine the matching columns includes computing cosine similarities.

### Claim 14 (depends on 12)

The method of claim 12, wherein the comparing of the one or more vectors in the first plurality of N-gram feature vectors with the one or more vectors in the second plurality of N-gram feature vectors includes projecting the one or more vectors in the first plurality of N-gram feature vectors and the one or more vectors in the second plurality of N-gram feature vectors in a vector space.

### Claim 15 (depends on 1)

The method of claim 1, wherein the identification of the plurality of matching columns and the plurality of non-matching columns further includes identifying among clustering results one or more columns that are not grouped into any clusters as non-matching columns.

### Claim 16 (depends on 2)

The method of claim 2, wherein:
N features are extracted, N being an integer greater than 2; and
the clustering is performed in an N-dimensional space.

### Claim 17

A system, comprising:
one or more processors configured to:
identify, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns:
a plurality of matching columns comprising:
one or more columns among the first plurality of columns; and
corresponding one or more matching columns among the second plurality of columns, wherein the one or more columns among the first plurality of columns and the corresponding one or more matching columns among the second plurality of columns have at least some matching content,
wherein to identify the plurality of matching columns includes to perform a clustering technique on an extracted plurality of features of cells in the first plurality of columns of the first data set and an extracted plurality of features of cells in the second plurality of columns of the second data set; and
a plurality of non-matching columns comprising:
one or more columns among the first plurality of columns, that are included in the first data set and that do not match with any columns among the second plurality of columns; and
one or more columns among the second plurality of columns, that are included in the second data set and that do not match with any columns among the first plurality of columns;
obtain a specification of:
a selection of a first column among non-matching columns of the first data set to be appended to a second column among non-matching columns of the second data set, a selection of a first column among non-matching columns of the second data set to be appended to a second column among non-matching columns of the first data set, or both;
a change to the plurality of matching columns; or
both; and
append at least a portion of the first data set and at least a portion of the second data set according to the plurality of matching columns and the specification, including to:
append at least some of the first one or more non-matching columns with at least some of the second one or more non-matching columns;
append matching columns that are not subject to the change to the plurality of matching columns; or
both; and
one or more memories coupled to the one or more processors and configured to provide the one or more processors with instructions.

### Claim 18 (depends on 17)

The system of claim 17, wherein to identify the plurality of matching columns and the plurality of non-matching columns includes to:
extract the plurality of features of cells in the first plurality of columns of the first data set and extract the plurality of features of cells in the second plurality of columns of the second data set.

### Claim 19 (depends on 17)

The system of claim 17, wherein to perform the clustering technique includes to perform a K-means based clustering technique.

### Claim 20 (depends on 17)

The system of claim 17, wherein to identify the plurality of matching columns and the plurality of non-matching columns further includes to identify one or more clusters with matching pairs, and one or more clusters with tied matching columns in which multiple columns from the first or the second data set independently match at least one column from the other data set.

### Claim 21 (depends on 20)

The system of claim 20, wherein the one or more processors are further configured to perform a pattern matching operation on the one or more clusters with the tied matching columns to identify one or more additional clusters with matching pairs.

### Claim 22 (depends on 21)

The system of claim 21, wherein the pattern matching operation is implemented as a TOPEI-based pattern matching operation.

### Claim 23 (depends on 21)

The system of claim 21, wherein the one or more processors are further configured to perform a title matching operation on one or more remaining clusters with the tied matching columns to identify one or more additional clusters with untied matching columns.

### Claim 24 (depends on 18)

The system of claim 18, wherein the features that are extracted for a column include one or more of: number of spaces in cells of the column, number of punctuations in the cells of the column, average length of values in the cells of the column, variance of values in the cells of the column, total number of words in the cells of the column, average number of words in the cells of the column, and/or number of symbol type transitions in the cells of the column.

### Claim 25 (depends on 17)

The system of claim 17, wherein the one or more processors are further configured to cause the plurality of non-matching columns to be presented to a user.

### Claim 26 (depends on 17)

The system of claim 17, wherein the one or more processors are further configured to cause a selection interface to be provided for a user, and the selection interface being configured for the user to: select the first column among non-matching columns of the first data set to be appended to the second column among non-matching columns of the second data set, select the first column among non-matching columns of the second data set to be appended to the second column among non-matching columns of the first data set, or both.

### Claim 27 (depends on 17)

The system of claim 17, wherein the plurality of matching columns are determined based on a plurality of N-gram text feature vectors.

### Claim 28 (depends on 17)

The system of claim 17, wherein the one or more processors are further configured to:
determine N-grams of entries in the first data set and in the second data set;
form a plurality of matrices based at least in part on the N-grams of the entries in the first data set and in the second data set;
determine, based at least in part on the plurality of matrices, a first plurality of N-gram feature vectors corresponding to the first plurality of columns and a second plurality of N-gram feature vectors corresponding to the second plurality of columns; and
compare one or more vectors in the first plurality of N-gram feature vectors with one or more vectors in the second plurality of N-gram feature vectors to determine the matching columns.

### Claim 29 (depends on 28)

The system of claim 28, wherein to compare the one or more vectors in the first plurality of N-gram feature vectors with the one or more vectors in the second plurality of N-gram feature vectors to determine the matching columns includes to compute cosine similarities.

### Claim 30 (depends on 28)

The system of claim 28, wherein to compare the one or more vectors in the first plurality of N-gram feature vectors with the one or more vectors in the second plurality of N-gram feature vectors includes to project the one or more vectors in the first plurality of N-gram feature vectors and the one or more vectors in the second plurality of N-gram feature vectors in a vector space.

### Claim 31 (depends on 17)

The system of claim 17, wherein the identification of the plurality of matching columns and the plurality of non-matching columns further includes identifying among clustering results one or more columns that are not grouped into any clusters as non-matching columns.

### Claim 32 (depends on 17)

The system of claim 17, wherein:
N features are extracted, N being an integer greater than 2; and
the clustering is performed in an N-dimensional space.

### Claim 33

A computer program product embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for:
identifying, based at least in part on contents of a first data set comprising a first plurality of columns and contents of a second data set comprising a second plurality of columns:
a plurality of matching columns comprising:
one or more columns among the first plurality of columns; and
corresponding one or more matching columns among the second plurality of columns, wherein the one or more columns among the first plurality of columns and the corresponding one or more matching columns among the second plurality of columns have at least some matching content,
wherein identifying the plurality of matching columns comprises performing a clustering technique on an extracted plurality of features of cells in the first plurality of columns of the first data set and an extracted plurality of features of cells in the second plurality of columns of the second data set; and
a plurality of non-matching columns comprising:
one or more columns among the first plurality of columns, that are included in the first data set and that do not match with any columns among the second plurality of columns; and
one or more columns among the second plurality of columns, that are included in the second data set and that do not match with any columns among the first plurality of columns;
obtaining a specification of:
a selection of a first column among non-matching columns of the first data set to be appended to a second column among non-matching columns of the second data set, a selection of a first column among non-matching columns of the second data set to be appended to a second column among non-matching columns of the first data set, or both;
a change to the plurality of matching columns; or
both; and
appending at least a portion of the first data set and at least a portion of the second data set according to the plurality of matching columns and the specification, including:
appending at least some of the first one or more non-matching columns with at least some of the second one or more non-matching columns;
appending matching columns that are not subject to the change to the plurality of matching columns; or
both.