# US 2017/0286551 A1 · Scalable processing of heterogeneous user-generated content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286551 A1
- Document ID: US20170286551A1
- Status: Published application
- Kind: A1
- Application number: 15/136,502
- Filed: 2016-04-22
- Published: 2017-10-05
- Earliest claimed: 2016-03-29
- Assignees: LinkedIn Corporation
- Applicants: LinkedIn Corporation
- Inventors: Bin Wu; Weiqin Ma; Weidong Zhang; Liwen Zhang; Srihari R. Duddukuru; SangHyun Park; Yongzheng Zhang; Yi Zheng; Hong Lu; Yurong Shi; Chi-Yi Kuan
- Web page: https://patentgrep.com/patent/US20170286551A1

## CPC

- G06F 16/2465: {Query processing support for facilitating data mining operations in structured databases}
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 40/186: Templates
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The disclosed embodiments provide a system for processing data. During operation, the system obtains a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source. For each content item in the first set of content items, the system uses mappings from the first configuration to transform original fields from the content item into required fields in a record representing the content item. Next, the system generates, from the required fields, a document key for the content item. The system also performs deduplication of multiple records with the document key and stores a single record with the document key. Finally, the system uses the second configuration to generate, from the second set of content items, a set of records independently of processing the first set of content items.

## Claims

### Claim 1

A method, comprising:
obtaining a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source, wherein the first and second sets of content items have different layouts; and
for each content item in the first set of content items, processing the content item on one or more computer systems by:
using a set of mappings from the first configuration to transform a set of original fields from the content item into a set of required fields in a record representing the content item;
generating, from one or more of the required fields, a document key for the content item;
when the document key is generated for another content item, performing deduplication of the record and another record for the other content item; and
storing the record for use in subsequent querying of content from the first data source; and
using the second configuration to generate, from the second set of content items, a second set of records for use in subsequent querying of content from the second data source independently of processing the first set of content items from the first data source.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
adding one or more additional fields comprising data related to the content item to the record.

### Claim 3 (depends on 2)

The method of claim 2, wherein adding the one or more additional fields to the record comprises:
including, in the record, one or more words from the content item that are found in a whitelist or a blacklist of significant words.

### Claim 4 (depends on 2)

The method of claim 2, wherein adding the one or more additional fields to the record comprises:
matching the content item to a user profile; and
including one or more attributes from the user profile in the record.

### Claim 5 (depends on 4)

The method of claim 4, wherein the one or more attributes from the user profile comprise at least one of:
a number of social network connections;
a member segment;
an industry;
a company;
an occupation; and
a seniority.

### Claim 6 (depends on 2)

The method of claim 2, wherein adding the one or more additional fields to the record comprises:
applying a text-mining technique to one or more of the required fields; and
including a result of the text-mining technique in the one or more additional fields.

### Claim 7 (depends on 6)

The method of claim 6, wherein the result is at least one of:
a sentiment;
a topic;
a relevance;
a product associated with a online professional network; and
a value proposition.

### Claim 8 (depends on 1)

The method of claim 1, wherein using the second configuration to generate the second set of records independently of processing the first set of content items from the first data source comprises:
obtaining a first set of values from the first configuration and a second set of values from the second configuration;
inserting the first set of values into a first instance of a processing template and the second set of values into a second instance of the processing template; and
using the first and second instances of the processing template to independently process the first and second sets of content items without requiring manual modification of the processing template.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first and second sets of content items comprise at least one of:
a customer survey;
a complaint;
a review;
a group discussion; and
social media content.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
outputting a set of statistics related to processing of the first set of content items into a set of records.

### Claim 11 (depends on 1)

The method of claim 1, wherein the configuration further comprises:
one or more inputs;
one or more outputs; and
one or more source-specific fields.

### Claim 12 (depends on 1)

The method of claim 1, wherein the set of required fields comprises:
a source name;
a date;
a title;
a content;
a language;
a location; and
a Uniform Resource Locator (URL).

### Claim 13

An apparatus, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
obtain a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source, wherein the first and second sets of content items have different layouts; and
for each content item in the first set of content items:
use a set of mappings from the first configuration to transform a set of original fields from the content item into a set of required fields in a record representing the content item;
generate, from one or more of the required fields, a document key for the content item;
when the document key is generated for another content item, perform deduplication of the record and another record for the other content item; and
store the record for use in subsequent querying of content from the first data source; and
use the second configuration to generate, from the second set of content items, a second set of records for use in subsequent querying of content from the second data source independently of processing the first set of content items from the first data source.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:
add one or more additional fields comprising data related to the content item to the record.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein adding the one or more additional fields to the record comprises:
including, in the record, one or more words from the content item that are found in a whitelist or blacklist of significant words.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein adding the one or more additional fields to the record comprises:
matching the content item to a user profile in an online professional network; and
including one or more attributes from the user profile in the record.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein adding the one or more additional fields to the record comprises:
applying a text-mining technique to one or more of the required fields; and
including a result of the text-mining technique in the one or more additional fields.

### Claim 18 (depends on 13)

The apparatus of claim 13, wherein using the second configuration to generate the second set of records independently of processing the first set of content items from the first data source comprises:
obtaining a first set of values from the first configuration and a second set of values from the second configuration;
inserting the first set of values into a first instance of a processing template and the second set of values into a second instance of the processing template; and
using the first and second instances of the processing template to independently process the first and second sets of content items without requiring manual modification of the processing template.

### Claim 19

A system, comprising:
an analysis module comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to:
obtain a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source, wherein the first and second sets of content items have different layouts;
for each content item in the first set of content items:
use a set of mappings from the first configuration to transform a set of original fields from the content item into a set of required fields in a record representing the content item;
generate, from one or more of the required fields, a document key for the content item; and
when the document key is generated for another content item, perform deduplication of the record and another record for the other content item; and
use the second configuration to generate, from the second set of content items, a second set of records from the second data source independently of processing the first set of content items from the first data source; and
a storage mechanism comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to store the record for use in subsequent querying of content from the first and second data sources.

### Claim 20 (depends on 19)

The system of claim 19, wherein using the second configuration to generate the second set of records independently of processing the first set of content items from the first data source comprises:
obtaining a first set of values from the first configuration and a second set of values from the second configuration;
inserting the first set of values into a first instance of a processing template and the second set of values into a second instance of the processing template; and
using the first and second instances of the processing template to independently process the first and second sets of content items without requiring manual modification of the processing template.
