Scalable processing of heterogeneous user-generated content

An apparatus, comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
• obtain a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source, wherein the first and second sets of content items have different layouts;
• and
• for each content item in the first set of content items:
• use a set of mappings from the first configuration to transform a set of original fields from the content item into a set of required fields in a record representing the content item;
• generate, from one or more of the required fields, a document key for the content item;
• when the document key is generated for another content item, perform deduplication of the record and another record for the other content item;
• and
• store the record for use in subsequent querying of content from the first data source;
• and
• use the second configuration to generate, from the second set of content items, a second set of records for use in subsequent querying of content from the second data source independently of processing the first set of content items from the first data source.

A system, comprising:

• an analysis module comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to:
• obtain a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source, wherein the first and second sets of content items have different layouts;
• for each content item in the first set of content items:
• use a set of mappings from the first configuration to transform a set of original fields from the content item into a set of required fields in a record representing the content item;
• generate, from one or more of the required fields, a document key for the content item;
• and
• when the document key is generated for another content item, perform deduplication of the record and another record for the other content item;
• and
• use the second configuration to generate, from the second set of content items, a second set of records from the second data source independently of processing the first set of content items from the first data source;
• and
• a storage mechanism comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to store the record for use in subsequent querying of content from the first and second data sources.