System and method for modelling and profiling in multiple languages

A method for generating feature vectors of documents in different languages, the feature vectors providing scores associated with keywords defined in a base language for use by a profiler for generating or updating a user profile defining user preferences, the method comprising:

• a. receiving at a processing unit a document and identifying the language of the document as one of a base language or a second language;
• b. selecting a corresponding tokenizer based on the identified language to parse the received document and produce a feature vector of the received document, the tokenizer selected from a plurality of tokenizers stored in a memory unit, each tokenizer of the plurality of tokenizers associated with a language and a respective keyword set of a plurality of keyword sets stored in the memory unit;
• c. parsing the received document using the selected tokenizer to identify keywords occurring in the received document, the keywords stored in a keyword set associated with the language of the selected tokenizer and an ID of a corresponding keyword in a base language keyword set;
• and
• d. generating a feature vector from a plurality of ID:score pairs, each pair associating a score with an ID of a keyword in the associated keyword set occurring in the document, the score based on the frequency of occurrence of the ID corresponding to the keyword in the document.

A computer readable memory storing instructions for configuring a processing unit and a memory unit for providing a method for generating feature vectors of documents in different languages, the feature vectors providing scores associated with keywords defined in a base language for use by a profiler for generating or updating a user profile defining user content preferences, the method comprising:

• a. receiving at the processing unit a document and identifying the language of the document as one of a base language or a second language;
• b. selecting a corresponding tokenizer based on the identified language to parse the received document and produce a feature vector of the received document, the tokenizer selected from a plurality of tokenizers stored in a memory unit, each tokenizer of the plurality of tokenizers associated with a language and a respective keyword set of a plurality of keyword sets stored in the memory unit;
• c. parsing the received document using the selected tokenizer to identify keywords occurring in the received document, the keywords stored in a keyword set associated with the language of the selected tokenizer and an ID of a corresponding keyword in a base language keyword set;
• and
• d. generating a feature vector from a plurality of ID:score pairs, each pair associating a score with an ID of a keyword in the associated keyword set occurring in the document, the score based on the frequency of occurrence of the ID corresponding to the keyword in the document.