Apparatus and method for processing natural language

An apparatus for processing natural language comprising:

• at least one memory configured to store program code;
• and
• at least one processor configured to read the program code and operate as instructed by the program code to perform:
• collecting documents having tags;
• extracting text from the collected documents and extracting tag-related information on a tag surrounding each extracted text;
• and
• generating tokens of a preset unit by tokenizing each extracted text, generating token position information for each token in full text of a document, and setting the token and the token position information as training data in matching with the tag-related information,
• wherein the tag-related information includes structural position information of a tag in which each text is positioned,
• wherein the structural position information of the tag includes depth information and relative position information of a corresponding tag,
• wherein the depth information is information indicating a depth level of the corresponding tag,
• wherein the relative position information includes a relative position information value that is assigned to tags having the same depth level to sequentially increase or decrease according to a relative position or order between the tags having the same depth level, and
• wherein relative position information values of two adjacent tags having different depth levels do not represent a relative position or order between the two adjacent tags.