Method and device for segmenting word based on cross-language data augmentation, and storage medium

A device for segmenting words based on cross-language data augmentation, comprising a memory and one or more processors;

• wherein
• the memory is configured to store one or more programs;
• and
• the one or more processors, when loading and running the one or more programs, are caused to perform a method for segmenting words based on cross-language data augmentation, wherein the method comprises:
• acquiring a plurality of groups of high-resource language (HRL) data, and acquiring a plurality of groups of first word segmentation language data by processing the plurality of groups of HRL data, wherein the plurality of groups of first word segmentation language data constitute a first word segmentation corpus;
• acquiring a plurality of groups of low-resource language (LRL) data, acquiring a plurality of candidate word segments by processing the plurality of groups of LRL data;
• selecting second word segmentation language data from the plurality of candidate word segments based on a matching degree between each of the candidate word segments and the first word segmentation corpus;
• acquiring a word segmentation model by training based on the second word segmentation language data, and outputting a plurality of candidate word segmentation results by inputting a plurality of groups of to-be-segmented data into the word segmentation model;
• and
• selecting the candidate word segmentation result with a highest matching degree as a word segmentation result based on a matching degree between each of the candidate word segmentation results and the first word segmentation corpus;
• wherein selecting the second word segmentation language data from the plurality of candidate word segments based on the matching degree between each of the candidate word segments and the first word segmentation corpus comprises:
• translating the plurality of candidate word segments into a plurality of high-resource candidate word segments corresponding to the HRL data through a translation model, wherein each of the high-resource candidate word segments corresponds to a translation probability assigned by the translation model;
• acquiring an occurrence frequency of a group of first word segmentation language data consistent with the high-resource candidate word segment in the first word segmentation corpus;
• calculating a frequency index by adding the translation probability of the high-resource candidate word segment and the occurrence frequency of the group of first word segmentation language data consistent with the high-resource candidate word segment in the first word segmentation corpus;
• and
• determining the high-resource candidate word segment corresponding to the group of first word segmentation language data with a highest frequency index as the second word segmentation language data.

A non-volatile computer-readable storage medium containing one or more computer-executable instructions, wherein the one or more computer-executable instructions, when loaded and run by a computer processor, cause the computer processor to perform a method for segmenting words based on cross-language data augmentation, wherein the method comprises:

• acquiring a plurality of groups of high-resource language (HRL) data, and acquiring a plurality of groups of first word segmentation language data by processing the plurality of groups of HRL data, wherein the plurality of groups of first word segmentation language data constitute a first word segmentation corpus;
• acquiring a plurality of groups of low-resource language (LRL) data, acquiring a plurality of candidate word segments by processing the plurality of groups of LRL data;
• selecting second word segmentation language data from the plurality of candidate word segments based on a matching degree between each of the candidate word segments and the first word segmentation corpus;
• acquiring a word segmentation model by training based on the second word segmentation language data, and outputting a plurality of candidate word segmentation results by inputting a plurality of groups of to-be-segmented data into the word segmentation model;
• and
• selecting the candidate word segmentation result with a highest matching degree as a word segmentation result based on a matching degree between each of the candidate word segmentation results and the first word segmentation corpus;
• wherein selecting the second word segmentation language data from the plurality of candidate word segments based on the matching degree between each of the candidate word segments and the first word segmentation corpus comprises:
• translating the plurality of candidate word segments into a plurality of high-resource candidate word segments corresponding to the HRL data through a translation model, wherein each of the high-resource candidate word segments corresponds to a translation probability assigned by the translation model;
• acquiring an occurrence frequency of a group of first word segmentation language data consistent with the high-resource candidate word segment in the first word segmentation corpus;
• calculating a frequency index by adding the translation probability of the high-resource candidate word segment and the occurrence frequency of the group of first word segmentation language data consistent with the high-resource candidate word segment in the first word segmentation corpus;
• and
• determining the high-resource candidate word segment corresponding to the group of first word segmentation language data with a highest frequency index as the second word segmentation language data.