Cluster-based word vector processing method, device, and apparatus

A cluster comprising a server cluster and a worker-computer cluster including a plurality of worker computers, wherein each of the worker computers is configured to perform operations including:

• obtaining a word and at least one context word of the word that are extracted from a respective portion of a corpus through distributed reading;
• creating a word pair according to the respective portion of the corpus, wherein the word pair comprises the word and a context word of the at least one context word;
• obtaining word vectors for the word and the at least one context word;
• training the word vectors according to one or more of a designated loss function, a designated negative sample word, or the created word pair to generate training results, by:
• traversing the corpus to obtain current words in the corpus;
• and
• calculating a gradient according to one or more of the designated loss function, the designated negative sample word, or the created word pair, and iteratively updating word vectors of the current words and context words of the current words according to the following formulas:
• {right arrow over (w)}i,t+1={right arrow over (w)}i,t+g{right arrow over (c)}i,t,w∈Bi,k and
• {right arrow over (c)}i,t+1={right arrow over (c)}i,t+g{right arrow over (w)}i,t,c∈Γ(w),
• wherein
• g
• =
• α
• ⁡
• (
• y
• -
• σ
• ⁡
• (
• w
• →
• ·
• c
• →
• )
• )
• ,
• y
• =
• {
• 1
• ,
• {
• w
• ,
• c
• }
• 0
• ,
• {
• w
• ,
• c
• ′
• }
• ,
• w represents a current word, c represents a context word of w, c′ represents a negative sample word, {right arrow over (w)} represents a word vector of w, {right arrow over (c)} represents a word vector of c, {right arrow over (w)}i,t and {right arrow over (c)}i,t represent a ith update on an ith worker computer, Bi,k represents a kth corpus on the ith worker computer, Γ(w) represents a context word set of w, α represents a learning rate, and σ is a sigmoid function;
• and
• forwarding to the server cluster the training results to enable the server cluster to update the word vectors for the word and the at least one context word based on the training results.

A non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform, at a cluster comprising a worker computer cluster including a plurality of worker computers and a server cluster, operations at each of worker computers:

• obtaining a word and at least one context word of the word that are extracted from a respective portion of a corpus through distributed reading;
• creating a word pair according to the respective portion of the corpus, wherein the word pair comprises the word and a context word of the at least one context word;
• obtaining word vectors for the word and the at least one context word;
• training the word vectors according to one or more of a designated loss function, a designated negative sample word, or the created word pair to generate training results, by:
• traversing the corpus to obtain current words in the corpus;
• and
• calculating a gradient according to one or more of the designated loss function, the designated negative sample word, or the created word pair, and iteratively updating word vectors of the current words and context words of the current words according to the following formulas:
• {right arrow over (w)}i,t+1={right arrow over (w)}i,t+g{right arrow over (c)}i,t,w∈Bi,k and
• {right arrow over (c)}i,t+1={right arrow over (c)}i,t+g{right arrow over (w)}i,t,c∈Γ(w),
• wherein
• g
• =
• α
• ⁡
• (
• y
• -
• σ
• ⁡
• (
• w
• →
• ·
• c
• →
• )
• )
• ,
• y
• =
• {
• 1
• ,
• {
• w
• ,
• c
• }
• 0
• ,
• {
• w
• ,
• c
• ′
• }
• ,
• w represents a current word, c represents a context word of w, c′ represents a negative sample word, {right arrow over (w)} represents a word vector of w, {right arrow over (c)} represents a word vector of c, {right arrow over (w)}i,t and {right arrow over (c)}i,t represent a ith update on an ith worker computer, Bi,k represents a kth corpus on the ith worker computer, Γ(w) represents a context word set of w, α represents a learning rate, and σ is a sigmoid function;
• and
• forwarding to the server cluster the training results to enable the server cluster to update the word vectors for the word and the at least one context word based on the training results.