Graph representations for identifying a next word

A system for approximating a softmax layer, the system comprising:

• an electronic processor:
• a memory including instructions that configured the electronic processor to:
• construct a small world graph comprising a plurality of nodes for a vocabulary of a natural language model (NLM);
• transform a context vector using an inner product preserving transformation;
• search the small world graph using the transformed context vector to identify a top-K hypothesis for a next word;
• determine a distance from the context vector for each of the top-K hypothesis;
• transform the distance to an original inner product space;
• compute a softmax distribution for the softmax layer over the inner product space of the top-K hypothesis for the next word;
• and
• identify the next word based on the top-K hypothesis.

A computer-readable storage medium storing computer-executable instructions for approximating a softmax layer, the stored instructions comprising:

• instructions to construct a small world graph comprising a plurality of nodes for a vocabulary of a natural language model (NLM);
• instructions to transform a context vector using an inner product preserving transformation;
• instructions to search the small world graph using the transformed context vector to identify a top-K hypothesis for a next word;
• instructions to determine a distance from the context vector for each of the top-K hypothesis;
• instructions to transform the distance to an original inner product space;
• instructions to compute a softmax distribution for the softmax layer over the inner product space of the top-K hypothesis for the next word;
• and
• instructions to identify the next word based on the top-K hypothesis.