# US 10,504,039 B2 · Short message classification for video delivery service and normalization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,039 B2 - Document ID: US10504039B2 - Status: Granted - Kind: B2 - Application number: 15/147,778 - Filed: 2016-05-05 - Published: 2017-11-09 - Granted: 2019-12-10 - Earliest claimed: 2016-05-05 - Assignees: Hulu, LLC - Applicants: Hulu, LLC - Inventors: Wei Feng; Tao Xiong; Di Wu; Zhibing Wang; Wei Wang - Web page: https://patentgrep.com/patent/US10504039B2 ## CPC - G06N 20/00: Machine learning - G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings} - G06N 5/047: Pattern matching networks; Rete networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In one embodiment, a method selects a set of accounts related to a video using a first classifier that is trained to identify the set of accounts from a plurality of accounts and selects a token used to identify the video. A second classifier is trained to identify short messages that are related to the video using a first set of short messages from the accounts related to the video and the second set of short messages including the tokens. A third set of short messages is identified that is related to the video from a plurality of short messages. The method normalizes words in the set of short messages using a mixed model including a lexicon model and a translation model. Then, the method generates a set of ratings for the set of short messages and uses the set of ratings in a service for a video delivery service. ## Claims ### Claim 1 A method comprising: identifying, by a computing device, a first set short messages that are related to a video using a second set of short messages from a set of accounts related to the video and a third set of short messages including a token that is used to identify the video, wherein at least a portion of the second set of short messages and the third set of short messages have a length restriction; identifying, by the computing device, a fourth set of short messages that are related to the video from a plurality of short messages, the plurality of short messages generated from a search for the video; normalizing, by the computing device, words in at least one or more of the first set of short messages and the fourth set of short messages using a mixed model including a lexicon model and a translation model, wherein normalizing comprises: building, by the computing device, the translation model using a lexicon of out of vocabulary words and corresponding in-vocabulary words and a corpus of words from short messages from a source, wherein the translation model is used to translate out of vocabulary words from the set of short messages to in-vocabulary words; inputting, by the computing device, a short message from the at least one or more of the first set of short messages and the fourth set of short messages into a decoder; and translating, by the computing device, a set of words from the short message using the translation model at the decoder, wherein when a word from the short message cannot be translated using the decoder, performing: generating, by the computing device, a character translation model to translate characters based on the lexicon; translating, by the computing device, the word that could not be translated by the translation model using the character translation model; and generating, by the computing device, a set of ratings for the at least one or more of the first set of short messages and the fourth set of short messages for the video. ### Claim 2 (depends on 1) The method of claim 1, further comprising: selecting the set of accounts, wherein selecting the set of accounts comprises: generating a training data set of the set of accounts; identifying features for the training data set; and classifying the set of accounts using features extracted from short messages of the set of accounts. ### Claim 3 (depends on 2) The method of claim 2, wherein selecting the set of accounts further comprises: performing a suggested accounts search that provides suggested accounts that are related to the video; and identifying accounts related to the video from at least a portion of the suggested accounts. ### Claim 4 (depends on 2) The method of claim 2, wherein selecting the set of accounts further comprises: performing a query suggestions search that provides query suggestions for accounts that are related to the video; and identifying accounts related to the video from at least a portion of query suggestions for accounts. ### Claim 5 (depends on 1) The method of claim 1, further comprising: selecting the token, wherein selecting the token comprises: selecting a group of tokens for the video; and using a frequency of use to select the token used to identify the video. ### Claim 6 (depends on 5) The method of claim 5, wherein using the frequency comprises: comparing the frequency of use for each of the group of tokens to a threshold; and selecting the token based on the comparing, wherein the token has a frequency of use that meets the threshold. ### Claim 7 (depends on 1) The method of claim 1, wherein identifying the first set of short messages comprises: selecting negative short messages that are not relevant to the video; and using the negative short messages to determine the first set of short messages. ### Claim 8 (depends on 7) The method of claim 7, wherein the negative short messages are selected as a sampling of a percentage of a total of short messages generated from the source. ### Claim 9 (depends on 1) The method of claim 1, further comprising: using the set of ratings in a service for a video delivery service. ### Claim 10 (depends on 1) The method of claim 1, further comprising: selecting the set of accounts related to the video using a first classifier that is trained to identify the set of accounts from a plurality of accounts. ### Claim 11 (depends on 10) The method of claim 10, further comprising: training a second classifier to identify the first set of short messages that are related to the video. ### Claim 12 A non-transitory computer-readable storage medium containing instructions, that when executed, control a computer system to be operable for: identifying a first set short messages that are related to a video using a second set of short messages from a set of accounts related to the video and a third set of short messages including a token that is used to identify the video, wherein at least a portion of the second set of short messages and the third set of short messages have a length restriction; identifying a fourth set of short messages that are related to the video from a plurality of short messages, the plurality of short messages generated from a search for the video; normalizing words in at least one or more of the first set of short messages and the fourth set of short messages using a mixed model including a lexicon model and a translation model, wherein normalizing comprises: building the translation model using a lexicon of out of vocabulary words and corresponding in-vocabulary words and a corpus of words from short messages from a source, wherein the translation model is used to translate out of vocabulary words from the set of short messages to in-vocabulary words; inputting a short message from the at least one or more of the first set of short messages and the fourth set of short messages into a decoder; and translating a set of words from the short message using the translation model at the decoder, wherein when a word from the short message cannot be translated using the decoder, performing: generating a character translation model to translate characters based on the lexicon; translating the word that could not be translated by the translation model using the character translation model; and generating a set of ratings for the at least one or more of the first set of short messages and the fourth set of short messages for the video. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, further operable for: selecting the set of accounts, wherein selecting the set of accounts comprises: generating a training data set of the set of accounts; identifying features for the training data set; and classifying the set of accounts using features extracted from short messages of the set of accounts. ### Claim 14 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein selecting the set of accounts further comprises: performing a suggested accounts search that provides suggested accounts that are related to the video; identifying accounts related to the video from at least a portion of the suggested accounts; performing a query suggestions search that provides query suggestions for accounts that are related to the video; and identifying accounts related to the video from at least a portion of query suggestions for accounts. ### Claim 15 (depends on 12) The non-transitory computer-readable storage medium of claim 12, further operable for: selecting the token, wherein selecting the token comprises: selecting a group of tokens for the video; and using a frequency of use to select the token used to identify the video. ### Claim 16 (depends on 12) The non-transitory computer-readable storage medium of claim 12, further operable for: using the set of ratings in a service for a video delivery service. ### Claim 17 (depends on 12) The non-transitory computer-readable storage medium of claim 12, further operable for: selecting the set of accounts related to the video using a first classifier that is trained to identify the set of accounts from a plurality of accounts. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, further operable for: training a second classifier to identify the first set of short messages that are related to the video. ### Claim 19 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein identifying the first set of short messages comprises: selecting negative short messages that are not relevant to the video; and using the negative short messages to determine the first set of short messages. ### Claim 20 An apparatus comprising: one or more computer processors; and a non-transitory computer-readable storage medium comprising instructions, that when executed, control the one or more computer processors to be configured for: identifying a first set short messages that are related to a video using a second set of short messages from a set of accounts related to the video and a third set of short messages including a token that is used to identify the video, wherein at least a portion of the second set of short messages and the third set of short messages have a length restriction; identifying a fourth set of short messages that are related to the video from a plurality of short messages, the plurality of short messages generated from a search for the video; normalizing words in at least one or more of the first set of short messages and the fourth set of short messages using a mixed model including a lexicon model and a translation model, wherein normalizing comprises: building the translation model using a lexicon of out of vocabulary words and corresponding in-vocabulary words and a corpus of words from short messages from the source, wherein the translation model is used to translate out of vocabulary words from the set of short messages to in-vocabulary words; inputting a short message from the at least one or more of the first set of short messages and the fourth set of short messages into a decoder; and translating a set of words from the short message using the translation model at the decoder, wherein when a word from the short message cannot be translated using the decoder, performing: generating a character translation model to translate characters based on the lexicon; and translating the word that could not be translated by the translation model using the character translation model; and generating a set of ratings for the at least one or more of the first set of short messages and the fourth set of short messages for the video.