# US 8,543,376 B2 · Apparatus and method for decoding using joint tokenization and translation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,543,376 B2 - Document ID: US8543376B2 - Status: Granted - Kind: B2 - Application number: 13/813,463 - Filed: 2011-05-25 - Published: 2013-05-23 - Granted: 2013-09-24 - Earliest claimed: 2010-08-23 - Assignees: SK Planet Co., Ltd. - Applicants: Young Sook Hwang; Sang-Bum Kim; Chang Hao Yin; Xinyan Xiao; Yang Liu; Qun Liu; Shouxun Lin - Inventors: Young Sook Hwang; Sang-Bum Kim; Chang Hao Yin; Xinyan Xiao; Yang Liu; Qun Liu; Shouxun Lin - Web page: https://patentgrep.com/patent/US8543376B2 ## CPC - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/44: Statistical methods, e.g. probability models - G06F 40/53: Processing of non-Latin text (kana-to-kanji conversion G06F40/129; vowelisation G06F40/232) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Disclosed are a joint decoding apparatus and a joint decoding method that joins a tokenization process and a translation process. More particularly, the present disclosure can generate all available candidate tokens, reduce translation errors, and obtain an optimal translation result by jointly conducting a decoding by simultaneously conducting the tokenization process and the translation process for an input character sequence while decoding. ## Claims ### Claim 1 A joint decoding apparatus that joins a tokenization process and a translation process, the joint decoding apparatus comprising: a candidate token generation unit configured to generate a plurality of candidate tokens by applying a maximum entropy model to an input character sequence; a probability calculation unit configured to calculate a tokenization probability of each of the generated candidate tokens by using a language model; an out of vocabulary (OOV) processing unit configured to process an OOV by using a word count and discount information for the OOV among the generated candidate tokens; and a translation unit configured to generate a corresponding phrase corresponding to the input character sequence according to a translation model by using calculated probability values of the candidate tokens and an OOV processing result. ### Claim 2 (depends on 1) The joint decoding apparatus of claim 1, further comprising: a translation model database configured to store a translation model trained from tokenized data of a parallel corpus; and a language model database configured to store a language model trained from a monolingual corpus. ### Claim 3 (depends on 2) The joint decoding apparatus of claim 2, wherein the translation model database stores a character string-based translation model. ### Claim 4 (depends on 1) The joint decoding apparatus of claim 1, wherein the candidate token generation unit tags at least one of a character corresponding to a beginning of a word, a character located in a middle position of the word, a character corresponding to an end of the word in the input character sequence, and a word consisting of a single character in terms of a word generation. ### Claim 5 (depends on 1) The joint decoding apparatus of claim 1, wherein the probability calculation unit calculates a probability by combining the tokenization probability of each of the generated candidate tokens and an N-gram language model. ### Claim 6 (depends on 1) The joint decoding apparatus of claim 1, wherein the OOV processing unit controls a word count for the OOV among the generated candidate tokens. ### Claim 7 (depends on 1) The joint decoding apparatus of claim 1, wherein the translation unit generates a translated sentence corresponding to the input character sequence by applying the calculated probability values of the candidate tokens and the OOV processing result to a log-linear model. ### Claim 8 A joint decoding method of joining a tokenization process and a translation process, the joint decoding method comprising: generating a candidate token by generating a plurality of candidate tokens by applying a maximum entropy model to an input character sequence; calculating a probability by combining a tokenization probability of each of the generated candidate tokens and a language model; processing out of vocabularly (OOV) by using a word count and discount information for an OOV among the generated candidate tokens; and conducting a translation that generates a corresponding phrase corresponding to the input character sequence by using calculated probability values of the candidate tokens and an OOV processing result. ### Claim 9 (depends on 8) The joint decoding method of claim 8, wherein the translation model is a character string based translation model. ### Claim 10 (depends on 8) The joint decoding method of claim 8, wherein the generating of the candidate token comprises tagging at least one of a character corresponding to a beginning of a word, a character located in a middle position of the word, a character corresponding to an end of the word in the input character sequence, and a word consisting of a single character in terms of a word generation. ### Claim 11 (depends on 8) The joint decoding method of claim 8, wherein the calculating of the probability calculates a probability by combining the tokenization probability of each of the generated candidate tokens and an N-gram language model. ### Claim 12 (depends on 8) The joint decoding method of claim 8, wherein the processing of the OOV comprises controlling a word count for the OOV among the generated candidate tokens. ### Claim 13 (depends on 8) The joint decoding method of claim 8, wherein the conducting of the translation generates a translated sentence corresponding to the input character sequence by applying the calculated probability values of the candidate tokens and the 00V processing result to a log-linear model. ### Claim 14 (depends on 8) A non-transitory computer readable recording medium for recording a program for executing the method according to claims 8. ### Claim 15 (depends on 9) A non-transitory computer readable recording medium for recording a program for executing the method according to claim 9. ### Claim 16 (depends on 10) A non-transitory computer readable recording medium for recording a program for executing the method according to claim 10. ### Claim 17 (depends on 11) A non-transitory computer readable recording medium for recording a program for executing the method according to claim 11. ### Claim 18 (depends on 12) A non-transitory computer readable recording medium for recording a program for executing the method according to claim 12. ### Claim 19 (depends on 13) A non-transitory computer readable recording medium for recording a program for executing the method according to claim 13.