Speech translation device, speech translation method, and recording medium therefor

A speech translation device, comprising:

• a memory configured to store a program;
• and
• a processor configured to execute the program and control the speech translation device to function as:
• a first beamformer which calculates first beamformer output which is a signal resulting from processing a speech signal obtained by a microphone array to direct a directivity for picking up sound in a first direction in which a first user is located;
• a second beamformer which calculates second beamformer output which is a speech signal resulting from processing the speech signal obtained by the microphone array to direct the directivity for picking up sound in a second direction different from the first direction, the second direction being a direction in which a second user different from the first user is located;
• a direction designator which designates, according to a user operation made by one of the first user and the second user, one output among output from the first beamformer and output from the second beamformer;
• a first recognizer which recognizes content indicated by the first beamformer output as first content in a first language by performing, on the first beamformer output, recognition processing in the first language when the one output designated by the direction designator is a first beam formed by the first beamformer;
• a first translator which translates the first content recognized by the first recognizer into a second language;
• a second recognizer which recognizes content indicated by the second beamformer output as second content in the second language by performing, on the second beamformer output, recognition processing in the second language when the one output designated by the direction designator is a second beam formed by the second beamformer;
• a second translator which translates the second content recognized by the second recognizer into the first language;
• a signal-to-noise (SN) ratio calculator which calculates an SN ratio, using the one output designated by the direction designator as a signal component in the SN ratio, and the other output not designated by the direction designator among the output from the first beamformer and the output from the second beamformer, as a noise component in the SN ratio;
• and
• a display determiner which determines whether a user has made an erroneous operation, based on designation by the direction designator, a magnitude of the output from the first beamformer, and a magnitude of the output from the second beamformer, and determines, when the display determiner determines that the user has made an erroneous operation, a first speaking instruction for correcting the erroneous operation, the first speaking instruction being to be notified to the user, the user being the one of the first user and the second user, wherein
• when the SN ratio is below a threshold, the display determiner determines that recognition of the one output designated by the direction designator is difficult, and determines an action for increasing the SN ratio to at least the threshold, as a second speaking instruction to be notified to the user,
• when the SN ratio is below the threshold and the action determined by the display determiner as the second speaking instruction instructs the user to move closer to the microphone array and speak, the display determiner switches, for input to one of the first recognizer and the second recognizer which is to receive the one output designated by the direction designator, from the one output to output from the microphone array, and causes the speech signal obtained by the microphone array to be input to the one of the first recognizer and the second recognizer, and
• the speech translation device further comprises a display which displays, in a display area, one of the output from the first translator, the output from the second translator, the first speaking instruction determined by the display determiner, and the second speaking instruction determined by the display determiner, according to a result of determination by the display determiner.

A speech translation method, comprising:

• (a) calculating first beamformer output which is a signal resulting from processing a speech signal obtained by a microphone array to direct a directivity for picking up sound in a first direction in which a first user is located;
• (b) calculating second beamformer output which is a signal resulting from processing the speech signal obtained by the microphone array to direct the directivity for picking up sound in a second direction different from the first direction, the second direction being a direction in which a second user different from the first user is located;
• (c) designating, according to a user operation made by one of the first user and the second user, one output among output in (a) and output in (b);
• (d) recognizing content indicated by the first beamformer output as first content in a first language by performing, on the first beamformer output, recognition processing in the first language when the one output designated in (c) is a first beam formed in (a);
• (e) translating the first content recognized in (d) into a second language;
• (f) recognizing content indicated by the second beamformer output as second content in the second language by performing, on the second beamformer output, recognition processing in the second language when the one output designated in (c) is a second beam formed in (b);
• (g) translating the second content recognized in (f) into the first language;
• (h) calculating a signal-to-noise (SN) ratio, using the one output designated in (c) as a signal component in the SN ratio, and the other output not designated in (c) among the output in (a) and the output in (b), as a noise component in the SN ratio;
• (i) determining, using the SN ratio calculated in (h), whether recognition of the one output designated in (c) is difficult and determining, when the recognition is determined to be difficult, a speaking instruction for overcoming difficulty of the recognition, the speaking instruction being to be notified to a user, the user being the one of the first user and the second user;
• and
• (j) displaying, in a display area, one of output in (e), output in (g), and the speaking instruction determined in (i), wherein
• when the SN ratio is below a threshold, the recognition is determined to be difficult, and an action for increasing the SN ratio to at least the threshold is determined as the speaking instruction, and
• when the SN ratio is below the threshold and the action determined as the speaking instruction instructs the user to move closer to the microphone array and speak, the method further comprises switching from the one output designated in (c) to output from the microphone array, and causing the speech signal obtained by the microphone array to be input as the first content recognized in (d) or the second content recognized in (f).

A non-transitory computer-readable recording medium having a program recorded thereon for causing a computer to execute:

• (a) calculating first beamformer output which is a signal resulting from processing a speech signal obtained by a microphone array to direct a directivity for picking up sound in a first direction in which a first user is located;
• (b) calculating second beamformer output which is a signal resulting from processing the speech signal obtained by the microphone array to direct the directivity for picking up sound in a second direction different from the first direction, the second direction being a direction in which a second user different from the first user is located;
• (c) designating, according to a user operation made by one of the first user and the second user, one output among output in (a) and output in (b);
• (d) recognizing content indicated by the first beamformer output as first content in a first language by performing, on the first beamformer output, recognition processing in the first language when the one output designated in (c) is a first beam formed in (a);
• (e) translating the first content recognized in (d) into a second language;
• (f) recognizing content indicated by the second beamformer output as second content in the second language by performing, on the second beamformer output, recognition processing in the second language when the one output designated in (c) is a second beam formed in (b);
• (g) translating the second content recognized in (f) into the first language;
• (h) calculating a signal-to-noise (SN) ratio, using the one output designated in (c) as a signal component in the SN ratio, and the other output not designated in (c) among the output in (a) and the output in (b), as a noise component in the SN ratio;
• (i) determining, using the SN ratio calculated in (h), whether recognition of the one output designated in (c) is difficult and determining, when the recognition is determined to be difficult, a speaking instruction for overcoming difficulty of the recognition, the speaking instruction being to be notified to a user, the user being the one of the first user and the second user;
• and
• (j) displaying, in a display area, one of output in (e), output in (g), and the speaking instruction determined in (i), wherein
• when the SN ratio is below a threshold, the recognition is determined to be difficult, and an action for increasing the SN ratio to at least the threshold is determined as the speaking instruction, and
• when the SN ratio is below the threshold and the action determined as the speaking instruction instructs the user to move closer to the microphone array and speak, the computer further executes switching from the one output designated in (c) to output from the microphone array, and causing the speech signal obtained by the microphone array to be input as the first content recognized in (d) or the second content recognized in (f).