Augmented reality language translation method and system using a situation-word database

An interpretation server, comprising:

• an analysis module that receives image information and voice information in a user's visual field from a user terminal, selects at least one speaker to be interpreted based on a face region detected in the received image information, and individually extracts the voice information of the speaker from the received voice information;
• and
• an interpretation module that generates an interpretation result according to a preset target language based on the voice information of the speaker and transmits the generated interpretation result to the terminal,
• wherein the analysis module extracts features of text, objects, distribution of multiple objects, features of the user's surrounding environment, and color from the received image information, and inputs the features to a pre-trained deep learning model to determine the situation based on the features,
• wherein the analysis module extracts words related to the situation, from a situation-word database and from text information which is recognized in the collected image information, and
• wherein the interpretation module generates the interpretation result according to the preset target language based on the words related to the situation and the voice information of the speaker.

A terminal, comprising:

• an input module that collects image information and voice information of a user's visual field;
• an analysis module that detects a face region of a speaker in the collected image information, selects at least one speaker to be interpreted based on the detected face region, and individually extracts the voice information of the speaker from the collected voice information and transmits the extracted voice information to an interpretation server;
• and
• an output module that receives an interpretation result for the voice information of the speaker from the interpretation server and outputs the interpretation result through at least one of a display and a speaker,
• wherein the analysis module infers a situation based on the collected image information, and transmits the inferred situation to the interpretation server,
• wherein the analysis module extracts features of text, objects, distribution of multiple objects, features of the user's surrounding environment, and color from the collected image information, and inputs the features to a pre-trained deep learning model to determine the situation based on the features,
• wherein the analysis module extracts words related to the situation, from a situation-word database and from text information which is recognized in the collected image information, and
• wherein the interpretation server generates the interpretation result according to a preset target language based on the words related to the situation and the voice information of the speaker.