Terminal device

A terminal device comprising:

• a wearable object to be mounted on a head of a user;
• a display apparatus provided in the wearable object to display a first screen;
• a mobile terminal that is separate from the wearable object and is equipped with a display unit;
• and
• an image capturing apparatus provided in the wearable object to capture an image of a finger or a specific input instructing tool when the user performs an operation on a second screen with the finger or the input instructing tool, and to output image data of the captured image to the mobile terminal wirelessly or wired,
• wherein the mobile terminal is connected to the display apparatus wirelessly or wired, and has a function of controlling the display apparatus to display on the display apparatus as the first screen a screen displayed on the display unit of the mobile terminal, a function of displaying a setting screen on the display unit of the mobile terminal, the setting screen having a plurality of buttons that control display settings, wherein activating one of the buttons causes a part of the screen displayed on the display unit of the mobile terminal to be displayed as the first screen, and activating another of the buttons causes a screen with an enlarged character in the screen displayed on the display unit of the mobile terminal to be displayed as the first screen, and the mobile terminal has a function of controlling, for the setting made by the user through the setting screen displayed on the display unit of the mobile terminal, the display apparatus to display on the display apparatus the first screen according to the setting made by the user through the setting screen, and
• the mobile terminal includes:
• a storage unit configured to store various types of data including data on the first screen;
• an operation determination unit configured to, when the image capturing apparatus captures the image of the finger or the input instructing tool with which the user performs an operation on the second screen, determine what content of the operation is performed with the finger or the input instructing tool among the various types of operations, based on the image data of the captured image;
• a position data generation unit configured to, when the image capturing apparatus captures the image of the finger or the input instructing tool with which the user performs an operation on the second screen, generate position data of the finger or the input instructing tool within an image capture range that is a range in which the image capturing apparatus is able to capture an image, based on the image data of the captured image;
• a reference data generation unit configured to, when the user performs an operation at one or more predetermined positions on the second screen with the finger or the input instructing tool, generate data on the second screen for identifying a position and a size of the second screen and store the generated data as reference data in the storage unit, by using the position data of the finger or the input instructing tool generated by the position data generation unit based on the image data for which the operation determination unit determines that the operation performed at each of the predetermined positions is a predetermined operation;
• and
• an input control unit configured to, when the user performs an operation on the second screen with the finger or the input instructing tool, recognize a content of an input instruction corresponding to the operation performed with the finger or the input instructing tool, by identifying a range of the second screen within the image capture range and retrieving a position where the operation is performed within the identified range of the second screen with the finger or the input instructing tool, based on data on the content of the operation performed with the finger or the input instructing tool, obtained as determined by the operation determination unit, the position data of the finger or the input instructing tool generated by the position data generation unit, the reference data on the second screen stored in the storage unit, and the data on the first screen corresponding to the second screen stored in the storage unit;
• and control a screen to be displayed on the display unit and the first screen to be displayed on the display apparatus, according to the recognized content of the input instruction, wherein
• the first screen is a screen that is projected or displayed on the display apparatus, and
• the second screen is a virtual screen that appears to be floating in midair to the user wearing the wearable object when the display apparatus projects or displays the first screen.