Information terminal apparatus, information processing method, and system, configured to correct posture of a virtual object depending on changed posture information

An information processing method, comprising:

• capturing an image of a subject by an imager to generate a captured image;
• displaying the captured image on a display;
• detecting, by a posture detector, changed posture information indicating a change in posture of the imager;
• generating, based on a conversion function for projecting a preset virtual object arranged on an object coordinate which is a virtual three-dimensional space on a device coordinate which is a two-dimensional coordinate representing a display screen of the captured image, a reference object image as a result of a projection, on the device coordinate,
• generating a reference synthetic image in which the object image is synthesized with the captured image,
• storing an initial posture of the imager in response to user operation with the reference synthetic image being displayed on the display,
• recording the changed posture information indicating a change, detected by the posture detector, in posture of the imager from the initial posture in synchronization with the captured image,
• correcting a posture of the virtual object depending on the changed posture information,
• generating, based on the conversion function, an object image as a result of a projection, on the device coordinate, of the virtual object whose posture has been corrected,
• generating a synthetic image in which the captured image is synthesized with the object image,
• displaying on the display the synthetic image including the captured image synthesized with the object image of the virtual object whose posture has been corrected,
• reproducing the captured image,
• synchronizing with the captured image and reading out the changed posture information recorded,
• correcting the posture of the virtual object depending on the changed posture information read out during the reproduction,
• generating the synthetic image in which the object image is synthesized with the reproduced captured image,
• receiving operation information with respect to the object image from a user on the device coordinate,
• synchronizing with the captured image and recording the operation information,
• correcting the conversion function so that the virtual object is shifted on the object coordinate by a shift amount depending on the operation information, wherein the object image is generated based on the corrected conversion function,
• synchronizing with the captured image and reading out the operation information recorded, wherein the conversion function is being corrected depending on the operation information read out during the reproduction,
• selecting one of a mode of reflecting an operation in recording and a mode of not reflecting an operation in recording based on an operation by the user during the reproduction,
• correcting the conversion function depending on the read out operation information in a case of the mode of reflecting the operation in recording, and
• correcting the conversion function depending on current operation information input by the user in a case of the mode of not reflecting the operation in recording.

A system comprising:

• a terminal device;
• an information processing device connectable to the terminal device via a network;
• an imager configured to capture an image of a subject to generate a captured image;
• a display configured to display the captured image;
• a posture detector configured to detect changed posture information indicating a change in posture of the imager;
• processing circuitry configured to
• generate, based on a conversion function for projecting a preset virtual object arranged on an object coordinate which is a virtual three-dimensional space on a device coordinate which is a two-dimensional coordinate representing a display screen of the captured image, a reference object image as a result of a projection, on the device coordinate
• generate a reference synthetic image in which the reference object image is synthesized with the captured image;
• store an initial posture of the imager in response to a user operation with the reference synthetic image being displayed on the display;
• record the changed posture information indicating a change, detected by the posture detector, in posture of the imager from the initial posture in synchronization with the captured image;
• correct a posture of the virtual object depending on the changed posture information;
• generate, based on the conversion function, an object image as a result of a projection, on the device coordinate, of the virtual object whose posture has been corrected;
• generate a synthetic image in which the captured image is synthesized with the object image,
• display on the display the synthetic image including the captured image synthesized with the object image of the virtual object whose posture has been corrected,
• reproduce the captured image,
• synchronize with the captured image and read out the changed posture information recorded,
• correct the posture of the virtual object depending on the changed posture information read out during the reproduction,
• generate the synthetic image in which the object image is synthesized with the reproduced captured image,
• receive operation information with respect to the object image from a user on the device coordinate,
• synchronize with the captured image and record the operation information,
• correct the conversion function so that the virtual object is shifted on the object coordinate by a shift amount depending on the operation information, wherein the object image is generated based on the corrected conversion function,
• synchronize with the captured image and read out the operation information recorded, wherein the conversion function is corrected depending on the operation information read out during the reproduction,
• select one of a mode of reflecting an operation in recording and a mode of not reflecting an operation in recording based on an operation by the user during the reproduction,
• correct the conversion function depending on the read out operation information in a case of the mode of reflecting the operation in recording, and
• correct the conversion function depending on current operation information input by the user in a case of the mode of not reflecting the operation in recording.