Information processing apparatus, information processing method, computer-readable recording medium, and information processing system

An information processing method comprising:

• generating an operation log of an operation target device which cannot generate or output the operation log, by identifying an operation content of the operation target device operated by a user, based on (i) a captured image obtained by capturing an image of the operation target device, and (ii) operation pattern information of the operation target device stored in a recording unit in advance, wherein
• the captured image includes an image of the user and the image of the operation target device,
• the operation pattern information is information in which (i) the operation content of the operation target device and (ii) image-derived information capable of being obtained from the image of the user and the image of the operation target device are associated with one another,
• the image-derived information includes at least one of (i) a pattern of a finger or a touch pen used for operation, (ii) a gesture of a person, (iii) the person rotating a switch of the operation target device, (iV) the person dragging on a touch panel of the operation target device, and (V) the person flicking on the touch panel, and
• the method further comprises:
• identifying the image-derived information by performing an image recognition process on the captured image;
• retrieving the operation pattern information using the identified image-derived information as a retrieval key;
• and then
• identifying the operation content which is associated with the identified image-derived information in the operation pattern information.

A computer-readable recording medium having stored therein an information processing program that causes a computer to execute a process of generating an operation log of an operation target device which cannot generate or output the operation log, by identifying an operation content of the operation target device operated by a user, based on (i) a captured image obtained by capturing an image of the operation target device, and (ii) operation pattern information of the operation target device stored in a recording unit in advance, wherein

• the captured image includes an image of the user and the image of the operation target device,
• the operation pattern information is information in which (i) the operation content of the operation target device and (ii) image-derived information capable of being obtained from the image of the user and the image of the operation target device are associated with one another,
• the image-derived information includes at least one of (i) a pattern of a finger or a touch pen used for operation, (ii) a gesture of a person, (iii) the person rotating a switch of the operation target device, (iv) the person dragging on a touch panel of the operation target device, and (V) the person flicking on the touch panel, and
• the information processing program causes the computer to further execute:
• identifying the image-derived information by performing an image recognition process on the captured image;
• retrieving the operation pattern information using the identified image-derived information as a retrieval key;
• and then
• identifying the operation content which is associated with the identified image-derived information in the operation pattern information.

An information processing system comprising:

• an image capturing device that captures an image of an operation target device operated by a user, and
• an information processing apparatus that generates an operation log of the operation target device which cannot generate or output the operation log, by identifying an operation content of the operation target device operated by the user, based on (i) a captured image of the operation target device captured by the image capturing device, and (ii) operation pattern information of the operation target device stored in a recording unit in advance, wherein
• the captured image includes an image of the user and the image of the operation target device,
• the operation pattern information is information in which (i) the operation content of the operation target device and (ii) image-derived information capable of being obtained from the image of the user and the image of the operation target device are associated with one another,
• the image-derived information includes at least one of (i) a pattern of a finger or a touch pen used for operation, (ii) a gesture of a person, (iii) the person rotating a switch of the operation target device, (iv) the person dragging on a touch panel of the operation target device, and (V) the person flicking on the touch panel, and
• the information processing apparatus identifies the image-derived information by performing an image recognition process on the captured image, retrieves the operation pattern information using the identified image-derived information as a retrieval key, and then identifies the operation content which is associated with the identified image-derived information in the operation pattern information.