Intelligent recommendation of guidance instructions

A system for intelligent recommendation of guidance instructions, comprising:

• one or more computers with executable instructions that when executed cause the system to:
• execute machine learning logic to train a neural network to recognize and classify a plurality of items in three-dimensional (3D) space using real-time image input of the plurality of items from one or more sensors of one or more Internet of Things (IoT) computing devices, wherein training the neural network includes training the neural network to match an item of the plurality of items with a known item in a known library of items comprised within a database according to a pixel-by-pixel feature extraction operation performed on the real-time image input, and wherein the trained neural network identifies the item using the pixel-by-pixel feature extraction operation by dividing
• the real-time image input into regions having predicted boundaries and weighting the predicted boundaries by one or more predicted probabilities that information contained within the predicted boundaries comprises at least a portion of the item;
• in conjunction with identifying the item, execute the machine learning logic to perform a visual surveillance and monitoring operation using the trained neural network to recognize a user is performing a selected task on the item according to movement information obtained in the real-time image input, wherein the recognizing is performed to identify the selected task such that the selected task is determined only after identifying the item and observing the user performing an activity with the item;
• provide one or more guidance instructions for assisting with performing the selected task according to contextual factors identified while observing the user performing the activity with the item and associated with identifying the selected task, wherein providing the one or more guidance instructions includes determining, once the user has commenced the selected task, that the user is unable to continue performing the selected task as identified by the movement information obtained in the real-time image input, and retrieving additional guidance instructions from one or more ranked sources according to the determination;
• and
• cognitively guide the user to perform the selected task using at least one of the one or more guidance instructions and the additional guidance instructions.

A computer program product for intelligent recommendation of guidance instructions by a processor, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

• an executable portion that executes machine learning logic to train a neural network to recognize and classify a plurality of items in three-dimensional (3D) space using real-time image input of the plurality of items from one or more sensors of one or more Internet of Things (IoT) computing devices, wherein training the neural network includes training the neural network to match an item of the plurality of items with a known item in a known library of items comprised within a database according to a pixel-by-pixel feature extraction operation performed on the real-time image input, and wherein the trained neural network identifies the item using the pixel-by-pixel feature extraction operation by dividing
• the real-time image input into regions having predicted boundaries and weighting the predicted boundaries by one or more predicted probabilities that information contained within the predicted boundaries comprises at least a portion of the item;
• an executable portion that, in conjunction with identifying the item, executes the machine learning logic to perform a visual surveillance and monitoring operation using the trained neural network to recognize a user is performing a selected task on the item according to movement information obtained in the real-time image input, wherein the recognizing is performed to identify the selected task such that the selected task is determined only after identifying the item and observing the user performing an activity with the item;
• an executable portion that provides one or more guidance instructions for assisting with performing the selected task according to contextual factors identified while observing the user performing the activity with the item and associated with identifying the selected task, wherein providing the one or more guidance instructions includes determining, once the user has commenced the selected task, that the user is unable to continue performing the selected task as identified by the movement information obtained in the real-time image input, and retrieving additional guidance instructions from one or more ranked sources according to the determination;
• and
• an executable portion that cognitively guides the user to perform the selected task using at least one of the one or more guidance instructions and the additional guidance instructions.