# US 12,118,897 B2 · Augmented reality tutorial generation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,897 B2 - Document ID: US12118897B2 - Status: Granted - Kind: B2 - Application number: 16/522,223 - Filed: 2019-07-25 - Published: 2021-01-28 - Granted: 2024-10-15 - Earliest claimed: 2019-07-25 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Philipp Schume; Shikhar Kwatra; Craig M. Trim; Zachary A. Silverstein - Web page: https://patentgrep.com/patent/US12118897B2 ## CPC - G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip - G09B 19/00: Teaching not covered by other main groups of this subclass (teaching or practice apparatus for gun-aiming or gun-laying F41G3/26) - G06F 16/2365: {Ensuring data consistency and integrity} - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 20/00: Machine learning - G06T 11/00: Two-dimensional [2D] image generation - G09B 19/003: {Repetitive work cycles; Sequence of movements} - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method, system and computer program product includes identifying, by one or more processors of a computer system, a relevant tutorial to a user using a machine learning ontology tree, and providing, by the one or more processors of the computer system, the identified relevant tutorial to an augmented reality device of the user. ## Claims ### Claim 1 A method comprising: capturing, by the one or more processors of the computer system, a user interaction with at least one surrounding object; determining, by the one or more processors of the computer system, a context of the user interaction based on machine learning technology, wherein the determining the context includes determining whether a user is actively taking steps associated with failure or frustration, wherein the context includes a contextual situation, including visual recognition of a location or environment; and feeding, by the one or more processors of the computer system, a machine learning ontology tree the determined context of user interaction; identifying, by one or more processors of a computer system, a relevant tutorial to a user using the machine learning ontology tree in real time to match the determined context of the user interaction based on a frustration level of the user, wherein the relevant tutorial has been updated by the machine learning technology based on a previous user interaction to determine differences in structure between structural components to be interacted with by the user; providing, by the one or more processors of the computer system, the identified relevant tutorial to an augmented reality device of the user; overlaying, by the one or more processors of the computer system, the detected relevant tutorial on the augmented reality device of the user; and highlighting, by the one or more processors of the computer system, the structural components to be interacted with by the user and the differences in structure during the detected relevant tutorial on the augmented reality device of the user. ### Claim 2 (depends on 1) The method of claim 1, wherein the capturing further comprises: receiving, by the one or more processors of the computer system, data related to the user interaction from at least one internet-of-things device, wherein the data includes inputs gathered by the internet-of-things device associated with determining user gestures. ### Claim 3 (depends on 1) The method of claim 1, wherein the identifying further comprises: determining, by the one or more processors of the computer system using the machine learning ontology tree based the determined context, which tutorial of a plurality of tutorials is the relevant tutorial. ### Claim 4 (depends on 1) The method of claim 1, further comprising: receiving, by the one or more processors of the computer system, an inaccuracy indication that the provided identified relevant tutorial is an incorrect match; and identifying, by the one or more processors of the computer system, a next relevant tutorial. ### Claim 5 (depends on 4) The method of claim 4, further comprising; storing, by the one or more processors of the computer system, the received inaccuracy indication as feedback; and using, by the one or more processors of the computer system, the received and stored inaccuracy indication to retrain the machine learning ontology tree. ### Claim 6 (depends on 1) The method of claim 1, wherein the determining the context includes determining, by the one or more processors of a computer system, that a frustration threshold is violated. ### Claim 7 A computer system, comprising: one or more processors; one or more memory devices coupled to the one or more processors; and one or more computer readable storage devices coupled to the one or more processors, wherein the one or more storage devices contain program code executable by the one or more processors via the one or more memory devices to implement a method, the method comprising: capturing, by the one or more processors of the computer system, a user interaction with at least one surrounding object; determining, by the one or more processors of the computer system, a context of the user interaction based machine learning technology, wherein the determining the context includes determining whether a user is actively taking steps associated with failure or frustration, wherein the context includes a contextual situation, including visual recognition of a location or environment; feeding, by the one or more processors of the computer system, a machine learning ontology tree the determined context of user interaction; identifying, by the one or more processors of the computer system, a relevant tutorial to a user using the machine learning ontology tree in real time to match the determined context of the user interaction based on a frustration level of the user, wherein the relevant tutorial has been updated by the machine learning technology based on a previous user interaction to determine differences in structure between structural components to be interacted with by the user; providing, by the one or more processors of the computer system, the identified relevant tutorial to an augmented reality device of the user; overlaying, by the one or more processors of the computer system, the detected relevant tutorial on the augmented reality device of the user; and highlighting, by the one or more processors of the computer system, the structural components to be interacted with by the user and the differences in structure during the detected relevant tutorial on the augmented reality device of the user. ### Claim 8 (depends on 7) The computer system of claim 7, wherein the capturing further comprises: receiving, by the one or more processors of the computer system, data related to the user interaction from at least one internet-of-things device, wherein the data includes inputs gathered by the internet-of-things device associated with determining user gestures. ### Claim 9 (depends on 7) The computer system of claim 7, wherein the identifying further comprises: determining, by the one or more processors of the computer system using the machine learning ontology tree based on the determined context, which tutorial of a plurality of tutorials is the relevant tutorial. ### Claim 10 (depends on 7) The computer system of claim 7, the method further comprising: receiving, by the one or more processors of the computer system, an inaccuracy indication that the provided identified relevant tutorial is an incorrect match; and identifying, by the one or more processors of the computer system, a next relevant tutorial. ### Claim 11 (depends on 10) The computer system of claim 10, the method further comprising; storing, by the one or more processors of the computer system, the received inaccuracy indication as feedback; and using, by the one or more processors of the computer system, the received and stored inaccuracy indication to retrain the machine learning ontology tree. ### Claim 12 (depends on 7) The computer system of claim 7, wherein the determining the context includes determining, by the one or more processors of a computer system, that a frustration threshold is violated. ### Claim 13 A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, the computer readable program code comprising an algorithm that when executed by one or more processors of a computer system implements a method, the method comprising: capturing, by the one or more processors of the computer system, a user interaction with at least one surrounding object; determining, by the one or more processors of the computer system, a context of the user interaction based on machine learning technology, wherein the determining the context includes determining whether a user is actively taking steps associated with failure or frustration, wherein the context includes a contextual situation, including visual recognition of a location or environment; feeding, by the one or more processors of the computer system, a machine learning ontology tree the determined context of user interaction; identifying, by the one or more processors of the computer system, a relevant tutorial to a user using the machine learning ontology tree in real time to match the determined context of the user interaction based on a frustration level of the user, wherein the relevant tutorial has been updated by the machine learning technology based on a previous user interaction to determine differences in structure between structural components to be interacted with by the user; providing, by the one or more processors of the computer system, the identified relevant tutorial to an augmented reality device of the user; overlaying, by the one or more processors of the computer system, the detected relevant tutorial on the augmented reality device of the user; and highlighting, by the one or more processors of the computer system, the structural components to be interacted with by the user and the differences in structure during the detected relevant tutorial on the augmented reality device of the user. ### Claim 14 (depends on 13) The computer program product of claim 13, wherein the capturing further comprises: receiving, by the one or more processors of the computer system, data related to the user interaction from at least one internet-of-things device, wherein the data includes inputs gathered by the internet-of-things device associated with determining user gestures. ### Claim 15 (depends on 13) The computer program product of claim 13, wherein the identifying further comprises: determining, by the one or more processors of the computer system using the machine learning ontology tree based on the determined context, which tutorial of a plurality of tutorials is the relevant tutorial. ### Claim 16 (depends on 13) The computer program product of claim 13, the method further comprising: receiving, by the one or more processors of the computer system, an inaccuracy indication that the provided identified relevant tutorial is an incorrect match; and identifying, by the one or more processors of the computer system, a next relevant tutorial. ### Claim 17 (depends on 16) The computer program product of claim 16, the method further comprising; storing, by the one or more processors of the computer system, the received inaccuracy indication as feedback; and using, by the one or more processors of the computer system, the received and stored inaccuracy indication to retrain the machine learning ontology tree. ### Claim 18 (depends on 13) The computer program product of claim 13, wherein the determining the context includes determining, by the one or more processors of a computer system, that a frustration threshold is violated.