# US 12,182,950 B2 · Remote expert method and system utilizing quantitative quality assurance in mixed reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,950 B2 - Document ID: US12182950B2 - Status: Granted - Kind: B2 - Application number: 18/111,458 - Filed: 2023-02-17 - Published: 2023-08-17 - Granted: 2024-12-31 - Earliest claimed: 2022-02-17 - Assignees: The United States of America as represented by the Secretary of the Navy - Applicants: The United States of America, AS Represented By the Secretary of the Navy - Inventors: Larry Clay Greunke - Web page: https://patentgrep.com/patent/US12182950B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 16/24573: {using data annotations, e.g. user-defined metadata} - G06T 13/20: Three-dimensional [3D] animation - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip - G09B 19/003: {Repetitive work cycles; Sequence of movements} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2210/52: Parallel processing - G06T 2219/004: Annotating, labelling - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract This disclosure and exemplary embodiments described herein provide a Parallel Content Authoring Method and Tool for Procedural Guidance, and a Remote Expert Method and System Utilizing Quantitative Quality Assurance in Mixed Reality. The implementation described herein is related to the generation of content/instruction set that can be viewed in different modalities, including but not limited to mixed reality, VR, audio text, however it is to be understood that the scope of this disclosure is not limited to such application. ## Claims ### Claim 1 A method for remote expert utilizing quantitative quality assurance in a mixed reality system, the method comprising: (a) presenting a 3D representation of target equipment to an expert user, wherein a synchronized 3D representation of the target equipment is presented to a novice user; (b) receiving a part selection and a requested action for an equipment part of the target equipment from the expert user; (c) parallel authoring, by a sensor, wherein the sensor creates a data structure based at least in part on information collected from a mixed reality environment, wherein the data structure is paralleled to author multiple outputs, the multiple outputs comprising position, corrosion type, and job control number; (e) presenting the novice user's performance of the requested action on the equipment part to the expert user, wherein the presentation of the novice user's performance comprises various sensor data collected at the novice user's physical environment location; (f) validating, by a direct measures feedback, the novice user's performance of the requested action; (g) generating a confidence assessment of the novice user's performance of the requested action, wherein the confidence assessment specifies whether the expert user rejects the requested action as failing performance requirements, based at least in part on an indirect measures or the direct measures feedback; (h) if the requested action is rejected as failing performance requirements, repeating steps (c)-(e) until the expert accepts the requested action as satisfying the performance requirements; and (i) repeating steps (a)-(h) until a plurality of requested actions are accepted by the expert as satisfying the performance requirements, wherein the plurality of requested actions complete a maintenance task. ### Claim 2 (depends on 1) The method of claim 1, wherein the expert has a digital twin of the target equipment that the novice user is manipulating, whereas the novice user manipulates physical characteristics of the target equipment, the manipulations are reproduced digitally in the expert user's digital twin model. ### Claim 3 (depends on 1) The method of claim 1, wherein the requested action can be sent, executed, and signed off asynchronously. ### Claim 4 (depends on 1) The method of claim 1, wherein the expert user can be a human or a computing device with artificial intelligence. ### Claim 5 (depends on 1) The method of claim 1, wherein a first cloud mesh of the 3D presentation of the target equipment is periodically aligned with a second cloud mesh of the synchronized 3D representation. ### Claim 6 (depends on 1) The method of claim 1, wherein the requested action comprises a description of the requested action. ### Claim 7 (depends on 1) The method of claim 1, wherein the requested action comprises an animated 3D overlay of the requested action. ### Claim 8 (depends on 1) The method of claim 1, wherein the requested action comprises a target performance metric for a smart tool being used by the novice user to perform the requested action. ### Claim 9 (depends on 1) The method of claim 1, wherein the confidence assessment comprises an algorithm derived confidence or an expert user's confidence that the requested action satisfies the performance requirements. ### Claim 10 (depends on 1) The method of claim 1, wherein the sensor data is collected by at least one sensor selected from a group consisting of smart gloves, electromyography (EMG) sensor body position data, electroencephalography (EEG) signals, head mounted camera, tripod mounted camera in a known physical, camera position, and a laser tool in a known physical, laser position, and wherein the confidence assessment further comprises measurements by an equipment sensor that provides completion data for automatically determining whether the requested action is completed. ### Claim 11 A remote expert system utilizing quantitative quality assurance, the remote expert system comprising: a mixed reality controller performing a method to: define a coordinate system frame of reference for a target physical system, the coordinate system frame of reference including an initial point of the target physical system and at least one directional axis that are specified by a user of the mixed reality controller; define additional points of the target physical system; and define interface elements of the target physical system; and a computer system performing a method to: generate a 3D model of the target physical system based on the coordinate system frame of reference, the additional points, and the interface elements; (a) present the 3D representation of target equipment to an expert user, wherein a synchronized 3D representation of the target equipment is presented to a novice user; (b) receive a part selection and a requested action for an equipment part of the target equipment from the expert user; (c) parallel author, by a sensor, wherein the sensor creates a data structure based at least in part on information collected from a mixed reality environment, wherein the data structure is paralleled to author multiple outputs, the multiple outputs comprising position, corrosion type, and job control number; (d) present the novice user's performance of the requested action on the equipment part to the expert user, wherein the presentation of the novice user's performance comprises various sensor data collected at the novice user's physical environment location; (e) validate, by a direct measures feedback, the novice user's performance of the requested action; (f) generate a confidence assessment of the novice user's performance of the requested action, wherein the confidence assessment specifies whether the expert user rejects the requested action as failing performance requirements, based at least in part on an indirect measures or the direct measures feedback; (g) if the requested action is rejected as failing performance requirements, repeat steps (c)-(e) until the expert accepts the requested action as satisfying the performance requirements; and (h) repeat steps (a)-(g) until a plurality of requested actions are accepted by the expert as satisfying the performance requirements, wherein the plurality of requested actions complete a maintenance task. ### Claim 12 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the expert has a digital twin of the target equipment that the novice user is manipulating, whereas the novice user manipulates physical characteristics of the target equipment, the manipulations are reproduced digitally in the expert user's digital twin model. ### Claim 13 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the requested action can be sent, executed, and signed off asynchronously. ### Claim 14 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the expert user can be a human or a computing device with artificial intelligence. ### Claim 15 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein a first cloud mesh of the 3D presentation of the target equipment is periodically aligned with a second cloud mesh of the synchronized 3D representation. ### Claim 16 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the requested action comprises a description of the requested action. ### Claim 17 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the requested action comprises an animated 3D overlay of the requested action. ### Claim 18 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the requested action comprises a target performance metric for a smart tool being used by the novice user to perform the requested action. ### Claim 19 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the confidence assessment comprises an algorithm derived confidence or an expert user's confidence that the requested action satisfies the performance requirements. ### Claim 20 (depends on 11) The remote expert system utilizing quantitative quality assurance according to claim 11, wherein the sensor data is collected by at least one sensor selected from a group consisting of smart gloves, electromyography (EMG) sensor body position data, electroencephalography (EEG) signals, head mounted camera, tripod mounted camera in a known physical, camera position, and a laser tool in a known physical, laser position, and wherein the confidence assessment further comprises measurements by an equipment sensor that provides completion data for automatically determining whether the requested action is completed. ### Claim 21 A non-transitory computer-readable medium comprising executable instructions for causing a computer system to perform a method for remote expert utilizing quantitative quality assurance in a mixed reality system, the instructions when executed causing the computer system to: (a) present a 3D representation of target equipment to an expert user, wherein a synchronized 3D representation of the target equipment is presented to a novice user; (b) receive a part selection and a requested action for an equipment part of the target equipment from the expert user; (c) present the novice user's performance of the requested action on the equipment part to the expert user, wherein the presentation of the novice user's performance comprises various sensor data collected at the novice user's physical environment location; (d) parallel author, by a sensor, wherein the sensor creates a data structure based at least in part on information collected from a mixed reality environment, wherein the data structure is paralleled to author multiple outputs, the multiple outputs comprising position, corrosion type, and job control number; (e) validate, by a direct measures feedback, the novice user's performance of the requested action; (f) generate a confidence assessment of the novice user's performance of the requested action, wherein the confidence assessment specifies whether the expert user rejects the requested action as failing performance requirements, based at least in part on an indirect measures or the direct measures feedback; (g) if the requested action is rejected as failing performance requirements, repeat steps (c)-(d) until the expert accepts the requested action as satisfying the performance requirements; and (h) repeat steps (a)-(g) until a plurality of requested actions are accepted by the expert as satisfying the performance requirements, wherein the plurality of requested actions complete a maintenance task.