# US 11,253,181 B2 · Method for objectively tracking and analyzing the social and emotional activity of a patient > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,253,181 B2 - Document ID: US11253181B2 - Status: Granted - Kind: B2 - Application number: 16/531,049 - Filed: 2019-08-03 - Published: 2020-02-06 - Granted: 2022-02-22 - Earliest claimed: 2018-08-03 - Assignees: From Zero, LLC - Applicants: From Zero, LLC - Inventors: Kevin Chaja; Natasha Chaja - Web page: https://patentgrep.com/patent/US11253181B2 ## CPC - A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety} - A61B 5/7445: {Display arrangements, e.g. multiple display units} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/012: {Head tracking input arrangements} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications} - G06T 7/0014: {using an image reference approach} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns - G06T 2207/30201: Face - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method and system for objectively tracking and analyzing the social and emotional activity of a patient using an augmented reality computing device is provided. A patient is permitted to manually manipulate a target object in the physical world while viewing an augmented version showing a unique animated character representing either an abstract language, emotions, or social skills, depending on the module. The present system tracks and records the active face and the time spent on the active face, where the active face is the face upon which the patient's focus is automatically estimated, through calculation, to be trained upon. An observer views the session, the data recorded, and an automatically generated graphical representation of the data, which permits the observer to speak to patient regarding the character or scene rendered on the face which is determined to be the active face, helping the student engage in the session. ## Claims ### Claim 1 A method for objectively tracking and analyzing the social and emotional activity of a patient using a computing device, the method comprising the steps of: implementing an augmented reality tracking application in memory on the computing device; upon the augmented tracking application being initialized on the computing device, detecting by at least one camera a target object manipulated by the patient, the target object comprising a first face and a second face, a first fiducial marker positioned on the first face and a second fiducial marker positioned on the second face; detecting at least one of the first fiducial marker and the second fiducial marker; displaying on a screen of the computing device a first augmented image overlaid on the target object, the first augmented image comprising a first image associated with the first fiducial marker and a second image associated with the second fiducial marker; displaying a second augmented image in place of displaying the first augmented image upon receiving an input activating a first image link, the second augmented image comprising a third image associated with the first fiducial marker and a fourth image associated with the second fiducial marker; displaying a third augmented image in place of displaying the first augmented image upon receiving a second input activating a second image link, the third augmented image comprising a fifth image associated with the first fiducial marker and a sixth image associated with the second fiducial marker; detecting a position and a rotation relative to the camera of the target object for each point in time for which a data point is collected; calculating a coordinate data set of the target object based on the position and the rotation detected; determining an active face for each point in time using the coordinate data set and a parameter indicative of the patient focusing attention on one of the first image and the second image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and storing as a data set the data point for each point in time, the data point comprising the coordinate data set, the active face, and a time. ### Claim 2 (depends on 1) The method of claim 1 further comprising the step of associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject. ### Claim 3 (depends on 2) The method of claim 2 further comprising the steps of transmitting the data set to a second computing device; and displaying to a user at least a portion of the data set on a second screen of the second computing device. ### Claim 4 (depends on 3) The method of claim 3 further comprising the steps of recording a video of the patient; transmitting the video to the second computing device; and displaying the video on the second screen of the second computing device. ### Claim 5 (depends on 3) The method of claim 3 further comprising the steps of transmitting a mirror video of the screen of the first computing device to the second computing device; and displaying the mirror video on the second screen of the second computing device. ### Claim 6 (depends on 3) The method of claim 3 further comprising the steps of retrieving from memory one or more talking prompts comprising a first talking prompt associated with the first therapeutic subject and a second talking prompt associated with the second therapeutic subject; and communicating the talking prompts through the second computing device to a user. ### Claim 7 (depends on 3) The method of claim 3 further comprising the steps of establishing a videotelephony session between the patient and the user to permit the transmission in real-time and audio-video signal. ### Claim 8 (depends on 3) The method of claim 3 further comprising the steps of processing the data set to automatically create a graphical representation of the data set; and displaying the graphical representation on the second screen of the second computing device. ### Claim 9 (depends on 1) The method of claim 1 wherein the target object is a cube comprising the first face, the second face, a third face with a third fiducial marker positioned on the third face, a fourth face with a fourth fiducial marker positioned on the fourth face, a fifth face with a fifth fiducial marker positioned on the fifth face, and a sixth face with a sixth fiducial marker positioned on the sixth face. ### Claim 10 (depends on 1) The method of claim 1 further comprising the steps of: determining, for the second augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the third image and the fourth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and determining, for the third augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the fifth image and the sixth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time. ### Claim 11 (depends on 1) The method of claim 1 further comprising the steps of: associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject; associating the third image with a first subject first subtopic of the first therapeutic subject and the fourth image with a first subject second subtopic of the first therapeutic subject, the data point further comprising the first subject first subtopic and the first subject second subtopic; and associating the fifth image with a second subject first subtopic of the second therapeutic subject and the sixth image with a second subject second subtopic of the first therapeutic subject, the data point further comprising the second subject first subtopic and the second subject second subtopic. ### Claim 12 A method for objectively tracking and analyzing the social and emotional activity of a patient using a computing device, the method comprising the steps of: implementing an augmented reality tracking application in memory on the computing device; upon the augmented tracking application being initialized on the computing device, detecting by at least one camera a target object manipulated by the patient, the target object comprising a first face and a second face, a first fiducial marker positioned on the first face and a second fiducial marker positioned on the second face; detecting at least one of the first fiducial marker and the second fiducial marker; displaying on a screen of the computing device a first augmented image overlaid on the target object, the first augmented image comprising a first image associated with the first fiducial marker and a second image associated with the second fiducial marker; detecting a position and a rotation relative to the camera of the target object for each point in time for which a data point is collected; calculating a coordinate data set of the target object based on the position and the rotation detected; determining an active face for each point in time using the coordinate data set and a parameter indicative of the patient focusing attention on one of the first image and the second image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and storing as a data set the data point for each point in time, the data point comprising the coordinate data set, the active face, and a time, wherein the step of determining the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the first image and the second image, wherein the parameter is one or more of determining which of the first face and the second face are oriented more upright compared to the other, determining which of the first face and the second face has a larger screen area, and determining which of the first face and the second face has received a touch input. ### Claim 13 (depends on 12) The method of claim 12 further comprising the step of associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject. ### Claim 14 (depends on 13) The method of claim 13 further comprising the steps of transmitting the data set to a second computing device; and displaying to a user at least a portion of the data set on a second screen of the second computing device. ### Claim 15 (depends on 14) The method of claim 14 further comprising the steps of recording a video of the patient; transmitting the video to the second computing device; and displaying the video on the second screen of the second computing device. ### Claim 16 (depends on 14) The method of claim 14 further comprising the steps of transmitting a mirror video of the screen of the first computing device to the second computing device; and displaying the mirror video on the second screen of the second computing device. ### Claim 17 (depends on 14) The method of claim 14 further comprising the steps of retrieving from memory one or more talking prompts comprising a first talking prompt associated with the first therapeutic subject and a second talking prompt associated with the second therapeutic subject; and communicating the talking prompts through the second computing device to a user. ### Claim 18 (depends on 14) The method of claim 14 further comprising the steps of establishing a videotelephony session between the patient and the user to permit the transmission in real-time and audio-video signal. ### Claim 19 (depends on 14) The method of claim 14 further comprising the steps of processing the data set to automatically create a graphical representation of the data set; and displaying the graphical representation on the second screen of the second computing device. ### Claim 20 (depends on 12) The method of claim 12, wherein the target object is a cube comprising the first face, the second face, a third face with a third fiducial marker positioned on the third face, a fourth face with a fourth fiducial marker positioned on the fourth face, a fifth face with a fifth fiducial marker positioned on the fifth face, and a sixth face with a sixth fiducial marker positioned on the sixth face. ### Claim 21 (depends on 12) The method of claim 12 further comprising the steps of: displaying a second augmented image in place of displaying the first augmented image upon receiving an input activating a first image link, the second augmented image comprising a third image associated with the first fiducial marker and a fourth image associated with the second fiducial marker; and displaying a third augmented image in place of displaying the first augmented image upon receiving a second input activating a second image link, the third augmented image comprising a fifth image associated with the first fiducial marker and a sixth image associated with the second fiducial marker. ### Claim 22 (depends on 21) The method of claim 21 further comprising the steps of: determining, for the second augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the third image and the fourth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time; and determining, for the third augmented image, the active face for each point in time using the coordinate data set and the parameter indicative of the patient focusing attention on one of the fifth image and the sixth image, wherein the active face can be dynamically assigned to one of either the first face or the second face at each point in time. ### Claim 23 (depends on 21) The method of claim 21 further comprising the steps of: associating the first image with a first therapeutic subject and the second image with a second therapeutic subject, the data point further comprising the first therapeutic subject and the second therapeutic subject; associating the third image with a first subject first subtopic of the first therapeutic subject and the fourth image with a first subject second subtopic of the first therapeutic subject, the data point further comprising the first subject first subtopic and the first subject second subtopic; and associating the fifth image with a second subject first subtopic of the second therapeutic subject and the sixth image with a second subject second subtopic of the first therapeutic subject, the data point further comprising the second subject first subtopic and the second subject second subtopic.