# US 11,744,495 B2 · Method for objectively tracking and analyzing the social and emotional activity of a patient

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,744,495 B2
- Document ID: US11744495B2
- Status: Granted
- Kind: B2
- Application number: 17/651,795
- Filed: 2022-02-19
- Published: 2022-06-02
- Granted: 2023-09-05
- Earliest claimed: 2018-08-03
- Assignees: From Zero, LLC
- Applicants: From Zero, LLC
- Inventors: Kevin Chaja; Natasha Chaja
- Web page: https://patentgrep.com/patent/US11744495B2

## CPC

- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/7445: {Display arrangements, e.g. multiple display units}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications}
- G06T 7/0014: {using an image reference approach}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06T 2207/30201: Face
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method and system for objectively tracking and analyzing the social and emotional activity of a patient using an augmented reality computing device is provided. A patient is permitted to manually manipulate a target object in the physical world while viewing an augmented version showing a unique animated character representing either an abstract language, emotions, or social skills, depending on the module. The present system tracks and records the active face and the time spent on the active face, where the active face is the face upon which the patient's focus is automatically estimated, through calculation, to be trained upon. An observer views the session, the data recorded, and an automatically generated graphical representation of the data, which permits the observer to speak to patient regarding the character or scene rendered on the face which is determined to be the active face, helping the student engage in the session.

## Claims

### Claim 1

A method for objectively tracking and analyzing the social and emotional activity of a patient using a computing device, the method comprising the steps of:
implementing an augmented reality tracking application in memory on the computing device;
detecting by at least one camera a multi-faced target object manipulated by the patient, the multi-faced target object comprising at least one face, each of the at least one face comprising a computer-readable fiducial marker;
displaying on a screen of the computing device at least a first set of augmented images or videos overlaid on the multi-faced target object, each of the first set of augmented images or videos being associated with one of the at least one face of the multi-faced target object, and wherein each of the at least first set of augmented images or videos is associating with a therapeutic subject;
detecting a position and a rotation data relative to the camera of the multi-faced target object for each point in time; wherein the position and rotation data comprises one or more parameters indicative of the patient focusing attention on each of the at least first set of augmented images or videos, wherein the one or more parameters comprises the time each of the at least one face is in an upright orientation, the time each of the at least one face has a larger screen area, the amount of patient interaction associated with each of the at least one face, and the therapeutic subject associated with each of the at least first set of augmented images or videos;
calculating a coordinate data set of the multi-faced target object based on the position and rotation data detected;
determining an active face for each point in time using the coordinate data set, wherein the active face can be dynamically assigned to each of the at least one face at each point in time and; and
storing as a data set the position and rotation data, the coordinate data set, the active face, and a time for each point in time.

### Claim 2 (depends on 1)

The method of claim 1, wherein the therapeutic subject comprises a module on emotion identification, a module on abstract language, a module on social skills, and/or a module on emotional quotient.

### Claim 3 (depends on 2)

The method of claim 2, wherein the module on emotion identification comprises the emotions happy, sad, angry, fear, bored and/or disgusted.

### Claim 4 (depends on 1)

The method of claim 1, further comprising the steps of:
displaying on a screen of the computing device at least second set of augmented images or videos in place of displaying the at least first set of augmented images or videos upon receiving an input activating a first image link associated with each of the at least first set of augmented images or videos, each of the at least second set of augmented images or videos being associated with one of the at least one face of the multi-faced target object, wherein each of the at least second set of augmented images or videos is different than the one associated with the each of the at least first set of augmented images or videos, and wherein each of the at least second set of augmented images or videos is associating with a therapeutic subtopic; the therapeutic subtopic providing additional information on the therapeutic subject associated with that therapeutic subtopic; and
determining, for the second set of augmented images or videos, an active face for each point in time using the coordinate data set, wherein the active face can be dynamically assigned to each of the at least one face at each point in time, and wherein the one or more parameters further comprises the therapeutic subtopic associated with each of the at least second set of augmented images or videos.

### Claim 5 (depends on 4)

The method of claim 4, wherein the therapeutic subtopic of the therapeutic subject for a module on emotion identification comprises:
a happy therapeutic subtopic comprises being joyful, being happy over an individual success, being happy over a team success, being congratulated by parents, being congratulated by friends, and/or receiving physical love;
a sad therapeutic subtopic comprises suffering a personal loss, suffering a family loss, suffering a tangible loss, being neglected, being harassed or bullied, being alone, and/or low self-image and self-reflection;
an angry therapeutic subtopic comprises fighting with others, lashing out, an internal tantrum, an external tantrum, a jealousy, and/or a loss;
a fear therapeutic subtopic comprises an external fear of objects, an anxiety or worry in a social situation, an internal fear, an external non-real fear, a real conflict, and/or a fear of physical danger;
a bored therapeutic subtopic comprises being alone, uninterested or lacking responsiveness with others, uninterested or lacking responsiveness with objects, uninterested or lacking responsiveness in school, uninterested or lacking responsiveness in playing, uninterested or lacking responsiveness with a chore, indifferent an authority figure, indifferent to affection; and/or
a disgusted therapeutic subtopic comprises being physically sick due to external stimulus, having a perceived sickness due to an external stimulus, smelling stinky laundry, smelling skunk odor, smelling garbage, stepping on gum, having dirty hands, and/or disgust with littering.

### Claim 6 (depends on 1)

The method of claim 1, wherein each of the at least first set of augmented images or videos overlaid on the multi-faced target object is displaying on a screen of the computing device even when the face associated with that augmented image or video is out of view of the at least one camera.

### Claim 7 (depends on 1)

The method of claim 1 further comprising the steps of retrieving from memory one or more talking prompts associated with the therapeutic subject associated with each of the at least first set of augmented images or videos; and communicating the one or more talking prompts through a second computing device to a user.

### Claim 8 (depends on 1)

The method of claim 1 further comprising the steps of transmitting the data set to a second computing device; and displaying to a user at least a portion of the data set on a second screen of the second computing device.

### Claim 9 (depends on 8)

The method of claim 8 further comprising the steps of collecting video and/or audio recordings a of the patient; transmitting the video and/or audio recordings to the second computing device; and playing the video and/or audio recordings on the second screen of the second computing device.

### Claim 10 (depends on 8)

The method of claim 8 further comprising the steps of transmitting a mirror video and/or audio recording of the screen of the first computing device to the second computing device; and playing the mirror video and/or audio recording on the second screen of the second computing device.

### Claim 11 (depends on 8)

The method of claim 8 further comprising the steps of establishing a videotelephony session between the patient and the user to permit the transmission in real-time and audio-video signal.

### Claim 12 (depends on 8)

The method of claim 8 further comprising the steps of processing the data set to automatically create a graphical representation of the data set; and displaying the graphical representation on the second screen of the second computing device.

### Claim 13 (depends on 1)

The method of claim 1, wherein the at least one face of the multi-faced target object comprises a first face and a second face, the first face comprising a first computer-readable fiducial marker and the second face comprising a second computer-readable fiducial marker, the at least first set of augmented images or videos comprise a first augmented image associated with the first fiducial marker and a second augmented image or video associated with the second fiducial marker, wherein the first augmented image is associated with a first therapeutic subject, the second augmented image is associated with a second therapeutic subject, and the one or more parameters further comprises the first therapeutic subject and the second therapeutic subject.

### Claim 14 (depends on 13)

The method of claim 13, wherein the at least one face of the multi-faced target object further comprises a third face, the third face comprising a third computer-readable fiducial marker and the at least first set of augmented images or videos overlaid on the multi-faced target object further comprise a third augmented image associated with the third fiducial marker, wherein the third augmented image is associated with a third therapeutic subject and the one or more parameters further comprising the third therapeutic subject.

### Claim 15 (depends on 14)

The method of claim 14, wherein the at least one face of the multi-faced target object further comprises a fourth face, the fourth face comprising a fourth computer-readable fiducial marker and the at least first set of augmented images or videos overlaid on the multi-faced target object further comprise a fourth augmented image associated with the fourth fiducial marker, wherein the fourth augmented image is associated with a fourth therapeutic subject and the one or more parameters further comprising the fourth therapeutic subject.

### Claim 16 (depends on 15)

The method of claim 15, wherein the at least one face of the multi-faced target object further comprises a fifth face, the fifth face comprising a fifth computer-readable fiducial marker and the at least first set of augmented images or videos overlaid on the multi-faced target object further comprise a fifth augmented image associated with the fifth fiducial marker, wherein the fifth augmented image is associated with a fifth therapeutic subject and the one or more parameters further comprising the fifth therapeutic subject.

### Claim 17 (depends on 16)

The method of claim 16, wherein the at least one face of the multi-faced target object further comprises a sixth face, the sixth face comprising a sixth computer-readable fiducial marker and the at least first set of augmented images or videos overlaid on the multi-faced target object further comprise a sixth augmented image associated with the sixth fiducial marker, wherein the sixth augmented image is associated with a sixth therapeutic subject and the one or more parameters further comprising the sixth therapeutic subject.