# US 10,503,964 B1 · Method and system for measuring and visualizing user behavior in virtual reality and augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,964 B1 - Document ID: US10503964B1 - Status: Granted - Kind: B1 - Application number: 15/894,104 - Filed: 2018-02-12 - Granted: 2019-12-10 - Earliest claimed: 2017-02-10 - Assignees: Aldin Dynamics, Ehf. - Applicants: Aldin Dynamics, Ehf. - Inventors: Gunnar Steinn Valgardsson; Hrafn Thorri Thorisson; Pall Arinbjarnar - Web page: https://patentgrep.com/patent/US10503964B1 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 16/24575: {using context} - G06F 16/26: Visual data mining; Browsing structured data - G06T 11/60: Creating or editing images; Combining images with text - G06V 20/20: in augmented reality scenes - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 11/26: Drawing of charts or graphs - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract According to one aspect of the disclosure, a method of analyzing users in a Virtual Reality (VR) or Augmented Reality (AR) environment is provided. The method includes acts of receiving a data stream including information indicative of one or more user actions performed by a user in a simulated environment, analyzing the data stream to determine whether a first condition is satisfied, capturing, responsive to determining that the first condition is satisfied, first user behavior data including at least a portion of the information indicative of the one or more user actions, receiving a request for user behavior data satisfying the first criterion, extracting, from the user behavior data, a plurality of user behavior data segments, transmitting, responsive to receiving the request, the plurality of user behavior data segments, and displaying a visualization of the plurality of user behavior data segments. ## Claims ### Claim 1 A method of analyzing users in a Virtual Reality (VR) or Augmented Reality (AR) environment, the method comprising: receiving a data stream including information indicative of one or more user actions performed by a user in a simulated environment; analyzing the data stream to determine whether a first condition is satisfied; capturing, responsive to determining that the first condition is satisfied, first user behavior data including at least a portion of the information indicative of the one or more user actions; receiving a request for user behavior data satisfying the first condition; extracting, from the user behavior data, a plurality of user behavior data segments; transmitting, responsive to receiving the request, the plurality of user behavior data segments; and displaying a visualization of the plurality of user behavior data segments. ### Claim 2 (depends on 1) The method of claim 1, further comprising filtering the plurality of user behavior data segments based on one or more of physical characteristics, physical behavior, emotional characteristics, and VR or AR hardware characteristics. ### Claim 3 (depends on 2) The method of claim 2, wherein the physical characteristics include at least one of body language, gestures, height, stature, reach, and handedness. ### Claim 4 (depends on 2) The method of claim 2, wherein the emotional characteristics are determined based on user body language. ### Claim 5 (depends on 4) The method of claim 4, wherein the emotional characteristics include at least one of excitement, boredom, agitation, fright, happiness, sadness, anger, fear, surprise, and disgust. ### Claim 6 (depends on 2) The method of claim 2, wherein the physical behavior is indicative of actions by users or physical movement of users. ### Claim 7 (depends on 6) The method of claim 6, wherein the physical behavior of users is determined based on one or more of a user activity level, a button selection frequency, and an environment interaction frequency. ### Claim 8 (depends on 1) The method of claim 1, wherein each user behavior data segment of the plurality of user behavior data segments corresponds to a respective user of a plurality of users, and wherein displaying the visualization of the plurality of user behavior data segments includes concurrently displaying a respective visualization of physical movement of all or a subset of the plurality of users. ### Claim 9 (depends on 1) The method of claim 1, wherein each behavior data segment of the plurality of behavior data segments includes at least one label indicative of a property of an object in the behavior data segment. ### Claim 10 (depends on 9) The method of claim 9, wherein the property of the object includes at least one of a timestamped position and a timestamped orientation of the object. ### Claim 11 A system for analyzing users in a Virtual Reality (VR) or Augmented Reality (AR) environment, the system comprising: a first computing device configured to: receive a data stream including information indicative of one or more user actions performed by a user in a simulated environment; analyze the data stream to determine whether a first condition is satisfied; capture, responsive to determining that the first condition is satisfied, first user behavior data including at least a portion of the information indicative of the one or more user actions; receive a request for user behavior data satisfying the first condition; extract a plurality of user behavior data segments from the user behavior data; and transmit, responsive to receiving the request, the plurality of user behavior data segments including the first user behavior data; and a second computing device configured to: receive the plurality of user behavior data segments; and display a visualization of the plurality of user behavior data segments. ### Claim 12 (depends on 11) The system of claim 11, wherein the second computing device is further configured to filter the plurality of user behavior data segments based on one or more of physical characteristics, physical behavior, emotional characteristics, and VR or AR hardware characteristics. ### Claim 13 (depends on 12) The system of claim 12, wherein the physical characteristics include at least one of body language, gestures, height, stature, reach, and handedness. ### Claim 14 (depends on 12) The system of claim 12, wherein the emotional characteristics are determined based on user body language. ### Claim 15 (depends on 14) The system of claim 14, wherein the emotional characteristics include at least one of excitement, boredom, agitation, fright, happiness, sadness, anger, fear, surprise, and disgust. ### Claim 16 (depends on 12) The system of claim 12, wherein the physical behavior is indicative of actions by users or physical movement of users. ### Claim 17 (depends on 16) The system of claim 16, wherein the physical behavior of users is determined based on one or more of a user activity level, a button selection frequency, and an environment interaction frequency. ### Claim 18 (depends on 11) The system of claim 11, wherein each user behavior data segment of the plurality of user behavior data segments corresponds to a respective user of a plurality of users, and wherein displaying the visualization of the plurality of user behavior data segments includes concurrently displaying a respective visualization of physical movement of all or a subset of the plurality of users. ### Claim 19 (depends on 11) The system of claim 11, wherein each behavior data segment of the plurality of behavior data segments includes at least one label indicative of a property of an object in the behavior data segment. ### Claim 20 A non-transitory computer-readable medium storing sequences of computer-executable instructions for analyzing users in a Virtual Reality (VR) or Augmented Reality (AR) environment, the sequences of computer-executable instructions including instructions that instruct at least one processor to: receive a data stream including information indicative of one or more user actions performed by a user in a simulated environment; analyze the data stream to determine whether a first condition is satisfied; capture, responsive to determining that the first condition is satisfied, first user behavior data including at least a portion of the information indicative of the one or more user actions; receive a request for user behavior data satisfying the first condition; extract a plurality of user behavior data segments from the user behavior data; and transmit, responsive to receiving the request, the plurality of user behavior data segments including the first user behavior data.