# US 2020/0401216 A1 · Technique for controlling virtual image generation system using emotional states of user > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401216 A1 - Document ID: US20200401216A1 - Status: Published application - Kind: A1 - Application number: 17/012,309 - Filed: 2020-09-04 - Published: 2020-12-24 - Earliest claimed: 2016-07-21 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: George Alistair Sanger; Samuel A. Miller; Graeme John Devine - Web page: https://patentgrep.com/patent/US20200401216A1 ## CPC - G06F 3/012: {Head tracking input arrangements} - A63F 13/21: characterised by their sensors, purposes or types - A63F 13/212: using sensors worn by the player, e.g. for measuring heart beat or leg activity - A63F 13/52: involving aspects of the displayed game scene - A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition - A63F 13/822: Strategy games; Role-playing games (A63F13/825, A63F13/828 take precedence) - G02B 27/017: {Head mounted} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 16/56: having vectorial format - G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G06V 40/175: {Static expression} - G02B 27/0172: {characterised by optical features} - G02B 2027/0134: {of stereoscopic type} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of operating a virtual image generation system comprises allowing an end user to interact with a three-dimensional environment comprising at least one virtual object, presenting a stimulus to the end user in the context of the three-dimensional environment, sensing at least one biometric parameter of the end user in response to the presentation of the stimulus to the end user, generating biometric data for each of the sensed biometric parameter(s), determining if the end user is in at least one specific emotional state based on the biometric data for the each of the sensed biometric parameter(s), and performing an action discernible to the end user to facilitate a current objective at least partially based on if it is determined that the end user is in the specific emotional state(s). ## Claims ### Claim 1 A method of operating an augmented reality (AR) display system that includes a wearable display device, the method comprising: responsive to presenting a stimulus to a user through the wearable display device, sensing at least one biometric parameter of the user, the at least one biometric parameter sensed using at least one sensor that is attached to the wearable display device; generating biometric data for each of the at least one sensed biometric parameter; determining at least one emotional state of the user based on the biometric data for the each of the at least one sensed biometric parameter; and presenting at least one additional virtual object through the AR display system based at least partly on determining that the user is in the at least one emotional state, wherein the at least one additional virtual object is selected to evoke another emotional state in the user different from the at least one emotional state of the user. ### Claim 2 (depends on 1) The method of claim 1, further comprising determining the at least one emotional state of the user for a duration, wherein the at least one additional virtual object is presented through the AR display system based at least partially on determining that the user is in the at least one emotional state for the duration. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one emotional state comprises a plurality of different emotional states. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one biometric parameter of the user is sensed at a plurality of different times in response to the presenting the stimulus, and the biometric data is generated at these different times. ### Claim 5 (depends on 1) The method of claim 1, wherein the stimulus is visually presented to the user through the wearable display device. ### Claim 6 (depends on 1) The method of claim 1, wherein the at least one sensed biometric parameter comprises a plurality of different sensed biometric parameters, and wherein determining the at least one emotional state of the user comprises performing a pattern recognition analysis on the generated biometric data. ### Claim 7 (depends on 1) The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one facial expression. ### Claim 8 (depends on 7) The method of claim 7, wherein the least one facial expression is one or both of an attitude of a mouth and crow's feet around eyes of the user, and wherein the at least one emotional state comprises happiness. ### Claim 9 (depends on 1) The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one of a facial expression, hunching of shoulders, respiration rate, heart rate, body temperature, blood pressure, frequency and/or location of hand movements, frequency and/or location of body twitches, and elapsed time between eye movements. ### Claim 10 (depends on 1) The method of claim 1, wherein the at least one sensed biometric parameter comprises at least one micro-expression. ### Claim 11 (depends on 1) The method of claim 1, wherein the at least one emotional state comprises at least one of anger, contempt, disgust, fear, happiness, sadness, surprise, confusion, shame, attentiveness, exhaustion, relaxation, frustration, boredom, embarrassment. ### Claim 12 (depends on 1) The method of claim 1, wherein determining at least one emotional state of the user comprises comparing the generated biometric data to reference biometric data correlated to the at least one emotional state. ### Claim 13 (depends on 12) The method of claim 12, wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric scalar data value, the reference biometric data comprises a reference biometric value range, and comparing the generated biometric data to the reference biometric data comprises determining whether the biometric scalar data value falls within the reference biometric value range. ### Claim 14 (depends on 12) The method of claim 12, wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric multi-dimensional data vector, the reference biometric data comprises a reference biometric multi-dimensional data vector, and comparing the generated biometric data to the reference biometric data comprises performing a correlation function between the generated biometric multi-dimensional data vector and the reference biometric multi-dimensional data vector. ### Claim 15 (depends on 12) The method of claim 12, wherein determining the at least one emotional state of the user further comprises retrieving the reference biometric data from a custom emotional state profile of the user. ### Claim 16 (depends on 1) The method of claim 1, further comprising: rendering a plurality of synthetic image frames of a three-dimensional environment; and sequentially displaying the plurality of synthetic image frames to the user through wearable display device, wherein the synthetic image frames are projected from a transparent display surface in a field of view of the user via a frame structure mounted to a head of the user, and wherein the synthetic image frames are superimposed over a real scene visualized by the user. ### Claim 17 (depends on 1) The method of claim 1, wherein the AR display system also includes a speaker coupled to the wearable display device, and wherein the stimulus is aurally presented through the speaker. ### Claim 18 (depends on 1) The method of claim 1, wherein the AR display system also includes a mechanical actuator, and wherein the stimulus is tactilely presented through the mechanical actuator. ### Claim 19 (depends on 1) The method of claim 1, wherein the at least one sensor comprises a rearward facing camera coupled to the wearable display device, and wherein the at least one biometric parameter comprises a facial movement of the user. ### Claim 20 (depends on 1) The method of claim 1, wherein the at least one sensor comprises an accelerometer coupled to a body part of the user, and wherein the at least one biometric parameter comprises a movement of the body part of the user.