# US 11,307,650 B1 · Modifying virtual content to invoke a target user state

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,307,650 B1
- Document ID: US11307650B1
- Status: Granted
- Kind: B1
- Application number: 16/859,879
- Filed: 2020-04-27
- Granted: 2022-04-19
- Earliest claimed: 2019-06-25
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Gutemberg B. Guerra Filho; Ian M. Richter; Raffi A. Bedikian
- Web page: https://patentgrep.com/patent/US11307650B1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 20/20: in augmented reality scenes
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow}
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one implementation, a method for generating computer-generated reality content (CGR) content in order to invoke a target state of a user based on a user model is performed at an electronic device. The method includes: while presenting reference CGR content via the one or more displays, obtaining a request from a user to invoke a target state for the user; generating, based on a user model associated with the user and the reference CGR content, modified CGR content to invoke the target state for the user, wherein the user model provides projected reactions to CGR content; and presenting, via the one or more displays, the modified CGR content. In some implementations, obtaining the request from the user to invoke the target state for the user includes determining whether the user provided informed consent to store user information in the user model associated with the user of the device.

## Claims

### Claim 1

A method comprising:
at an electronic device including one or more processors, a non-transitory memory, and one or more displays:
while presenting reference computer-generated reality (CGR) content via the one or more displays, obtaining a request from a user to invoke a target state for the user;
in response to obtaining the request from the user to invoke the target state for the user:
obtaining sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user;
determining, using a qualitative mood classifier, a current measured state of the user based on the sensor information associated with the user; and
generating modified CGR content to invoke the target state for the user by modifying the reference CGR content based on a user model associated with the user, the current measured state of the user, and the reference CGR content, wherein the user model provides projected reactions to the modified CGR content; and
presenting, via the one or more displays, the modified CGR content.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
after presenting the modified CGR content, obtaining sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user;
determining, using a qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user; and
updating the user model based at least in part on the resultant state of the user and the modified CGR content.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
obtaining the sensor information associated with the user via one or more sensors of the electronic device.

### Claim 4 (depends on 2)

The method of claim 2, wherein the one or more physiological measurements of the user include at least one of eye tracking information, pupil dilation information, body pose characteristics, speech characteristics, heart rate, glucose level, and blood oximetry.

### Claim 5 (depends on 2)

The method of claim 2, wherein determining the resultant state of the user includes determining whether the user provided informed consent to monitor one or more physiological modalities associated with the user.

### Claim 6 (depends on 1)

The method of claim 1, wherein the reference CGR content corresponds to a virtual agent.

### Claim 7 (depends on 6)

The method of claim 6, wherein modifying the reference CGR content includes at least one of changing an expression of the virtual agent and changing one or more actions of the virtual agent.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the modified CGR content includes selecting a portion of the user model based at least in part on the current measured state and the target state.

### Claim 9 (depends on 1)

The method of claim 1, wherein modifying the reference CGR content includes adding CGR content to the reference CGR content based at least in part on the user model.

### Claim 10 (depends on 1)

The method of claim 1, wherein modifying the reference CGR content includes scaling the reference CGR content based at least in part on the user model.

### Claim 11 (depends on 1)

The method of claim 1, wherein modifying the reference CGR content includes modifying a set of available interactions associated with the reference CGR content based at least in part on the user model.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
obtaining user information from the user model associated with the user of the device stored on a secure local non-transitory memory of the electronic device.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
storing user information in the user model associated with the user of the device to a secure local non-transitory memory of the electronic device, wherein the user model is stored in the secure local non-transitory memory of the electronic device.

### Claim 14 (depends on 1)

The method of claim 1, wherein obtaining the request from the user to invoke the target state for the user includes determining whether the user provided informed consent to store user information in the user model associated with the user of the device.

### Claim 15 (depends on 1)

The method of claim 1, wherein the modified CGR content corresponds to at least one of predetermined CGR content and emergent CGR content.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
obtaining a training data corpus that includes a plurality of sensor information sets, wherein each of the plurality of sensor information sets is associated with a respective state of the user, wherein each of the plurality of sensor information sets is associated with a respective one or more qualitative mood indicator values;
generating, using a qualitative mood classifier, at least one candidate qualitative mood indicator value corresponding to a portion of the plurality of sensor information sets;
comparing the at least one candidate qualitative mood indicator value against a corresponding qualitative mood indicator value within the training data corpus;
in response to determining that a result of the comparison between the at least one candidate mood indicator value against the corresponding qualitative mood indicator value does not satisfy an error metric, changing an operational value of the qualitative mood classifier; and
in response to determining that a result of the comparison between the at least one candidate qualitative mood indicator value against the corresponding qualitative mood indicator value satisfies the error metric and that a sufficient portion of the training data corpus is utilized, outputting a convergence indicator associated with the qualitative mood classifier.

### Claim 17 (depends on 16)

The method of claim 16, wherein the qualitative mood classifier corresponds to a neural network.

### Claim 18 (depends on 1)

The method of claim 1, wherein the electronic device corresponds to at least one of a near-eye system, a mobile phone, or a tablet.

### Claim 19 (depends on 1)

The method of claim 1, further comprising:
after presenting the modified CGR content, obtaining sensor information associated with a user reaction to the modified CGR content, wherein the sensor information corresponds to one or more physiological measurements of the user;
determining, using the qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user reaction to the modified CGR content;
in accordance with a determination that the resultant state of the user corresponds to the target state for the user, updating the user model to indicate that the modified CGR content successfully invoked the target state for the user; and
in accordance with a determination that the resultant state of the user does not correspond to the target state for the user:
updating the user model to indicate that the modified CGR content did not successfully invoke the target state for the user;
generating subsequent modified CGR content different from the modified CGR content to invoke the target state for the user by modifying the reference CGR content based on the user model associated with the user, the resultant state of the user, and the reference CGR content; and
presenting, via the one or more displays, the subsequent modified CGR content.

### Claim 20

An electronic device comprising:
one or more processors;
one or more displays;
a non-transitory memory; and
one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to:
while presenting reference computer-generated reality (CGR) content via the one or more displays, obtain a request from a user to invoke a target state for the user;
in response to obtaining the request from the user to invoke the target state for the user:
obtain sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user;
determine, using a qualitative mood classifier, a current measured state of the user based on the sensor information associated with the user; and
generate modified CGR content to invoke the target state for the user by modifying the reference CGR content based on a user model associated with the user, the current measured state of the user, and the reference CGR content, wherein the user model provides projected reactions to the modified CGR content; and
present, via the one or more displays, the modified CGR content.

### Claim 21 (depends on 20)

The electronic device of claim 20, wherein the one or more programs further cause the device to:
after presenting the modified CGR content to the user: obtain sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user;
determine, using a qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user; and
update the user model based at least in part on the resultant state of the user and the modified CGR content.

### Claim 22 (depends on 21)

The electronic device of claim 21, wherein the one or more programs further cause the device to:
obtain the sensor information associated with the user via one or more sensors of the electronic device.

### Claim 23 (depends on 20)

The electronic device of claim 20, wherein the one or more programs further cause the device to:
after presenting the modified CGR content, obtain sensor information associated with a user reaction to the modified CGR content, wherein the sensor information corresponds to one or more physiological measurements of the user;
determine, using the qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user reaction to the modified CGR content;
in accordance with a determination that the resultant state of the user corresponds to the target state for the user, update the user model to indicate that the modified CGR content successfully invoked the target state for the user; and
in accordance with a determination that the resultant state of the user does not correspond to the target state for the user:
update the user model to indicate that the modified CGR content did not successfully invoke the target state for the user;
generate subsequent modified CGR content different from the modified CGR content to invoke the target state for the user by modifying the reference CGR content based on the user model associated with the user, the resultant state of the user, and the reference CGR content; and
present, via the one or more displays, the subsequent modified CGR content.

### Claim 24

A non-transitory memory storing one or more programs, which, when executed by one or more processors of an electronic device with one or more displays, cause the device to:
while presenting reference computer-generated reality (CGR) content via the one or more displays, obtain a request from a user to invoke a target state for the user;
in response to obtaining the request from the user to invoke the target state for the user:
obtain sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user;
determine, using a qualitative mood classifier, a current measured state of the user based on the sensor information associated with the user; and
generate modified CGR content to invoke the target state for the user by modifying the reference CGR content based on a user model associated with the user, the current measured state of the user, and the reference CGR content, wherein the user model provides projected reactions to the modified CGR content; and
present, via the one or more displays, the modified CGR content.

### Claim 25 (depends on 24)

The non-transitory memory of claim 24, wherein the one or more programs further cause the device to:
after presenting the modified CGR content obtain sensor information associated with the user, wherein the sensor information corresponds to one or more physiological measurements of the user;
determine, using a qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user; and
update the user model based at least in part on the resultant state of the user and the modified CGR content.

### Claim 26 (depends on 25)

The non-transitory memory of claim 25, wherein the one or more programs further cause the device to:
obtain the sensor information associated with the user via one or more sensors of the electronic device.

### Claim 27 (depends on 24)

The non-transitory memory of claim 24, wherein the one or more programs further cause the device to:
after presenting the modified CGR content, obtain sensor information associated with a user reaction to the modified CGR content, wherein the sensor information corresponds to one or more physiological measurements of the user;
determine, using the qualitative mood classifier, a resultant state of the user based on the sensor information associated with the user reaction to the modified CGR content;
in accordance with a determination that the resultant state of the user corresponds to the target state for the user, update the user model to indicate that the modified CGR content successfully invoked the target state for the user; and
in accordance with a determination that the resultant state of the user does not correspond to the target state for the user:
update the user model to indicate that the modified CGR content did not successfully invoke the target state for the user;
generate subsequent modified CGR content different from the modified CGR content to invoke the target state for the user by modifying the reference CGR content based on the user model associated with the user, the resultant state of the user, and the reference CGR content; and
present, via the one or more displays, the subsequent modified CGR content.