# US 10,484,597 B2 · Emotional/cognative state-triggered recording > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,484,597 B2 - Document ID: US10484597B2 - Status: Granted - Kind: B2 - Application number: 16/183,609 - Filed: 2018-11-07 - Published: 2019-03-07 - Granted: 2019-11-19 - Earliest claimed: 2016-05-18 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: John C. Gordon; Cem Keskin - Web page: https://patentgrep.com/patent/US10484597B2 ## CPC - H04N 23/611: where the recognised objects include parts of the human body - A61B 3/112: {for measuring diameter of pupils} - A61B 3/113: for determining or recording eye movement - A61B 5/01: Measuring temperature of body parts {; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue} (clinical contact thermometers G01K13/20) - A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety} - G02B 27/0172: {characterised by optical features} - G06F 3/012: {Head tracking input arrangements} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06V 40/174: {Facial expression recognition} - G06V 40/18: Eye characteristics, e.g. of the iris - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - H04N 5/77: between a recording apparatus and a television camera - H04N 5/772: {the recording apparatus and the television camera being placed in the same enclosure} - H04N 9/8205: {involving the multiplexing of an additional signal and the colour video signal} - A61B 2503/12: Healthy persons not otherwise provided for, e.g. subjects of a marketing survey - G02B 2027/0138: {comprising image capture systems, e.g. camera} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Emotional/cognitive state-triggered recording is described. A buffer is used to store captured video content until a change in an emotional or cognitive state of a user is detected. Sensor data indicating a change in an emotional or cognitive state of a user triggers the creation of a video segment based on the current contents of the buffer. ## Claims ### Claim 1 A system comprising: a camera configured to capture video data; one or more processors; one or more sensors configured to obtain sensor data; and memory storing instructions that, when executed by the one or more processors, cause the system to: detect, based at least in part on the sensor data, a change from a first emotional or cognitive state of a user to a second emotional or cognitive state of the user; in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, add at least a portion of the video data being captured by the camera to a video segment; detect, based at least in part on additional sensor data, a change from the second emotional or cognitive state of the user to a third emotional or cognitive state of the user; and cease adding the at least the portion of the video data to the video segment based at least in part on detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user. ### Claim 2 (depends on 1) A system as recited in claim 1, wherein the instructions further cause the system to initiate a recording of the video segment in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user. ### Claim 3 (depends on 1) A system as recited in claim 1, wherein: the video segment begins at a first time associated with the detecting of the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user; and the video segment ends at a second time associated with the detecting of the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user. ### Claim 4 (depends on 1) A system as recited in claim 1, further comprising a buffer configured to store the video data captured by the camera. ### Claim 5 (depends on 4) A system as recited in claim 4, wherein the buffer is configured as a ring buffer. ### Claim 6 (depends on 1) A system as recited in claim 1, wherein the one or more sensors are configured to obtain the sensor data and the additional sensor data based on one or more of: a galvanic skin response, a skin temperature, electrical activity of a brain, electrical activity of a heart, an eye movement, a facial expression, a pupil dilation, a pupil contraction, a volume of speech, or a rate of speech. ### Claim 7 (depends on 1) A system as recited in claim 1, wherein the instructions further cause the system to generate video segment metadata indicative of the second emotional or cognitive state of the user. ### Claim 8 (depends on 7) A system as recited in claim 7, wherein the instructions further cause the system to determine a gaze target of the user, wherein the video segment metadata indicates the gaze target of the user. ### Claim 9 (depends on 1) A system as recited in claim 1, implemented, at least in part, as a head-mounted display device. ### Claim 10 A method comprising: receiving, from one or more sensors, sensor data; detecting, based at least in part on the sensor data, a change from a first emotional or cognitive state of a user to a second emotional or cognitive state of the user; in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, adding, by one or more processors, at least a portion of video data being captured by a camera to a video segment; detecting, based at least in part on additional sensor data, a change from the second emotional or cognitive state of the user to a third emotional or cognitive state of the user; and cease adding the at least the portion of the video data to the video segment in response to detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user. ### Claim 11 (depends on 10) A method as recited in claim 10, further comprising initiating a recording of the video segment in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user. ### Claim 12 (depends on 10) A method as recited in claim 10, wherein: the video segment begins at a first time associated with the detecting of the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user; and the video segment ends at a second time associated with the detecting of the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user. ### Claim 13 (depends on 10) A method as recited in claim 10, further comprising obtaining, by the one or more sensors, the sensor data and the additional sensor data based on one or more of: a galvanic skin response, a skin temperature, electrical activity of a brain, electrical activity of a heart, an eye movement, a facial expression, a pupil dilation, a pupil contraction, a volume of speech, or a rate of speech. ### Claim 14 (depends on 10) A method as recited in claim 10, further comprising generating video segment metadata indicative of the second emotional or cognitive state of the user. ### Claim 15 (depends on 14) A method as recited in claim 14, further comprising determining a gaze target of the user, wherein the video segment metadata indicates the gaze target of the user. ### Claim 16 A system comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the system to: receive, from one or more sensors, sensor data; detect, based at least in part on the sensor data, a change from a first emotional or cognitive state of a user to a second emotional or cognitive state of the user; in response to detecting the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user, initiate recording or storing of a video segment that includes video data being captured by a camera; detect, based at least in part on additional sensor data, a change from the second emotional or cognitive state of the user to a third emotional or cognitive state of the user; and cease the recording or storing of the video segment based at least in part on detecting the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user. ### Claim 17 (depends on 16) A system as recited in claim 16, wherein: the video segment begins at a first time associated with the detecting of the change from the first emotional or cognitive state of the user to the second emotional or cognitive state of the user; and the video segment ends at a second time associated with the detecting of the change from the second emotional or cognitive state of the user to the third emotional or cognitive state of the user. ### Claim 18 (depends on 16) A system as recited in claim 16, wherein the instructions further cause the system to generate video segment metadata indicative of the second emotional or cognitive state of the user. ### Claim 19 (depends on 18) A system as recited in claim 18, wherein the instructions further cause the system to determine a gaze target of the user, wherein the video segment metadata indicates the gaze target of the user. ### Claim 20 (depends on 16) A system as recited in claim 16, implemented, at least in part, as a head-mounted display device.