# US 10,310,073 B1 · System and method for determining engagement level of a human being using a millimeter-wave radar sensor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,073 B1 - Document ID: US10310073B1 - Status: Granted - Kind: B1 - Application number: 15/891,086 - Filed: 2018-02-07 - Granted: 2019-06-04 - Earliest claimed: 2018-02-07 - Assignees: Infineon Technologies AG - Applicants: Infineon Technologies AG - Inventors: Avik Santra; Ashutosh Baheti - Web page: https://patentgrep.com/patent/US10310073B1 ## CPC - G01S 13/89: for mapping or imaging - A61B 5/024: Measuring pulse rate or heart rate - A61B 5/0507: using microwaves or terahertz waves - A61B 5/0816: {Measuring devices for examining respiratory frequency (measuring frequency of electric signals G01R23/00)} - A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety} - G01S 7/411: {Identification of targets based on measurements of radar reflectivity (G01S7/415 takes precedence)} - G01S 13/536: using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves - G01S 13/56: for presence detection {(presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26)} - G06V 10/82: using neural networks - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/174: {Facial expression recognition} - H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)} - A61B 5/00: Measuring for diagnostic purposes (radiation diagnosis A61B6/00; diagnosis by ultrasonic, sonic or infrasonic waves A61B8/00); Identification of persons - A61B 2503/12: Healthy persons not otherwise provided for, e.g. subjects of a marketing survey - G06F 2218/10: by analysing the shape of a waveform, e.g. extracting parameters relating to peaks - G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An embodiment method may include receiving radar data at a millimeter-wave radar sensor, the radar data being generated in response to an incident radio-frequency signal reflecting off a biological target located in a field of view of the millimeter-wave radar sensor; extracting a filtered vital-Doppler signal from the radar data; determining an emotion of the biological target based on the filtered vital-Doppler signal; and generating a user-engagement rating indicative of a level of engagement of the biological target in an activity performed in the field of view based on the emotion. ## Claims ### Claim 1 A method, comprising: receiving radar data at a millimeter-wave radar sensor, the radar data being generated in response to an incident radio-frequency signal reflecting off a biological target located in a field of view of the millimeter-wave radar sensor; extracting a filtered vital-Doppler signal from the radar data, wherein extracting the filtered vital-Doppler signal from the radar data comprises: capturing the radar data corresponding to the biological target across a vital-Doppler frame; performing a first filtering on the vital-Doppler frame to produce a first-filtered vital-Doppler frame; and performing a second filtering on the vital-Doppler frame to produce a second-filtered vital-Doppler frame; designating at least one of the first-filtered vital-Doppler frame or the second-filtered vital-Doppler frame as a time-domain vital-Doppler signal; filtering the time-domain vital-Doppler signal with a window function that is localized in time and frequency to generate a windowed time-frequency signal, the filtering the time-domain vital-Doppler signal comprising convolving the time-domain vital-Doppler signal with the window function; determining initial filter coefficients of a frequency filter; performing a Gabor time-frequency transform on the windowed time-frequency signal; filtering a result of the Gabor time-frequency transform step using the frequency filter having the initial filter coefficients; performing an inverse Gabor time-frequency transform on an output of the frequency filter to generate an interim filtered vital-Doppler signal; determining a ratio of an out-of-band energy of the interim filtered vital-Doppler signal to a total energy of the interim filtered vital-Doppler signal; and in response to the ratio being greater than a threshold, iteratively performing the following steps: selecting new filter coefficients of the frequency filter based on a predetermined sequence of filter coefficients; performing the Gabor time-frequency transform; filtering the result of the Gabor time-frequency transform step using the frequency filter having the new filter coefficient; and performing the inverse Gabor time-frequency transform; determining an emotion of the biological target based on the filtered vital-Doppler signal; and generating a user-engagement rating indicative of a level of engagement of the biological target in an activity performed in the field of view based on the emotion. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the emotion of the biological target based on the filtered vital-Doppler signal comprises: determining a number of times a respective emotion of a plurality of emotions occurred during the activity. ### Claim 3 (depends on 2) The method of claim 2, wherein generating the user-engagement rating comprises: generating the user-engagement rating based on the plurality of emotions and the number of times each of the plurality of emotions occurred during the activity. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the emotion of the biological target based on the filtered vital-Doppler signal comprises: determining a time instance at which a respective emotion of a plurality of emotions occurred during the activity; and determining a probability of the respective emotion occurring at the time instance. ### Claim 5 (depends on 4) The method of claim 4, wherein generating the user-engagement rating comprises: generating the user-engagement rating based on the plurality of emotions, the time instance associated with each of the plurality of emotions, and the probability associated with each of the plurality of emotions. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the emotion of the biological target based on the filtered vital-Doppler signal comprises determining respective emotions of each of a plurality of biological targets based on the filtered vital-Doppler signal. ### Claim 7 (depends on 6) The method of claim 6, wherein generating the user-engagement rating indicative of the level of engagement of the biological target in the activity based on the emotion comprises generating a respective user-engagement rating indicative of a respective level of engagement of each of the plurality of biological targets based on the respective emotions of each of the plurality of biological targets. ### Claim 8 (depends on 7) The method of claim 7, further comprising generating a collective statistic indicative of the level of engagement of the plurality of biological targets based on the respective user-engagement ratings. ### Claim 9 (depends on 1) The method of claim 1, wherein the first filtering comprises a band-pass filtering centered at 0.8 Hz with a bandwidth of 0.6 Hz. ### Claim 10 (depends on 1) The method of claim 1, wherein the second filtering comprises a band-pass filtering centered at 2 Hz with a bandwidth of 3 Hz. ### Claim 11 (depends on 1) The method of claim 1, wherein a result of the inverse Gabor time-frequency transform is designated as the filtered vital-Doppler signal in response to the ratio being less than the threshold. ### Claim 12 (depends on 1) The method of claim 1, wherein the out-of-band energy of the interim filtered vital-Doppler signal comprises an energy of the interim filtered vital-Doppler signal outside a frequency range of 0.6 Hz to 5 Hz. ### Claim 13 (depends on 1) The method of claim 1, wherein determining the emotion of the biological target based on the filtered vital-Doppler signal comprises: performing a set of reference radar measurements for a plurality of reference emotion labels on the biological target using the millimeter-wave radar sensor; producing a training data set for the plurality of reference emotion labels based on the set of reference radar measurements; forming a set of stored feature vectors based on the training data set; and associating a respective emotion label to each stored feature vector of the set of stored feature vectors. ### Claim 14 (depends on 13) The method of claim 13, wherein forming the set of stored feature vectors comprises iteratively adjusting the forming of the set stored feature vectors using a machine learning algorithm. ### Claim 15 (depends on 14) The method of claim 14, wherein determining the emotion of the biological target based on the filtered vital-Doppler signal comprises: extracting live feature vectors from the filtered vital-Doppler signal; and determining the emotion of the biological target based on the live feature vectors and the set stored feature vectors using the machine learning algorithm. ### Claim 16 A system, comprising: a processing system configured to be coupled to a millimeter-wave radar sensor, wherein the processing system is configured to: instruct the millimeter-wave radar sensor to transmit a series of chirps within a field of view of the millimeter-wave radar sensor; identify a set of targets within the field of view based on radar data received by the millimeter-wave radar sensor and in response to transmission of the series of chirps; extract a filtered vital-Doppler signal from the radar data, wherein extracting the filtered vital-Doppler signal from the radar data comprises: capturing the radar data corresponding to the set of targets across a vital-Doppler frame; performing a first filtering on the vital-Doppler frame to produce a first-filtered vital-Doppler frame; and performing a second filtering on the vital-Doppler frame to produce a second-filtered vital-Doppler frame; designating at least one of the first-filtered vital-Doppler frame or the second-filtered vital-Doppler frame as a time-domain vital-Doppler signal; filtering the time-domain vital-Doppler signal with a window function that is localized in time and frequency to generate a windowed time-frequency signal, the filtering the time-domain vital-Doppler signal comprising convolving the time-domain vital-Doppler signal with the window function; determining initial filter coefficients of a frequency filter; performing a Gabor time-frequency transform on the windowed time-frequency signal; filtering a result of the Gabor time-frequency transform step using the frequency filter having the initial filter coefficients; performing an inverse Gabor time-frequency transform on an output of the frequency filter to generate an interim filtered vital-Doppler signal; determining a ratio of an out-of-band energy of the interim filtered vital-Doppler signal to a total energy of the interim filtered vital-Doppler signal; and in response to the ratio being greater than a threshold, iteratively performing the following steps: selecting new filter coefficients of the frequency filter based on a predetermined sequence of filter coefficients; performing the Gabor time-frequency transform; filtering the result of the Gabor time-frequency transform step using the frequency filter having the new filter coefficient; and performing the inverse Gabor time-frequency transform; determine an emotion of the set of targets based on the filtered vital-Doppler signal; and generate a user-engagement rating indicative of a level of engagement of the set of targets in an activity performed in the field of view based on the emotion. ### Claim 17 A non-transitory computer readable storage medium with an executable program stored thereon, the executable program including instructions to: instruct a millimeter-wave radar sensor to transmit a series of chirps within a field of view of the millimeter-wave radar sensor; identify a biological target within the field of view based on radar data received by the millimeter-wave radar sensor and in response to transmission of the series of chirps; extract a filtered vital-Doppler signal from the radar data, wherein extracting the filtered vital-Doppler signal from the radar data comprises: capturing the radar data corresponding to the biological target across a vital-Doppler frame; performing a first filtering on the vital-Doppler frame to produce a first-filtered vital-Doppler frame; and performing a second filtering on the vital-Doppler frame to produce a second-filtered vital-Doppler frame; designating at least one of the first-filtered vital-Doppler frame or the second-filtered vital-Doppler frame as a time-domain vital-Doppler signal; filtering the time-domain vital-Doppler signal with a window function that is localized in time and frequency to generate a windowed time-frequency signal, the filtering the time-domain vital-Doppler signal comprising convolving the time-domain vital-Doppler signal with the window function; determining initial filter coefficients of a frequency filter; performing a Gabor time-frequency transform on the windowed time-frequency signal; filtering a result of the Gabor time-frequency transform step using the frequency filter having the initial filter coefficients; performing an inverse Gabor time-frequency transform on an output of the frequency filter to generate an interim filtered vital-Doppler signal; determining a ratio of an out-of-band energy of the interim filtered vital-Doppler signal to a total energy of the interim filtered vital-Doppler signal; and in response to the ratio being greater than a threshold, iteratively performing the following steps: selecting new filter coefficients of the frequency filter based on a predetermined sequence of filter coefficients; performing the Gabor time-frequency transform; filtering the result of the Gabor time-frequency transform step using the frequency filter having the new filter coefficient; and performing the inverse Gabor time-frequency transform; determine an emotion of the biological target based on the filtered vital-Doppler signal; and generate a user-engagement rating indicative of a level of engagement of the biological target in an activity performed in the field of view based on the emotion.