# US 2019/0174039 A1 · Bio-sensing and eye-tracking system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0174039 A1
- Document ID: US20190174039A1
- Status: Published application
- Kind: A1
- Application number: 16/325,567
- Filed: 2017-08-15
- Published: 2019-06-06
- Earliest claimed: 2016-08-15
- Assignees: The Regents of the University of California
- Applicants: The Regents of the University of California
- Inventors: Tzyy-Ping Jung; Siddharth Siddharth; Terrence Sejnowski
- Web page: https://patentgrep.com/patent/US20190174039A1

## CPC

- H04N 23/57: Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/18: Eye characteristics, e.g. of the iris
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and devices are disclosed for implementing a low-cost wearable multi-modal bio-sensing system capable of recording bio-markers and eye-gaze overlaid on world view in real-world settings. In an exemplary embodiment, a bio-sensing system uses at least two cameras and one or more bio-sensors to record a variety of events and bio-marker data. In another exemplary embodiment, the recorded information is used to track the eye position, calculate the pupil dimensions, and calculate the gaze of a human being. The eye position and dimensions of the pupil can be used for emotion recognition.

## Claims

### Claim 1

A bio-sensing system, comprising:
a frame structured to be placed on a user's face;
a first camera coupled to the frame and facing towards an eye of the user to capture a first set of images of the eye;
a second camera coupled to the frame and facing away from the user and configured to capture a second set of images of an environment from the user's perspective;
one or more sensors configured to measure biological functions of the user and to generate sensor data; and
a computer electrically coupled to the one or more sensors, the first camera and the second camera.

### Claim 2 (depends on 1)

The bio-sensing system of claim 1, wherein the computer includes at least one processor in communication with a memory operable to execute instructions to cause the computer to perform a method comprising:
receiving the first set of images from the first camera;
detecting location of a pupil of the eye based on the received first set of images; and
calculating coordinates of the pupil and a diameter of the pupil based on the received first set of images.

### Claim 3 (depends on 2)

The bio-sensing system of claim 2, wherein the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:
calculating a confidence of the pupil being present in the first set of images.

### Claim 4 (depends on 2)

The bio-sensing system of claim 2, wherein the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:
receiving the second set of images from the second camera;
detecting one or more objects based on the received second set of images;
transmitting the coordinates of the pupil and the second set of images to a remote computer; and
transmitting coordinates of the detected one or more objects to the remote computer, wherein the remote computer includes at least one second processor in communication with a second memory operable to execute instructions to cause the remote computer to perform a method comprising:
highlighting on a display the detected one or more objects.

### Claim 5 (depends on 4)

The bio-sensing system of claim 4, wherein the at least one processor is operable to execute instructions to further cause the computer to perform a method comprising:
determining user excitement by measuring a rate of increase of the diameter of the pupil

### Claim 6

(canceled)

### Claim 7 (depends on 4)

The bio-sensing system of claim 4, wherein the remote computer includes at least one second processor in communication with the second memory operable to execute instructions to cause the remote computer to perform a method comprising:
receiving the coordinates of the pupil and the second set of images;
prompting the user to look at one or more locations;
calibrating a gaze of the user based on the calculated coordinates of the pupil and based on the user's pupil response to the prompting to look at the one or more locations; and
overlaying a position of the pupil onto the second set of images.

### Claim 8 (depends on 1)

The bio-sensing system of claim 1, wherein
the one or more sensors comprises a photoplethysmogram (PPG) sensor and an accelerometer, wherein
the PPG sensor and the accelerometer are structured to be attached to an earlobe of the user,
the PPG sensor is configured to generate PPG sensor data by capturing infrared light reflected from the earlobe of the user,
the accelerometer is configured to generate accelerometer data by detecting movement of the user;
the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:
filtering the PPG sensor data;
filtering the accelerometer data;
converting the filtered PPG sensor data to digital PPG sensor data;
converting the filtered accelerometer data to digital accelerometer data;
estimating, based on the digital accelerometer data, noise contributed from user movement; and
removing from the digital PPG sensor data the noise contributed from user movement to obtain a noise filtered PPG sensor data.

### Claim 9

(canceled)

### Claim 10

(canceled)

### Claim 11

(canceled)

### Claim 12 (depends on 8)

The bio-sensing system of claim 8, wherein
the one or more sensors further comprises an electroencephalogram (EEG) sensor, wherein the EEG sensor is configured to generate an EEG signal based on electrical activity of a brain of the user;
the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:
filtering the EEG signal to remove noise;
performing fast Fourier transform (FFT) on the filtered EEG signal to obtain an FFT EEG signal;
calculating biomarkers from the FFT EEG signal;
calculating heart rate based on the noise filtered PPG sensor data; and
measuring heart rate variability based on the noise filtered PPG sensor data.

### Claim 13 (depends on 12)

The bio-sensing system of claim 12, wherein the at least one processor is operable to execute instructions to further cause the computer to perform the method comprising:
recognizing an emotion of the user based on the determined user excitement, the calculated heart rate, the measured heart rate variability, and the calculated biomarkers; and
tagging the emotion of the user associated with the object seen by the user.

### Claim 14 (depends on 13)

The bio-sensing system of claim 13, wherein the tagging of the emotion with the object includes recording a time when the user saw the object.

### Claim 15

(canceled)

### Claim 16

(canceled)

### Claim 17 (depends on 1)

The bio-sensing system of claim 1, wherein the first camera is an infrared camera, and wherein the second camera includes a wide-angle lens.

### Claim 18

(canceled)

### Claim 19

(canceled)

### Claim 20

(canceled)

### Claim 21

A method of operating a bio-sensing system, the method comprising:
receiving, by a first computer, a first set of images from a first camera;
detecting, by the first computer, a location of a pupil of an eye based on the received first set of images; and
calculating, by the first computer, coordinates of the pupil and a diameter of the pupil based on the received first set of images;
receiving, by the first computer, a second set of images from a second camera; and
transmitting, by the first computer, the coordinates of the pupil and the second set of images to a remote computer.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
calculating a confidence of the pupil being present in the first set of images.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
determining user excitement by measuring a rate of increase of the diameter of the pupil.

### Claim 24 (depends on 21)

The method of claim 21, further comprising:
detecting, by the first computer, one or more objects based on the received second set of images; and transmitting coordinates of the detected one or more objects to the remote computer.

### Claim 25

(canceled)

### Claim 26 (depends on 21)

The method of claim 21, further comprising:
filtering PPG sensor data received from a photoplethysmogram (PPG) sensor;
filtering accelerometer data received from an accelerometer;
converting the filtered PPG sensor data to digital PPG sensor data;
converting the filtered accelerometer data to digital accelerometer data;
estimating, based on the digital accelerometer data, noise contributed from user movement; and
removing from the digital PPG sensor data the noise contributed from user movement to obtain a noise filtered PPG sensor data.

### Claim 27

(canceled)

### Claim 28

(canceled)

### Claim 29 (depends on 26)

The method of claim 26, further comprising:
receiving an electroencephalogram (EEG) signal from an EEG sensor;
filtering the EEG signal to remove noise;
performing fast Fourier transform (FFT) on the filtered EEG signal to obtain an FFT EEG signal;
calculating biomarkers from the FFT EEG signal;
calculating heart rate based on the noise filtered PPG sensor data; and
measuring heart rate variability based on the noise filtered PPG sensor data.

### Claim 30 (depends on 29)

The method of claim 29, further comprising:
recognizing an emotion of the user based on the determined user excitement, the calculated heart rate, the measured heart rate variability, and the calculated biomarkers; and
tagging the emotion of the user associated with the object seen by the user.

### Claim 31 (depends on 30)

The method of claim 30, wherein the tagging of the emotion with the object includes recording a time when the user saw the object.

### Claim 32

(canceled)

### Claim 33

(canceled)

### Claim 34 (depends on 21)

The method of claim 21, wherein the first camera is an infrared camera, and wherein the second camera includes a wide-angle lens.

### Claim 35

(canceled)
