# US 2024/0004476 A1 · Gesture detection via image capture of subdermal tissue from a wrist-pointing camera system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004476 A1
- Document ID: US20240004476A1
- Status: Published application
- Kind: A1
- Application number: 17/809,772
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Dongeek Shin; Andrea Colaco; Stiven Guillaume Francois Morvan; Adam James Banfield; Shahram Izadi
- Web page: https://patentgrep.com/patent/US20240004476A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06V 10/766: using regression, e.g. by projecting features on hyperplanes
- G06V 10/82: using neural networks
- G06V 20/20: in augmented reality scenes
- G06V 40/1318: {using electro-optical elements or layers, e.g. electroluminescent sensing}
- G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques of operating an AR system include determining hand gestures formed by a user based on a sequence of two-dimensional images through skin of the user's wrist acquired from a near-infrared camera. Specifically, an image capture device disposed on a band worn around a user's wrist includes a source of electromagnetic radiation, e.g., light-emitting diodes in the infrared (IR) wavelength band that emit the radiation into the user's wrist and an IR detector which produces the sequence of two-dimensional images of a region within a dermal layer in the user's wrist. From this sequence, gesture detection circuitry determines values of a biological flow metric, e.g., a change in perfusion index (PI) between frames of the sequence, based on a trained model that generates the metric from the sequence. Finally, the gesture detection circuitry maps the values of the biological flow metric to specific hand/finger movements that determine a gesture.

## Claims

### Claim 1

A method comprising:
capturing a sequence of images through skin of a wrist of a user;
determining a biological fluid flow metric based on the sequence of images, including applying a first model that maps the sequence of images to the biological fluid flow metric;
determining a gesture formed by the user based on the biological fluid flow metric; and
triggering execution of a command related to an object being displayed in an augmented reality (AR) system based on the gesture.

### Claim 2 (depends on 1)

The method as in claim 1, wherein determining the gesture formed by the user includes:
applying a second model that maps the biological fluid flow metric to the gesture.

### Claim 3 (depends on 2)

The method as in claim 2, wherein the first model includes a convolutional regression neural network; and
wherein the method further comprises:
training the convolutional regression neural network based on a dataset including sequences of images and corresponding values of the biological fluid flow metric.

### Claim 4 (depends on 1)

The method as in claim 1, wherein the biological fluid flow metric includes changes in a perfusion index between frames of the sequence of images.

### Claim 5 (depends on 1)

The method as in claim 1, wherein capturing the sequence of images through the skin of the wrist includes:
emitting electromagnetic radiation from a radiation source into an interior of the wrist;
receiving the electromagnetic radiation reflected from the interior of the wrist into a radiation detector; and
forming the sequence of images by sampling the electromagnetic radiation reflected from the interior of the wrist into a radiation detector at a specified frame rate.

### Claim 6 (depends on 5)

The method as in claim 5, wherein the radiation source includes an infrared (IR) projector mounted in a camera, the camera including the radiation detector.

### Claim 7 (depends on 5)

The method as in claim 5, wherein the radiation source includes a left radiation source and a right radiation source; and
wherein emitting the electromagnetic radiation from the radiation source into the interior of the wrist includes:
emitting electromagnetic radiation from the left radiation source into a left side of the interior of the wrist and from a right radiation source into a right side of the interior of the wrist.

### Claim 8 (depends on 5)

The method as in claim 5, wherein the radiation detector includes a left channel detector and a right channel detector; and
wherein receiving the electromagnetic radiation reflected from the interior of the wrist into the radiation detector includes:
receiving the electromagnetic radiation reflected from the interior of the wrist at the left channel detector and the right channel detector.

### Claim 9 (depends on 8)

The method as in claim 8, wherein sampling the electromagnetic radiation reflected from the interior of the wrist includes:
capturing a respective image at each of the left channel detector and the right channel detector substantially simultaneously.

### Claim 10 (depends on 7)

The method as in claim 7, wherein each of the left radiation source and the right radiation source includes a respective pair of infrared (IR) light-emitting diode (LED).

### Claim 11

An augmented reality (AR) system, comprising:
an image capture device configured to capture a sequence of images through skin of a wrist of a user; and
gesture detection circuitry coupled to a memory, the gesture detection circuitry being configured to:
determine a biological fluid flow metric based on the sequence of images, the gesture detection circuitry being configured to apply a first model that maps the sequence of images to the biological fluid flow metric;
determine a gesture formed by the user based on the biological fluid flow metric; and
trigger execution of a command related to an object being displayed in an augmented reality (AR) system based on the gesture.

### Claim 12 (depends on 11)

The AR system as in claim 11, wherein the image capture device is disposed on a wristband worn around the wrist.

### Claim 13 (depends on 11)

The AR system as in claim 11, wherein the image capture device includes:
a source of electromagnetic radiation, the source being configured to emit the electromagnetic radiation in an infrared (IR) wavelength band; and
a detector configured to detect the electromagnetic radiation reflected from an interior of the wrist, the detector being configured to detect electromagnetic radiation in the IR wavelength band.

### Claim 14 (depends on 13)

The AR system as in claim 13, further comprising:
an IR controller configured to control emission of the electromagnetic radiation by the source of the electromagnetic radiation according to a schedule.

### Claim 15 (depends on 13)

The AR system as in claim 13, wherein the source of electromagnetic radiation and the detector are co-located in a single housing.

### Claim 16 (depends on 13)

The AR system as in claim 13, wherein the detector includes a left channel detector and a right channel detector; and
wherein the source of electromagnetic radiation includes a first pair of IR light-emitting diodes (LEDs) and a second pair of IR LEDs, the first pair of IR LEDs being configured to emit the electromagnetic radiation such that the electromagnetic radiation is received at the left channel detector, the second pair of IR LEDs being configured to emit the electromagnetic radiation such that the electromagnetic radiation is received at the right channel detector.

### Claim 17

A computer program product comprising a nontransitory storage medium, the computer program product including code that, when executed by processing circuitry, causes the processing circuitry to perform a method, the method comprising:
capturing a sequence of images through skin of a wrist of a user;
determining a biological fluid flow metric based on the sequence of images, including applying a first model that maps the sequence of images to the biological fluid flow metric;
determining a gesture formed by the user based on the biological fluid flow metric; and
triggering execution of a command related to an object being displayed in an augmented reality (AR) system based on the gesture.

### Claim 18 (depends on 17)

The computer program product as in claim 17, wherein determining the gesture formed by the user includes:
applying a second model that maps the biological fluid flow metric to the gesture.

### Claim 19 (depends on 18)

The computer program product as in claim 18, wherein the first model includes a convolutional regression neural network; and
wherein the method further comprises:
training the convolutional regression neural network based on a dataset including sequences of two-dimensional images and corresponding values of the biological fluid flow metric.

### Claim 20 (depends on 17)

The computer program product as in claim 17, wherein the biological fluid flow metric includes changes in a perfusion index between frames of the sequence of images.

### Claim 21 (depends on 1)

The method as in claim 1, wherein the first model includes a convolutional regression neural network.

### Claim 22 (depends on 21)

The method as in claim 21, wherein the biological fluid flow metric is evaluated at a set of keypoints in the sequence of images, the set of keypoints being a regressor of the convolutional regression neural network.
