# US 2019/0377409 A1 · Neural network image processing apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377409 A1
- Document ID: US20190377409A1
- Status: Published application
- Kind: A1
- Application number: 16/005,610
- Filed: 2018-06-11
- Published: 2019-12-12
- Earliest claimed: 2018-06-11
- Assignees: FotoNation Limited
- Applicants: FotoNation Limited
- Inventors: Joseph LEMLEY; Liviu-Cristian DUTU; Stefan MATHE; Madalin DUMITRU-GUZU
- Web page: https://patentgrep.com/patent/US20190377409A1

## CPC

- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/166: {using acquisition arrangements}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/19: Sensors therefor
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A neural network image processing apparatus arranged to acquire images from an image sensor and to: identify a ROI containing a face region in an image; determine at plurality of facial landmarks in the face region; use the facial landmarks to transform the face region within the ROI into a face region having a given pose; and use transformed landmarks within the transformed face region to identify a pair of eye regions within the transformed face region. Each identified eye region is fed to a respective first and second convolutional neural network, each network configured to produce a respective feature vector. Each feature vector is fed to respective eyelid opening level neural networks to obtain respective measures of eyelid opening for each eye region. The feature vectors are combined and to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values.

## Claims

### Claim 1

A neural network image processing apparatus arranged to acquire one or more images from an image sensor, the apparatus being configured to:
identify at least one region of interest containing a face region in at least one of said one or more images;
determine at plurality of facial landmarks in a face region within a region of interest;
use said plurality of said facial landmarks to transform said face region within said region of interest into a face region having a given pose;
use transformed landmarks within said transformed face region to identify a pair of eye regions within said transformed face region;
feed each identified eye region of said pair of eye regions to a respective first and second convolutional neural network, each network configured to produce a respective feature vector comprising a plurality of numerical values;
feed each feature vector to respective eyelid opening level neural networks to obtain respective measures of eyelid opening for each eye region;
combine the feature vectors; and
feed the concatenated vector to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values, wherein said first and second eyelid opening and gaze angle neural networks are jointly trained based on a common training set.

### Claim 2 (depends on 1)

The apparatus according to claim 1 further comprising a multi-processor neural network core which substantially simultaneously processes layers from said first and second convolutional networks and which substantially simultaneously processes layers from said first and second eyelid opening and gaze angle neural networks.

### Claim 3 (depends on 1)

The apparatus according to claim 1 wherein each of said feature vectors comprises a plurality of floating point numbers.

### Claim 4 (depends on 3)

The apparatus according to claim 3 wherein each of said feature vectors comprise at least 128 32-bit floating point numbers.

### Claim 5 (depends on 1)

The apparatus according to claim 1 wherein said given pose is a front facing pose.

### Claim 6 (depends on 1)

The apparatus according to claim 1 wherein said pair of eye regions include an area including any eye related or eyebrow related landmarks.

### Claim 7 (depends on 1)

The apparatus according to claim 1 wherein said pair of eye regions are rectangular.

### Claim 8 (depends on 7)

The apparatus according to claim 7 wherein said pair of eye regions have a fixed aspect ratio.

### Claim 9 (depends on 1)

The apparatus according to claim 1 wherein each of said first and second convolutional neural networks comprises respective combinations of convolutional and pooling layers.

### Claim 10 (depends on 9)

The apparatus according to claim 9 wherein each convolution layer includes an activation function.

### Claim 11 (depends on 1)

The apparatus according to claim 1 wherein said gaze angle network comprises only fully connected layers.

### Claim 12 (depends on 1)

The apparatus according to claim 1 wherein said eyelid opening network comprises only fully connected layers.

### Claim 13 (depends on 1)

The apparatus according to claim 1 wherein said landmarks delineate any combination of a face region: jaw, mouth, nose, eyes or eyebrows.

### Claim 14 (depends on 1)

The apparatus according to claim 1 wherein said landmarks do not delineate any feature on an eye.

### Claim 15 (depends on 1)

The apparatus according to claim 1 wherein said apparatus is configured to combine said feature vectors by concatenating the feature vectors.

### Claim 16 (depends on 1)

The apparatus according to claim 1 wherein said feature vectors comprise either 1×M or A×B=M values.
