# US 2022/0044406 A1 · Neural network for eye image segmentation and image quality estimation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044406 A1
- Document ID: US20220044406A1
- Status: Published application
- Kind: A1
- Application number: 17/407,763
- Filed: 2021-08-20
- Published: 2022-02-10
- Earliest claimed: 2016-09-29
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Alexey Spizhevoy; Adrian Kaehler; Vijay Badrinarayanan
- Web page: https://patentgrep.com/patent/US20220044406A1

## CPC

- G06T 7/12: Edge-based segmentation
- G06F 18/2413: based on distances to training or reference patterns
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 10/993: {Evaluation of the quality of the acquired pattern}
- G06V 40/193: {Preprocessing; Feature extraction}
- G06V 40/197: {Matching; Classification}
- G06T 2207/10024: Color image
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30041: Eye; Retina; Ophthalmic
- G06T 2207/30168: Image quality inspection
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A method for training a convolutional neural network for eye image segmentation and image quality estimation, the method being performed by a system of one or more processors, and the method comprising:
obtaining a training set of eye images;
providing a convolutional neural network with the training set of eye images; and
training the convolutional neural network with the training set of eye images,
wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein an output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,
and wherein the output layer of the shared layers is connected to an input layer of the quality estimation layer.

### Claim 22 (depends on 21)

The method of claim 21, wherein training the convolutional neural network with the training set of eye images comprises:
processing an eye image of the training set using the segmentation tower to generate a segmentation of the eye image;
processing the eye image of the training set using the quality estimation tower to generate a quality estimation of the eye image;
computing a first difference between the segmentation of the eye image and a reference segmentation of the eye image; computing a second difference between the quality estimation of the eye image and a reference quality estimation of the eye image; and
updating parameters of the convolutional neural network using the first difference and the second difference.

### Claim 23 (depends on 22)

The method of claim 22, wherein updating the parameters of the convolutional neural network using the first difference and the second difference comprises setting a first percentage of the parameters of the convolutional neural network to values of zero during a first training cycle when training the convolutional neural network.

### Claim 24 (depends on 23)

The method of claim 23, wherein setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network comprises randomly setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network.

### Claim 25 (depends on 23)

The method of claim 23, wherein updating the parameters of the convolutional neural network using the first difference and the second difference further comprises setting a second percentage of the parameters of the convolutional neural network to values of zero during a second training cycle when training the convolutional neural network.

### Claim 26 (depends on 25)

The method of claim 25, wherein setting the second percentage of the parameters of the convolutional neural network to values of zero during the second training cycle when training the convolutional neural network comprises randomly setting the second percentage of the parameters of the convolutional neural network to values of zero during the second training cycle when training the convolutional neural network.

### Claim 27 (depends on 26)

The method of claim 26, wherein the first percentage or the second percentage is between 50% and 30%.

### Claim 28 (depends on 22)

The method of claim 22, wherein the segmentation of the eye image comprises a background, a sclera, an iris, or a pupil of the eye image, and wherein the reference segmentation of the eye image comprises a reference background, a reference sclera, a reference iris, or a reference pupil of the eye image.

### Claim 29 (depends on 21)

The method of claim 21, wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, a softsign layer, or any combination thereof.

### Claim 30 (depends on 21)

The method of claim 21, wherein an output of a first intermediate layer of the shared layers is connected to a third input layer of the segmentation layers.

### Claim 31

A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform operations comprising:
obtaining a training set of eye images;
providing a convolutional neural network with the training set of eye images; and
training the convolutional neural network with the training set of eye images,
wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein an output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,
and wherein the output layer of the shared layers is connected to an input layer of the quality estimation layer.

### Claim 32 (depends on 31)

The system of claim 31, wherein training the convolutional neural network with the training set of eye images comprises:
processing an eye image of the training set using the segmentation tower to generate a segmentation of the eye image;
processing the eye image of the training set using the quality estimation tower to generate a quality estimation of the eye image;
computing a first difference between the segmentation of the eye image and a reference segmentation of the eye image; computing a second difference between the quality estimation of the eye image and a reference quality estimation of the eye image; and
updating parameters of the convolutional neural network using the first difference and the second difference.

### Claim 33 (depends on 32)

The system of claim 32, wherein updating the parameters of the convolutional neural network using the first difference and the second difference comprises setting a first percentage of the parameters of the convolutional neural network to values of zero during a first training cycle when training the convolutional neural network.

### Claim 34 (depends on 33)

The system of claim 33, wherein setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network comprises randomly setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network.

### Claim 35 (depends on 33)

The system of claim 33, wherein updating the parameters of the convolutional neural network using the first difference and the second difference further comprises setting a second percentage of the parameters of the convolutional neural network to values of zero during a second training cycle when training the convolutional neural network.

### Claim 36 (depends on 32)

The system of claim 32, wherein the segmentation of the eye image comprises a background, a sclera, an iris, or a pupil of the eye image, and wherein the reference segmentation of the eye image comprises a reference background, a reference sclera, a reference iris, or a reference pupil of the eye image.

### Claim 37 (depends on 32)

The system of claim 32, wherein an output of a first intermediate layer of the shared layers is connected to a third input layer of the segmentation layers.

### Claim 38

Non-transitory computer storage media storing instructions that when executed by a system of one or more processors, cause the one or more processors to perform operations comprising:
obtaining a training set of eye images;
providing a convolutional neural network with the training set of eye images; and
training the convolutional neural network with the training set of eye images,
wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein an output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,
and wherein the output layer of the shared layers is connected to an input layer of the quality estimation layer.

### Claim 39 (depends on 38)

The computer storage media of claim 38, wherein training the convolutional neural network with the training set of eye images comprises:
processing an eye image of the training set using the segmentation tower to generate a segmentation of the eye image;
processing the eye image of the training set using the quality estimation tower to generate a quality estimation of the eye image;
computing a first difference between the segmentation of the eye image and a reference segmentation of the eye image; computing a second difference between the quality estimation of the eye image and a reference quality estimation of the eye image; and
updating parameters of the convolutional neural network using the first difference and the second difference.

### Claim 40 (depends on 38)

The computer storage media of claim 38, wherein an output of a first intermediate layer of the shared layers is connected to a third input layer of the segmentation layers.
