# US 2017/0255837 A1 · Retinal encoder for machine vision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0255837 A1
- Document ID: US20170255837A1
- Status: Published application
- Kind: A1
- Application number: 15/408,178
- Filed: 2017-01-17
- Published: 2017-09-07
- Earliest claimed: 2011-08-25
- Assignees: Cornell University
- Applicants: Cornell University
- Inventors: Sheila NIRENBERG; Illya BOMASH
- Web page: https://patentgrep.com/patent/US20170255837A1

## CPC

- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06T 7/0012: {Biomedical image inspection}
- G06T 9/002: {using neural networks}
- G06T 9/007: {Transform coding, e.g. discrete cosine transform}
- G06V 10/449: {Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- H04N 19/60: using transform coding
- H04N 19/62: by frequency transforming in three dimensions [3D] (H04N19/63 takes precedence)
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- G06T 2207/20024: Filtering details
- G06T 2207/20048: Transform domain processing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30041: Eye; Retina; Ophthalmic
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method is disclosed including: receiving raw image data corresponding to a series of raw images; processing the raw image data with an encoder to generate encoded data, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina; and applying a first machine vision algorithm to data generated based at least in part on the encoded data.

## Claims

### Claim 1

A method including:
receiving raw image data corresponding to a series of raw images;
processing the raw image data with an encoder to generate encoded data, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina; and
applying a first machine vision algorithm to data generated based at least in part on the encoded data.

### Claim 2 (depends on 1)

The method of claim 1, further including generating a series of retinal images based on the encoded data.

### Claim 3 (depends on 2)

The method of claim 2, including determining pixel values in the retinal images based on the encoded data, where determining pixel values in the retinal images based on the encoded data includes determining a pixel intensity or color based on encoded data indicative of a retinal cell response, and where the data indicative of a retinal cell response is indicative of at least one from the list consisting of: a retinal cell firing rate, a retinal cell output pulse train, and a generator potential.

### Claim at position 4

4-5. (canceled)

### Claim 6 (depends on 2)

The method of claim 2, further including applying the first machine vision algorithm to the series of retinal images, where the machine vision algorithm includes at least one select from the list consisting of: an object recognition algorithm, an image classification algorithm, a facial recognition algorithm, an optical character recognition algorithm, a content-based image retrieval algorithm, a pose estimation algorithm, a motion analysis algorithm, an egomotion determination algorithm, a movement tracking algorithm, an optical flow determination algorithm, a scene reconstruction algorithm, a 3D volume recognition algorithm, and a navigation algorithm.

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The method of claim 1, where the machine vision algorithm exhibits better performance when applied to the series of retinal images than when applied to a corresponding set of raw images that have not been processed using the encoder, where the machine vision algorithm exhibits better performance when applied to a series of retinal images including natural scenes than when applied to a corresponding series of raw images that have not been processed using the encoder.

### Claim 9

(canceled)

### Claim 10 (depends on 8)

The method of claim 8, where the machine vision algorithm includes an algorithm for the detection or identification of a human within a series of images; and where the machine vision algorithm exhibits better detection or identification accuracy when applied to a range of retinal images including the human than when applied to a corresponding set of raw images that have not been processed using the encoder, where the series of images including the human includes images of the human located in a natural scene, and where the series of images including the human includes images of the human located in a natural scene that is different from any natural scenes used to train the machine vision algorithm.

### Claim at position 10

11-12. (canceled)

### Claim 13 (depends on 8)

The method of claim 8, where the machine vision algorithm includes an algorithm for navigation through a real or virtual environment, and where the machine vision algorithm exhibits better navigation performance when applied to a series of retinal images including a natural scene than when applied to a corresponding set of raw images that have not been processed using the encoder, where the machine vision algorithm exhibits fewer unwanted collision events during navigation when applied to a series of retinal images including a natural scene than when applied to a corresponding set of raw images that have not been processed using the encoder, and where the series of retinal images correspond to an environment that was not used to train the machine vision algorithm.

### Claim at position 12

14-15. (canceled)

### Claim 16 (depends on 1)

The method of claim 1, further including:
applying a machine imaging algorithm to the series of retinal images to identify one or more retinal images of interest;
identifying one or more raw images of interest corresponding to the retinal images of interest; and
processing the raw images of interest, where the processing the raw images of interest includes applying a second machine vision algorithm to the raw images of interest.

### Claim at position 14

17-18. (canceled)

### Claim 19 (depends on 16)

The method of claim 16, where:
the first machine vision algorithm includes an algorithm that has been trained on a set of retinal images; and
the second machine vision algorithm includes an algorithm that has been trained on a set of raw images.

### Claim 20 (depends on 1)

The method of claim 1, where applying the first machine vision algorithm includes applying a navigation algorithm, where applying the navigation algorithm includes:
processing the series of retinal images to determine motion information indicative of motion at a plurality of image locations in the series of images;
classifying spatial regions in the series of images based on the motion information; and
generating a navigation decision based on the classification of the spatial regions.

### Claim 21

(canceled)

### Claim 22 (depends on 20)

The method of claim 20, where motion information is indicative of an optical flow in the series of images, the method further including:
using a convolutional neural network to classify the spatial regions; and
controlling the motion of a robotic apparatus based on results from navigation algorithm.

### Claim at position 19

23-24. (canceled)

### Claim 25 (depends on 20)

The method of claim 20, further including controlling the motion of a virtual object in a virtual space based on results from navigation algorithm, wherein the navigation algorithm was trained based on image data representative of a virtual space.

### Claim 26

(canceled)

### Claim 27 (depends on 1)

The method of claim 1, further including training a machine vision algorithm based on the retinal images, where training the machine vision algorithm includes:
(i) applying the machine vision algorithm to a set of retinal images to generate an output;
(ii) determining performance information indicative of the performance of the machine vision algorithm based on the output;
(iii) modifying one or more characteristics of the machine vision algorithm based on the performance information; and
iteratively repeating steps (i) through (iii) until a selected performance criteria is reached; and
where the trained machine vision algorithm is characterized by a set of parameters, and where the parameters differ from the corresponding parameters that would be obtained by equivalent training of the machine vision algorithm using raw images corresponding to the retinal images.

### Claim at position 23

28-30. (canceled)

### Claim 31 (depends on 6)

The method of claim 6, where:
processing the raw image data with an encoder to generate encoded data includes generating encoded data that contains a reduced amount of information relative to the corresponding raw image data;
where the machine vision algorithm exhibits better performance when applied to the series of retinal images than when applied to a corresponding set of raw images that have not been processed using the encoder, and where the amount of information contained in the encoded data is compressed by a factor of at least about 2 relative to the corresponding raw image data.

### Claim 32

(canceled)

### Claim 33 (depends on 31)

The method claim 31, where the amount of information contained in the encoded data is compressed by a factor of at least about 5 relative to the corresponding raw image data.

### Claim 34 (depends on 31)

The method claim 31, where the amount of information contained in the encoded data is compressed by a factor of at least about 10 relative to the corresponding raw image data.

### Claim 35 (depends on 1)

The method of claim 1, where the vertebrate includes at least one selected from the list consisting of: a mouse, and a monkey, where the retinal cells include ganglion cells, where the retinal cells include at least two classes of cells, and where the at least two classes of cells includes ON cells and OFF cells.

### Claim at position 29

36-38. (canceled)

### Claim 39 (depends on 1)

The method of claim 1, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina over a range of input that includes natural scene images, including spatio-temporally varying images.

### Claim 40

An apparatus including:
at least one memory storage device configured to store raw image data; and
at least one processor operably coupled with the memory and programmed to:
receive raw image data corresponding to a series of raw images;
process the raw image data with an encoder to generate encoded data, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina; and
apply a first machine vision algorithm to data generated based at least in part on the encoded data.

### Claim 41

A non-transitory computer-readable medium having computer-executable instructions for implementing operations comprising
receiving raw image data corresponding to a series of raw images;
processing the raw image data with an encoder to generate encoded data, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina; and
applying a first machine vision algorithm to data generated based at least in part on the encoded data.
