# US 10,769,483 B2 · Retinal encoder for machine vision > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,483 B2 - Document ID: US10769483B2 - Status: Granted - Kind: B2 - Application number: 16/421,147 - Filed: 2019-05-23 - Published: 2019-09-12 - Granted: 2020-09-08 - Earliest claimed: 2011-08-25 - Assignees: Cornell University - Applicants: Cornell University - Inventors: Shelia Nirenberg; Illya Bomash - Web page: https://patentgrep.com/patent/US10769483B2 ## CPC - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs - G06T 7/0012: {Biomedical image inspection} - G06T 9/002: {using neural networks} - G06T 9/007: {Transform coding, e.g. discrete cosine transform} - G06V 10/449: {Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - H04N 19/60: using transform coding - H04N 19/62: by frequency transforming in three dimensions [3D] (H04N19/63 takes precedence) - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - G06T 2207/20024: Filtering details - G06T 2207/20048: Transform domain processing - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06V 2201/03: Recognition of patterns in medical or anatomical images - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method is disclosed including: receiving raw image data corresponding to a series of raw images; processing the raw image data with an encoder to generate encoded data, where the encoder is characterized by an input/output transformation that substantially mimics the input/output transformation of one or more retinal cells of a vertebrate retina; and applying a first machine vision algorithm to data generated based at least in part on the encoded data. ## Claims ### Claim 1 A method including: applying, by an encoding module, a spatiotemporal transformation to image data to generate retinal output cell response values; generating, by the encoding module, encoded data based on the retinal output cell response values; applying, by a machine vision module, a machine vision algorithm to the encoded data; monitoring, by a controller, performance of the machine vision algorithm, wherein monitoring the performance of the machine vision algorithm comprises: calculating an error rate of the machine vision algorithm; and comparing the error rate to a threshold level; and adjusting, by the controller, the machine vision algorithm based on the monitored performance. ### Claim 2 (depends on 1) The method of claim 1, further comprising generating retinal images from the retinal output cell response values, and wherein applying the machine vision algorithm to the encoded data comprises applying the machine vision algorithm to the retinal images. ### Claim 3 (depends on 2) The method of claim 2, further comprising determining pixel values in the retinal images based on the encoded data, where determining pixel values in the retinal images includes determining a pixel intensity or color indicative of a retinal cell response, and wherein the data indicative of a retinal cell response is indicative of at least one of a retinal cell firing rate, a retinal cell output pulse train, and a generator potential. ### Claim 4 (depends on 1) The method of claim 1, wherein the machine vision algorithm includes at least one selected from the list consisting of: an object recognition algorithm, an image classification algorithm, a facial recognition algorithm, an optical character recognition algorithm, a content-based image retrieval algorithm, a pose estimation algorithm, a motion analysis algorithm, an egomotion determination algorithm, a movement tracking algorithm, an optical flow determination algorithm, a scene reconstruction algorithm, a 3D volume recognition algorithm, and a navigation algorithm. ### Claim 5 (depends on 1) The method of claim 1, wherein adjusting the machine vision algorithm is responsive to a determination that the error rate exceeds the threshold level. ### Claim 6 (depends on 5) The method of claim 5, wherein adjusting the machine vision algorithm comprises iteratively adjusting one or more parameters until the error rate of the machine vision algorithm satisfies the threshold level. ### Claim 7 (depends on 1) The method of claim 1, wherein adjusting the machine vision algorithm comprises modifying a parameter of the machine vision algorithm. ### Claim 8 (depends on 1) The method of claim 1, where applying the machine vision algorithm comprises applying a navigation algorithm, where applying the navigation algorithm includes: processing the encoded data to determine motion information indicative of motion at a plurality of image locations in the encoded data; classifying spatial regions in the encoded data based on the motion information; and generating a navigation decision based on the classification of the spatial regions. ### Claim 9 (depends on 8) The method of claim 8, wherein the motion information is indicative of an optical flow in the encoded data, the method further including: using a convolutional neural network to classify the spatial regions; and controlling the motion of a robotic apparatus based on results from navigation algorithm. ### Claim 10 A method including: applying, by an encoding module, a spatiotemporal transformation to image data to generate retinal output cell response values; generating, by the encoding module, encoded data based on the retinal output cell response values; applying, by a machine vision module, a machine vision algorithm to the encoded data; monitoring, by a controller, performance of the machine vision algorithm, wherein monitoring the performance of the machine vision algorithm comprises: calculating an error rate of the machine vision algorithm; and comparing the error rate to a threshold level; and adjusting, by the controller, the machine vision algorithm based on the monitored performance, wherein adjusting the machine vision algorithm comprises modifying a parameter of the machine vision algorithm, and wherein the machine vision algorithm comprises an artificial neural network, and wherein adjusting the machine vision algorithm comprises changing a plurality of connections in an artificial neural network. ### Claim 11 A method including: applying, by an encoding module, a spatiotemporal transformation to image data to generate retinal output cell response values; generating, by the encoding module, encoded data based on the retinal output cell response values; applying, by a machine vision module, a machine vision algorithm to the encoded data: monitoring, by a controller, performance of the machine vision algorithm, wherein monitoring the performance of the machine vision algorithm comprises: calculating an error rate of the machine vision algorithm; and comparing the error rate to a threshold level; and adjusting, by the controller, the machine vision algorithm based on the monitored performance, wherein adjusting the machine vision algorithm comprises iteratively adjusting one or more parameters of the machine vision algorithm until an incremental increase in performance per iteration falls below a threshold level. ### Claim 12 An apparatus including: a memory storage device configured to store image data corresponding to a series of images; and a processor operably coupled with the memory and programmed to: receive the image data corresponding to the series of images; generate encoded data from the image data, wherein, to generate the encoded data, the processor is configured to: apply a spatiotemporal transformation to image data to generate retinal output cell response values; generate encoded data based on the retinal output cell response values; apply a machine vision algorithm to the encoded data; monitor performance of the machine vision algorithm, wherein, to monitor the performance of the machine vision algorithm, the processor is further configured to: calculate an error rate of the machine vision algorithm; and compare the error rate to a threshold level; and adjust the machine vision algorithm based on the monitored performance. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the processor is further configured to generate retinal images from the retinal output cell response values, and wherein applying the machine vision algorithm to the encoded data comprises applying the machine vision algorithm to the retinal images. ### Claim 14 (depends on 12) The apparatus of claim 12, wherein, to adjust the machine vision algorithm, the processor is further configured to iteratively adjust one or more parameters of the machine vision algorithm until an error rate of the machine vision algorithm satisfies the threshold level. ### Claim 15 (depends on 12) The apparatus of claim 12, wherein, to adjust the machine vision algorithm, the processor is further configured to iteratively adjust one or more parameters of the machine vision algorithm until an incremental increase in performance of the machine vision algorithm per iteration falls below a threshold level. ### Claim 16 A non-transitory computer-readable medium having computer-executable instructions for implementing operations comprising applying a spatiotemporal transformation to image data to generate retinal output cell response values; generating encoded data based on the retinal output cell response values; applying a machine vision algorithm to the encoded data; monitoring performance of the machine vision algorithm; and adjusting the machine vision algorithm based on the monitored performance, wherein adjusting the machine vision algorithm comprises iteratively adjusting one or more parameters of the machine vision algorithm until an incremental increase in performance per iteration of the machine vision algorithm falls below a threshold level. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein monitoring the performance of the machine vision algorithm comprises: calculating an error rate of the machine vision algorithm; and comparing the error rate to a threshold level; and wherein adjusting the machine vision algorithm is responsive to a determination that the error rate exceeds the threshold level.