# US 2019/0102641 A1 · Digital neuromorphic (nm) sensor array, detector, engine and methodologies > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0102641 A1 - Document ID: US20190102641A1 - Status: Published application - Kind: A1 - Application number: 16/194,462 - Filed: 2018-11-19 - Published: 2019-04-04 - Earliest claimed: 2016-12-21 - Assignees: Volkswagen Aktiengesellschaft; Audi AG; Porsche AG - Applicants: Volkswagen AG; Audi AG; Porsche AG - Inventors: Edmund Dawes ZINK; Douglas Allen HAUGER; Lutz JUNGE; Luis Marcial HERNANDEZ GONZALEZ; Jerramy L. GIPSON; Anh VU; Martin HEMPEL; Nikhil J. GEORGE - Web page: https://patentgrep.com/patent/US20190102641A1 ## CPC - G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes} - G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs - G06N 3/063: using electronic means - G06T 7/215: Motion-based segmentation - G06T 7/248: {involving reference images or patches} - G06V 10/20: Image preprocessing - G06V 10/955: {using specific electronic processors} - G06V 20/44: {Event detection} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and methodologies for neuromorphic vision simulate conventional analog NM system functionality and generate digital NM image data that facilitate improved object detection, classification, and tracking. ## Claims ### Claim 1 (canceled) ### Claim 2 (canceled) ### Claim 3 (canceled) ### Claim 4 (canceled) ### Claim 5 (canceled) ### Claim 6 (canceled) ### Claim 7 (canceled) ### Claim 8 (canceled) ### Claim 9 (canceled) ### Claim 10 (canceled) ### Claim 11 (canceled) ### Claim 12 (canceled) ### Claim 13 (canceled) ### Claim 14 (canceled) ### Claim 15 (canceled) ### Claim 16 (canceled) ### Claim 17 (canceled) ### Claim 18 (canceled) ### Claim 19 (canceled) ### Claim 20 (canceled) ### Claim 21 (canceled) ### Claim 22 (canceled) ### Claim 23 (canceled) ### Claim 24 (canceled) ### Claim 25 A neuromorphic vision system generating and processing image data, the system comprising: a composite image sensor comprised of a plurality of photoreceptors for generating image data indicating whether an intensity value measured by each photoreceptor exceeds a threshold; a plurality of computational elements corresponding to the plurality of photoreceptors, wherein each of the plurality of computational elements generates image spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and a digital neuromorphic engine including one or more processors running software configured to generate, based on the generated image spike data, digital neuromorphic output data that includes velocity vector data representing a velocity space that is a spatial and temporal representation of the image data. ### Claim 26 (depends on 25) The neuromorphic vision system of claim 25, wherein the digital neuromorphic engine aggregates and associates the velocity vector data to perform velocity segmentation to identify and differentiate objects within the image data based on relative motion identified over frames of image data. ### Claim 27 (depends on 25) The neuromorphic vision system of claim 25, wherein the digital neuromorphic engine compresses high framerate input video data generated by the plurality of photodetectors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon. ### Claim 28 (depends on 25) The neuromorphic vision system of claim 25, wherein the neuromorphic vision system is incorporated in a stereo neuromorphic pair or a compound camera. ### Claim 29 (depends on 25) The neuromorphic vision system of claim 25, further comprising at least one sensor measuring data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data measured by the at least one sensor. ### Claim 30 (depends on 25) The neuromorphic vision system of claim 25, wherein the plurality of computational elements are implemented in a field-programmable gate array or using complementary metal-oxide-semiconductor technology. ### Claim 31 (depends on 25) The neuromorphic vision system of claim 25, wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing by the digital neuromorphic engine. ### Claim 32 A method for generating and processing neuromorphic vision image data, the method comprising: generating image data using a plurality of photoreceptors; generating spike data using a plurality of computation elements, the spike data indicating whether the measured intensity values exceed a threshold, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and generating digital neuromorphic output data using a digital neuromorphic engine coupled to the plurality of computational elements, wherein the digital neuromorphic output data includes velocity vector data generated based on the generated spike data and representing a velocity space that is a spatial and temporal representation of the image data generated by the plurality of photodetectors. ### Claim 33 (depends on 32) The method of claim 32, further comprising aggregating and associating the velocity vector data by the digital neuromorphic engine to perform velocity segmentation to identify and differentiate objects within the image data based on relative motion identified over frames of image data. ### Claim 34 (depends on 32) The method of claim 32, further comprising compressing, by the digital neuromorphic engine, high framerate input video data generated by the plurality of photodetectors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon. ### Claim 35 (depends on 32) The method of claim 32, further comprising measuring, by at least one sensor, data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data measured by the at least one sensor. ### Claim 36 (depends on 32) The method of claim 32, wherein the plurality of computational elements are implemented in a field-programmable gate array or using complementary metal-oxide-semiconductor technology. ### Claim 37 (depends on 32) The method of claim 32, wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing by the digital neuromorphic engine. ### Claim 38 A non-transitory computer readable medium including software instructions for performing a method for generating and processing neuromorphic vision image data when the software is run on one or more processors, the method comprising: generating image data using a plurality of photoreceptors; generating spike data using a plurality of computation elements, the spike data indicating whether the measured intensity values exceed a threshold, wherein each of the plurality of computational elements generates spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor; and generating digital neuromorphic output data using a digital neuromorphic engine coupled to the plurality of computational elements, wherein the digital neuromorphic output data includes velocity vector data generated based on the generated spike data and representing a velocity space that is a spatial and temporal representation of the image data generated by the plurality of photodetectors. ### Claim 39 (depends on 38) The non-transitory computer readable medium of claim 38, further comprising aggregating and associating the velocity vector data by the digital neuromorphic engine to perform velocity segmentation to identify and differentiate objects within the image data based on relative motion identified over frames of image data. ### Claim 40 (depends on 38) The non-transitory computer readable medium of claim 38, further comprising compressing, by the digital neuromorphic engine, high framerate input video data generated by the plurality of photodetectors by performing feature extraction to generate an encoded version of the image data by capturing differences between a current frame of image data and one or more previous frames of image data and applying a velocity transformation thereon. ### Claim 41 (depends on 38) The non-transitory computer readable medium of claim 38, further comprising measuring, by at least one sensor, data for a characteristic including audio, temperature, force, direction, location, and motion, wherein operation of the image sensor is controlled based on the measured data measured by the at least one sensor. ### Claim 42 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the plurality of computational elements are implemented in a field-programmable gate array or using complementary metal-oxide-semiconductor technology. ### Claim 43 (depends on 38) The non-transitory computer readable medium of claim 38, wherein the plurality of computational elements cooperate to generate an encoded version of image data that includes only data indicating differences indicative of movement and surrounding spatio-temporal regions for subsequent image processing by the digital neuromorphic engine. ### Claim 44 A digital neuromorphic engine comprising: one or more processors running software configured to generate digital neuromorphic output data that includes velocity vector data representing a velocity space that is a spatial and temporal representation of image data, wherein the image data is generated by a composite image sensor comprised of a plurality of photoreceptors, wherein the image data indicates whether an intensity value measured by each photoreceptor exceeds a threshold, wherein a plurality of computational elements corresponding to the plurality of photoreceptors generate image spike data for the respective corresponding photoreceptor based on the image data generated by at least two of the plurality of photoreceptors including the respective corresponding photoreceptor and a photoreceptor neighboring the respective corresponding photoreceptor, and wherein the one or more processors generate the digital neuromorphic output data based on the generated image spike data.