# US 12,223,620 B2 · Systems, methods, and media for eulerian single-photon computer vision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,620 B2
- Document ID: US12223620B2
- Status: Granted
- Kind: B2
- Application number: 18/310,399
- Filed: 2023-05-01
- Published: 2024-11-07
- Granted: 2025-02-11
- Earliest claimed: 2023-05-01
- Assignees: Wisconsin Alumni Research Foundation
- Applicants: Wisconsin Alumni Research Foundation
- Inventors: Mohit Gupta; Shantanu Gupta
- Web page: https://patentgrep.com/patent/US12223620B2

## CPC

- G06T 5/20: using local operators
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G01J 1/0228: {Control of working procedures; Failure detection; Spectral bandwidth calculation}
- G01J 1/44: Electric circuits {(for command of an exposure part G03B7/02)}
- G06T 5/70: Denoising; Smoothing
- G06T 7/13: Edge detection
- G06T 7/248: {involving reference images or patches}
- H04N 25/76: Addressed sensors, e.g. MOS or CMOS sensors
- H04N 25/773: comprising photon counting circuits, e.g. single photon detection [SPD] or single photon avalanche diodes [SPAD]
- G01J 2001/442: {Single-photon detection or photon counting}
- G01J 2001/4466: {Avalanche}
- G06T 2207/10016: Video; Image sequence
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In accordance with some embodiments, systems, methods, and media for Eulerian single-photon computer vision are provided. In some embodiments, the system comprises: an image sensor comprising detectors configured to detect arrival of individual photons, and arranged in an array; a processor programmed to: cause the image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels; perform, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames, wherein each of the plurality of frames is based on one or more of the images of the sequence of images; generate, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and perform a computer vision process based on the plurality of filter responses.

## Claims

### Claim 1

A system for facilitating single-photon computer vision tasks, comprising:
an image sensor comprising a plurality of detectors configured to detect arrival of individual photons, the plurality of detectors arranged in an array;
at least one processor that is programmed to:
cause the image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels;
perform, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames,
wherein each of the plurality of frames is based on one or more of the images of the sequence of images;
generate, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and
perform a computer vision process based on the plurality of filter responses.

### Claim 2 (depends on 1)

The system of claim 1, wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).

### Claim 3 (depends on 1)

The system of claim 1, wherein each image of the sequence of images comprises a binary image that represents photons detected by the image sensor during an exposure time T.

### Claim 4 (depends on 1)

The system of claim 1, wherein each of the plurality of three dimensional filters comprises a velocity tuned filter, and
wherein a first subset of the plurality of three dimensional filters are tuned to a three dimensional frequency k:=(kx, ky, kt), where kx and ky represent spatial patterns, and kt represents a pattern in time, each of the three dimensional filters of the first subset having a different scale.

### Claim 5 (depends on 1)

The system of claim 1, wherein the at least one processor that is further programmed to:
determine a z-score for each of the plurality of filter bank responses;
map each z-score to a weight associated with the respective filter bank for which the z-score was determined; and
utilize the weighted filter bank responses to perform the computer vision process based on the plurality of filter responses.

### Claim 6 (depends on 5)

The system of claim 5, wherein the at least one processor that is further programmed to:
estimate a variance value Var(Rk[p]) using a relationship Vk[p]:=V(c[p])Σq|hk[q]|2,
where Rk[p] is a filter bank response generated by applying a filter hk[q] to the plurality of frames B[p], V(ĉ[p]) is a variance of an estimated local average flux over B[p], and Σq|hk[q]|2 is a sum of the filter hk[q] over q<p; and
determine the z-score using a relationship
z
k
[
p
]
:=
❘
"\[LeftBracketingBar]"
R
k
[
p
]
❘
"\[RightBracketingBar]"
V
k
[
p
]
.

### Claim 7 (depends on 6)

The system of claim 6, wherein the at least one processor that is further programmed to:
map each z-score to a weight associated with the respective filter bank for which the z-score was determined using a relationship w(z):=1−exp(−max(0, z−z0)), where z0 comprises a threshold z-score.

### Claim 8 (depends on 1)

The system of claim 1, wherein the computer vision process is an edge detection process, and
wherein at least one processor that is further programmed to:
perform the computer vision process based on a phase congruency associated with each of the plurality of filter responses.

### Claim 9 (depends on 8)

The system of claim 8, wherein at least one processor that is further programmed to:
detect one or more corners based on the phase congruency associated with each of the plurality of filter responses.

### Claim 10 (depends on 1)

The system of claim 1, wherein the computer vision process is a motion estimation process, and
wherein at least one processor that is further programmed to:
perform the computer vision process for each of the plurality of pixels based on a phase constancy relationship vx cos {tilde over (θ)}+vy sin {tilde over (θ)}=cot {tilde over (ϕ)},
where (vx, vy) is a velocity at the respective pixel, vn=cot {tilde over (ϕ)} is a component velocity of a respective three dimensional filter {tilde over (k)} in a spatial direction {tilde over (θ)}, and {tilde over (k)}:={tilde over (s)}·(sin {tilde over (ϕ)} cos {tilde over (θ)}, sin {tilde over (ϕ)} sin {tilde over (θ)}, cos {tilde over (ϕ)}),
where ({tilde over (θ)}, {tilde over (ϕ)}) is a spatio-temporal direction of {tilde over (k)}.

### Claim 11

A method for facilitating single-photon computer vision tasks, the method comprising:
causing an image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels,
wherein the image sensor comprises a plurality of detectors configured to detect arrival of individual photons, the plurality of detectors arranged in an array;
performing, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames,
wherein each of the plurality of frames is based on one or more of the images of the sequence of images;
generating, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and
performing a computer vision process based on the plurality of filter responses.

### Claim 12 (depends on 11)

The method of claim 11, wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).

### Claim 13 (depends on 11)

The method of claim 11, wherein each image of the sequence of images comprises a binary image that represents photons detected by the image sensor during an exposure time T.

### Claim 14 (depends on 11)

The method of claim 11, wherein each of the plurality of three dimensional filters comprises a velocity tuned filter, and
wherein a first subset of the plurality of three dimensional filters are tuned to a three dimensional frequency k:=(kx, ky, kt), where kx and ky represent spatial patterns, and kt represents a pattern in time, each of the three dimensional filters of the first subset having a different scale.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
determining a z-score for each of the plurality of filter bank responses; mapping each z-score to a weight associated with the respective filter bank for which the z-score was determined; and
utilize the weighted filter bank responses to perform the computer vision process based on the plurality of filter responses.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
estimating a variance value Var(Rk[p]) using a relationship Vk[p]:=V(ĉ[p])Σq|hk[q]|2,
where Rk[p] is a filter bank response generated by applying a filter hk[q] to the plurality of frames B[p], V(ĉ[p]) is a variance of an estimated local average flux over B[p], and Σq|hk[q]|2 is a sum of the filter hk[q] over q<p; and
determining the z-score using a relationship
z
k
[
p
]
:=
❘
"\[LeftBracketingBar]"
R
k
[
p
]
❘
"\[RightBracketingBar]"
V
k
[
p
]
.

### Claim 17 (depends on 11)

The method of claim 11, wherein the computer vision process is an edge detection process based on a phase congruency associated with each of the plurality of filter responses.

### Claim 18 (depends on 11)

The method of claim 11, wherein the computer vision process is a motion estimation process based on a phase constancy relationship.

### Claim 19

A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for facilitating single-photon computer vision tasks, the method comprising:
causing an image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels,
wherein the image sensor comprises a plurality of detectors configured to detect arrival of individual photons, the plurality of detectors arranged in an array;
performing, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames,
wherein each of the plurality of frames is based on one or more of the images of the sequence of images;
generating, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and
performing a computer vision process based on the plurality of filter responses.

### Claim 20 (depends on 19)

The non-transitory computer readable medium of claim 19, wherein each of the plurality of detectors comprises a single photon avalanche diode (SPAD).

### Claim 21 (depends on 19)

The non-transitory computer readable medium of claim 19, wherein each image of the sequence of images comprises a binary image that represents photons detected by the image sensor during an exposure time T.