# US 2022/0279203 A1 · Real-time motion tracking in moving scenes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0279203 A1
- Document ID: US20220279203A1
- Status: Published application
- Kind: A1
- Application number: 17/625,751
- Filed: 2020-07-09
- Published: 2022-09-01
- Earliest claimed: 2019-07-09
- Inventors: Shmuel MANGAN
- Web page: https://patentgrep.com/patent/US20220279203A1

## CPC

- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- H04N 19/513: Processing of motion vectors
- H04N 7/0127: {by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter}
- H04N 7/0137: {dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144)}
- H04N 19/527: Global motion vector estimation
- H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 21/234381: {by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method comprising: receiving a high frame rate video stream of a scene; continuously dividing, in real time, the video stream into consecutive sequences of frames each; with respect to each current sequence: (i) estimating pixel motion between pairs of frames in the sequence, to calculate a current motion vector field for each pixel in the sequence, (ii) co-locating all of the pixels to current representative pixel positions associated with a desired time point in the sequence, and (iii) calculating an inter-sequence motion vector field, based on estimating motion between the current representative pixel positions and an immediately preceding sequence of the sequences; and outputting, in real time, at a rate that is lower than the high frame rate, at least one of (x) the current motion vector field, (y) the inter-sequence motion vector field, and (z) pixel values associated with the current representative pixel positions.

## Claims

### Claim 1

A system comprising:
at least one hardware processor; and
a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
receive a high frame rate video stream of a scene, wherein said scene comprises at least one object in motion relative to an imaging device acquiring said video stream,
process, in real time, each current sequence of n frames in said video stream to:
(i) estimate pixel motion between a first pair of adjacent frames in said current sequence,
(ii) iteratively estimate pixel motion with respect to each new frame in said current sequence, relative to a cumulative value of all preceding pixel motion estimations in said current sequence, to calculate a current motion vector field for each pixel in said current sequence,
(iii) calculate, based on all of said current motion vector fields, aggregated pixel values corresponding to each of said pixels in said current sequence, and
(iv) repeat (i)-(iii) with respect to a next one of said sequences in said video stream, and
output, in real time, at least one of (x) said current motion vector field, (y) a cumulative value of all of said calculated current motion vector fields, and (z) said aggregate pixel values.

### Claim 2 (depends on 1)

The system of claim 1, wherein said program instructions are further executable to repeat steps (i)-(iv) over consecutive pairs of said processed current sequences, and wherein said output further comprises at least one of: (a) said current motion vector fields for each pixel over a current one of said consecutive pair of said sequences, (b) a cumulative value of all of said calculated current motion vector fields, and (c) said aggregate pixel values for each pixel over said current consecutive pair of said sequences.

### Claim 3

(canceled)

### Claim 4 (depends on 1)

The system of claim 1, wherein said cumulative value of all of said calculated current motion vector fields is calculated based, at least in part, on solving multi-frame multi-level temporal-spatial smoothness constraints.

### Claim 5

(canceled)

### Claim 6 (depends on 1)

The system of claim 1, wherein said estimating is performed with respect to at least one non-adjacent pair of frames in said current sequence.

### Claim 7

(canceled)

### Claim 8 (depends on 1)

The system of claim 1, wherein said pixel motion estimations are performed using an optical flow algorithm.

### Claim 9

(canceled)

### Claim 10

(canceled)

### Claim 11 (depends on 1)

The system of claim 1, wherein said high frame rate is between 60-10,000 frames per second (fps).

### Claim 12 (depends on 1)

The system of claim 1, wherein said program instructions are further executable to generate a current representative frame with respect to each of said current sequences, wherein said current representative frame is associated with a desired time point in said sequence, and wherein each pixel in said current representative frame is assigned (i) a pixel position associated with said desired time point, and (ii) said respective aggregate pixel value.

### Claim 13 (depends on 12)

The system of claim 12, wherein said program instructions are further executable to output each of said current representative frames at a frame rate that is lower than said high frame rate.

### Claim 14

(canceled)

### Claim 15 (depends on 12)

The system of claim 12, wherein said program instructions are further executable to receive one of: depth information, position-motion sensor information, and ego-motion information, with respect to said scene, wherein said pixel positions comprise position information in relation to three-dimensional (3D) world coordinates.

### Claim 16 (depends on 1)

The system of claim 1, wherein said program instructions are further executable to (i) receive two of said video streams of said scene, and (ii) determine depth information with respect to said scene based, at least in part, on said two video streams.

### Claim 17

(canceled)

### Claim 18

A method comprising:
receiving a high frame rate video stream of a scene, wherein said scene comprises at least one object in motion relative to an imaging device acquiring said video stream;
processing, in real time, each current sequence of n frames in said video stream by:
estimating pixel motion between a first pair of adjacent frames in said current sequence,
(ii) iteratively estimating pixel motion with respect to each new frame in said current sequence, relative to a cumulative value of all preceding pixel motion estimations in said current sequence, to calculate a current motion vector field for each pixel in said current sequence,
(iii) calculating, based on all of said current motion vector fields, aggregated pixel values corresponding to each of said pixels in said current sequence, and
(iv) repeating (i)-(iii) with respect to a next one of said sequences in said video stream; and
outputting, in real time, at least one of (x) said current motion vector field, (y) a cumulative value of all of said calculated current motion vector fields, and (z) said aggregate pixel values.

### Claim 19 (depends on 18)

The method of claim 18, further comprising repeating steps (i)-(iv) over consecutive pairs of said processed current sequences, and wherein said outputting further comprises at least one of: (a) said current motion vector fields for each pixel over a current one of said consecutive pair of said sequences, (b) a cumulative value of all of said calculated current motion vector fields, and (c) said aggregate pixel values for each pixel over said current consecutive pair of said sequences.

### Claim 20

(canceled)

### Claim 21 (depends on 18)

The method of claim 18, wherein said cumulative value of all of said calculated current motion vector fields is calculated based, at least in part, on solving multi-frame multi-level temporal-spatial smoothness constraints.

### Claim 22

(canceled)

### Claim 23 (depends on 18)

The method of claim 18, wherein said pixel motion estimating is performed with respect to at least one non-adjacent pair of frames in said sequence.

### Claim 24

(canceled)

### Claim 25 (depends on 18)

The method of claim 18, wherein said estimating is performed using an optical flow algorithm.

### Claim 26

(canceled)

### Claim 27

(canceled)

### Claim 28 (depends on 18)

The method of claim 18, wherein said high frame rate is between 60-10,000 frames per second (fps).

### Claim 29 (depends on 18)

The method of claim 18, further comprising generating a current representative frame with respect to each of said current sequences, wherein said current representative frame is associated with a desired time point in said sequence, and wherein each pixel in said current representative frame is assigned (i) a pixel position associated with said desired time point, and (ii) said respective aggregate pixel values.

### Claim 30 (depends on 18)

The method of claim 18, further comprising to outputting each of said current representative frames at a frame rate that is lower than said high frame rate.

### Claim 31

(canceled)

### Claim 32 (depends on 30)

The method of claim 30, further comprising receiving one of: depth information, position-motion sensor information, and ego-motion information, with respect to said scene, wherein said pixel positions comprise position information in relation to three-dimensional (3D) world coordinates.

### Claim 33

(canceled)

### Claim 34

A computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to:
receive a high frame rate video stream of a scene, wherein said scene comprises at least one object in motion relative to an imaging device acquiring said video stream;
process, in real time, each sequence of n frames in said video stream to:
(i) estimate pixel motion between a first pair of adjacent frames in said current sequence,
(ii) iteratively estimate pixel motion with respect to each new frame in said current sequence, relative to a cumulative value of all preceding pixel motion estimations in said current sequence, to calculate a current motion vector field for each pixel in said current sequence,
(iii) calculate, based on all of said current motion vector fields, aggregated pixel values corresponding to each of said pixels in said current sequence, and
(iv) repeat (i)-(iii) with respect to a next one of said sequences in said video stream; and
output, in real time, at least one of (x) said current motion vector field, (y) a cumulative value of all of said motion vector fields, and (z) said aggregate pixel values.

### Claim at position 35

-49. (canceled)
