# US 11,910,001 B2 · Real-time image generation in moving scenes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,910,001 B2 - Document ID: US11910001B2 - Status: Granted - Kind: B2 - Application number: 17/625,452 - Filed: 2020-07-09 - Published: 2022-08-11 - Granted: 2024-02-20 - Earliest claimed: 2019-07-09 - Assignees: Voyage81 Ltd. - Applicants: Voyage81 Ltd. - Inventors: Shmuel Mangan - Web page: https://patentgrep.com/patent/US11910001B2 ## CPC - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - H04N 19/513: Processing of motion vectors - H04N 7/0127: {by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter} - H04N 7/0137: {dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144)} - H04N 19/527: Global motion vector estimation - H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence - H04N 19/597: specially adapted for multi-view video sequence encoding - H04N 21/234381: {by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping} - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method comprising: receiving a high frame rate video stream of a scene, wherein the scene comprises at least one object in motion relative to an imaging device acquiring the video stream; continuously dividing, in real time, the video stream into at least one consecutive sequences of n frames each; with respect to each current sequence: (i) estimating pixel motion between at least some pairs of frames in the sequence, (ii) calculating a motion vector field for each pixel in the sequence, (iii) generating a representative frame which co-locates all of the pixels to respective pixel positions, based on the calculated motion vector fields, and (iv) aggregating, for each of the respective pixel positions, pixel values from all frames in the sequence; and outputting, in real time, a stream of the representative frames, wherein the stream has a lower frame rate than the high frame rate. ## Claims ### Claim 1 A system comprising: at least one hardware processor; and a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to: receive a high frame rate video stream of a scene, wherein said scene comprises at least one object in motion relative to an imaging device acquiring said video stream, continuously divide, in real time, said video stream into consecutive sequences of n frames each, with respect to each current sequence: (i) estimate motion for each pixel between adjacent pairs of frames in said current sequence, (ii) calculate a motion vector field for each of said pixels in said current sequence, (iii) generate a representative frame which co-locates each of said pixels to a respective pixel positions, based on said calculated motion vector fields, and (iv) aggregate, for each of said respective pixel positions, pixel values associated with its corresponding co-located pixel from all frames in said current sequence, and output, in real time, a stream of said representative frames, wherein said stream has a lower frame rate than said high frame rate. ### Claim 2 (depends on 1) The system of claim 1, wherein at least some of said pairs of frames are non-adjacent pairs of frames. ### Claim 3 (depends on 1) The system of claim 1, wherein said estimating is initialized with at least one of: a said estimating associated with a preceding one of said pairs in said sequence; a said estimating associated with a preceding time point in said video stream; and a hierarchical motion estimation. ### Claim 4 (depends on 1) The system of claim 1, wherein said estimating is performed using an optical flow method. ### Claim 5 (depends on 1) The system of claim 1, wherein said calculating is further refined over (i) a subset of said frames in said sequence, and (ii) a subset of resolution levels, by jointly solving multi-frame multi-level temporal-spatial smoothness constraints with respect to said motion vector field. ### Claim 6 (depends on 1) The system of claim 1, wherein said high frame rate is between 120-10,000 frames per second (fps). ### Claim 7 (depends on 1) The system of claim 1, wherein, for each of said respective pixel positions, said aggregating is performed using pixel averaging. ### Claim 8 A method comprising: receiving a high frame rate video stream of a scene, wherein said scene comprises at least one object in motion relative to an imaging device acquiring said video stream; continuously dividing, in real time, said video stream into consecutive sequences of n frames each; with respect to each current sequence: estimating motion for each pixel between adjacent pairs of frames in said current sequence, (ii) calculating a motion vector field for each of said pixels in said current sequence, (iii) generating a representative frame which co-locates each of said pixels to a respective pixel positions, based on said calculated motion vector fields, and (iv) aggregating, for each of said respective pixel positions, pixel values associated with its corresponding co-located pixel from all frames in said current sequence; and outputting, in real time, a stream of said representative frames, wherein said stream has a lower frame rate than said high frame rate. ### Claim 9 (depends on 8) The method of claim 8, wherein at least some of said pairs of frames are non-adjacent pairs of frames. ### Claim 10 (depends on 8) The method of claim 8, wherein said estimating is initialized with at least one of: a said estimating associated with a preceding one of said pairs in said sequence; a said estimating associated with a preceding time point in said video stream; and a hierarchical motion estimation. ### Claim 11 (depends on 8) The method of claim 8, wherein said estimating is performed using an optical flow method. ### Claim 12 (depends on 8) The method of claim 8, wherein said calculating is further refined over (i) a subset of said frames in said sequence, and (ii) a subset of resolution levels, by jointly solving multi-frame multi-level temporal-spatial smoothness constraints with respect to said motion vector field. ### Claim 13 (depends on 8) The method of claim 8, wherein said high frame rate is between 120-10,000 frames per second (fps). ### Claim 14 (depends on 8) The method of claim 8, wherein, for each of said respective pixel positions, said aggregating is performed using pixel averaging. ### Claim 15 A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to: receive a high frame rate video stream of a scene, wherein said scene comprises at least one object in motion relative to an imaging device acquiring said video stream; continuously divide, in real time, said video stream into consecutive sequences of n frames each; with respect to each current sequence: (i) estimate motion for each pixel between adjacent pairs of frames in said current sequence, (ii) calculate a motion vector field for each of said pixels in said current sequence, (iii) generate a representative frame which co-locates each all of said pixels to a respective pixel positions, based on said calculated motion vector fields, and (iv) aggregate, for each of said respective pixel positions, pixel values associated with its corresponding co-located pixel from all frames in said sequence; and output, in real time, a stream of said representative frames, wherein said stream has a lower frame rate than said high frame rate. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein at least some of said pairs of frames are non-adjacent pairs of frames. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein said estimating is initialized with at least one of: a said estimating associated with a preceding one of said pairs in said sequence; a said estimating associated with a preceding time point in said video stream; and a hierarchical motion estimation. ### Claim 18 (depends on 15) The computer program product of claim 15, wherein said estimating is performed using an optical flow method. ### Claim 19 (depends on 15) The computer program product of claim 15, wherein said calculating is further refined over (i) a subset of said frames in said sequence, and (ii) a subset of resolution levels, by jointly solving multi-frame multi-level temporal-spatial smoothness constraints with respect to said motion vector field. ### Claim 20 (depends on 15) The computer program product of claim 15, wherein said high frame rate is between 120-10,000 frames per second (fps).