# US 12,217,439 B2 · Real-time marine snow noise removal from underwater video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,439 B2 - Document ID: US12217439B2 - Status: Granted - Kind: B2 - Application number: 18/211,566 - Filed: 2023-06-19 - Published: 2024-04-11 - Granted: 2025-02-04 - Earliest claimed: 2020-02-25 - Applicants: Jack Wade - Inventors: Jack Wade - Web page: https://patentgrep.com/patent/US12217439B2 ## CPC - G06T 7/254: involving subtraction of images - G06T 7/269: using gradient-based methods - G06T 5/70: Denoising; Smoothing - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 7/215: Motion-based segmentation - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20224: Image subtraction - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Optical flow refers to the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene. Optical flow algorithms can be used to detect and delineate independently moving objects, even in the presence of camera motion. The present invention uses optical-flow algorithms to detect and remove marine snow particles from live video. Portions of an image scene which are identified as marine snow are reconstructed in a manner intended to reveal underwater scenery which had been occluded by the marine snow. Pixel locations within the regions of marine snow are replaced with new pixel values that are determined based on either historical data for each pixel or a mathematical operation, such as one which uses data from neighboring pixels. ## Claims ### Claim 1 An optical flow sensor to detect the movement of water while immersed under the surface of said water comprising: a camera for detecting optical flow events within said water; an optical flow comparison process wherein independently moving objects in the presence of an aquatic environment said water are isolated against a backdrop of water flow not associated with said moving objects and wherein movement of said camera is subtracted from said movement associated with said object; a process performed by a computer wherein video signals produced by said camera wherein marine snow noise are is identified and removed from an overall underwater image, so that said overall underwater image consists of said moving objects and wherein said overall underwater image is relatively clear for human observation in a video output display; wherein underwater scenery otherwise occluded by said marine snow noise are unobstructed and clear for said human observation, become more clearly visible for human observation; and wherein said optical flow comparison process comprises a chroma mask for each frame to identify pixel clusters of said marine snow noise. ### Claim 2 (depends on 1) The optical flow sensor of claim 1, further comprising of said camera for detecting optical flow presenting a video input and output stream. ### Claim 3 (depends on 1) The optical flow sensor of claim 1, wherein said video signals produced are shown in a video display module presenting processed video frames with reduced marine snow noise to provide a clear and enhanced view of an underwater video output stream. ### Claim 4 (depends on 1) The optical flow sensor of claim 1, wherein said overall underwater image is processed through a user interface and an FPGA to run an algorithm using said optical flow sensor to clarify said underwater image in real time. ### Claim 5 (depends on 4) The optical flow sensor of claim 4, further comprising of said overall underwater image featuring a synthetic video frame with unobstructed unoccluded live pixels from a current video frame and historical pixels with occluded pixels from previous frames. ### Claim 6 (depends on 1) The optical flow sensor of claim 1, wherein said process entails creating a dynamic chroma mask for each frame using said algorithm from said optical flow sensor and said optical flow algorithm to identify and track floating pixel clusters. ### Claim 7 (depends on 6) The process of claim 6, further comprising of said pixel clusters being geo-positioned on said chroma mask. ### Claim 8 A system for detecting the movement of water with an optical flow sensor while immersed under the surface of said water, the system comprising of: a live video stream, deriving from a video recording device and camera; a video processing platform with an integrated central processing unit for command and control, presented by way of a video display connected to said video recording device and said camera and a network module to process a video output; a DVR storage module, for storing and retrieving video data; an image processing module through a user interface on said video processing platform with a designated FPGA to run an optical flow algorithm; wherein said optical flow algorithm utilizes dynamic chroma keying to identify, track and remove marine snow noise; and a video controller, for managing live frames retrieved from said live video stream and for merging said live frames to removal occluded pixels. ### Claim 9 (depends on 8) The system of claim 8, wherein said image processing module uses a dynamic chroma mask for said live frames retrieved from said live video stream using said optical flow algorithms to identify and track floating pixel clusters, and wherein a synthetic frame is updated to provide a live frame of said live video stream while removing said marines snow noise in a merged frame. ### Claim 10 (depends on 9) The system of claim 9, wherein said image processing module utilizes said a synthetic video frame for said dynamic chroma mask for inverted chroma keying to clarify a video frame and clarify occlusion of said floating pixel clusters. ### Claim 11 (depends on 10) The system of claim 10, wherein said floating pixel clusters are geo-positioned on said dynamic chroma mask in chroma key color. ### Claim 12 (depends on 10) The system of claim 10, wherein pixels in said synthetic video frame comprise of said pixels convolved from neighboring pixels and wherein said occluded pixels are replaced by said pixels convolved from said neighboring pixels in said synthetic video frame. ### Claim 13 (depends on 10) The system of claim 10, wherein said synthetic video frame has unobstructed unoccluded live pixels from a current video frame and historical pixels from previous frames that replace said occluded pixels comprising of marine snow pixels. ### Claim 14 (depends on 8) The system of claim 8, wherein said image processing module employs a graphic processing unit for generating a visual display of said live video stream and said video output with processed video frames and reduced marine snow noise. ### Claim 15 A method for detecting the movement of water with an optical flow sensor while immersed under the surface of said water, the method comprising of: capturing a video stream by way of a video recording device and camera in an aquatic environment; transmitting said video stream to a video processing platform with an integrated central processing unit for command and control and a graphic processing unit for presenting a visual display of occluded video frames; storing data of said video stream to a DVR storage module; processing said video stream on said video processing platform, by way of a video display, over a network module to process a video output; processing video frames of said video stream on an image processing module through a user interface on said video processing platform with a designated FPGA to run an optical flow algorithm; removing occluded pixels in said video frames of said video stream using a dynamic chroma key mask operated managing said video frames by way of a video controller; creating a synthetic video frame by merging unobstructed pixels from said video frames with occluded pixels from previous frames using said dynamic chroma key mask; and fusing incoming videos with said chroma masks to encode and decode internet protocol video streams over said network module. ### Claim 16 (depends on 15) The method of claim 15, wherein said image processing module uses a dynamic chroma mask for said video frames retrieved from said video stream using said optical flow algorithm to identify and track floating pixel clusters. ### Claim 17 (depends on 16) The method of claim 16, wherein said image processing module utilizes said a synthetic video frame for said dynamic chroma mask for inverted chroma keying to clarify a video frame and clarify occlusions comprising of said floating pixel clusters. ### Claim 18 (depends on 16) The method of claim 16, wherein said floating pixel clusters are geo-positioned on said chroma mask in chroma key color. ### Claim 19 (depends on 17) The method of claim 17, wherein pixels in said synthetic video frame comprise of said pixels convolved from neighboring pixels, and wherein said occluded pixels are replaced by said pixels convolved from said neighboring pixels in said synthetic video frame. ### Claim 20 (depends on 19) The method of claim 19, further comprising wherein said synthetic video frame has unobstructed unoccluded live pixels from a current video frame and historical pixels from a previous frames that replace said occluded pixels comprising of marine snow pixels.