Example-based cross-modal denoising
AssigneeTechnion Research & Development Foundation Limited
InventorsDana Segev, Yoav Schechner, Michael Elad
Life of this patent
Estimating the term…
Abstract
A method for cross-modal signal denoising, the method comprising using at least one hardware processor for: providing a first multi-modal signal comprising at least two relatively clear modalities; correlating features exhibited simultaneously in the at least two relatively clear modalities of the first multi-modal signal; providing a second multi-modal signal comprising at least one relatively noisy modality and at least one relatively clear modality; and denoising the at least one relatively noisy modality of the second multi-modal signal by associating between (a) features exhibited in the at least one relatively noisy modality of the second multi-modal signal and (b) the features of the first multi-modal signal.
Claims27
A method for cross-modal signal denoising, the method comprising using at least one hardware processor for:
- providing a first multi-modal signal comprising at least two relatively clear modalities;
- correlating features exhibited simultaneously in the at least two relatively clear modalities of the first multi-modal signal;
- providing a second multi-modal signal comprising at least one relatively noisy modality and at least one relatively clear modality;
- and
- denoising the at least one relatively noisy modality of the second multi-modal signal by associating between (a) features exhibited in the at least one relatively noisy modality of the second multi-modal signal and (b) the correlated features of the first multi-modal signal.
An apparatus comprising:
- an image sensor configured for video capture;
- a microphone;
- a non-transient memory having stored thereon correlated features exhibited simultaneously in a relatively clear video modality and in a relatively clear audio modality both belonging to a first multi-modal signal;
- and
- at least one hardware processor configured to:
- (a) receive a second multi-modal signal comprising a relatively clear video modality from said image sensor and a relatively noisy audio modality from said microphone, and
- (b) denoise the relatively noisy audio modality of the second multi-modal signal by associating between (i) features exhibited in the relatively noisy audio modality of the second multi-modal signal and (ii) the correlated features of the first multi-modal signal.
A computer program product for cross-modal signal denoising, comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to:
- provide a first multi-modal signal comprising at least two relatively clear modalities;
- correlate features exhibited simultaneously in the at least two relatively clear modalities of the first multi-modal signal;
- provide a second multi-modal signal comprising at least one relatively noisy modality and at least one relatively clear modality;
- and
- denoise the at least one relatively noisy modality of the second multi-modal signal by associating between (a) features exhibited in the at least one relatively noisy modality of the second multi-modal signal and (b) the correlated features of the first multi-modal signal.
The computer program product according to claim 16, wherein said replacing is based on a statistical analysis of the features of:
- one of the at least two relatively clear modalities of the first multi-modal signal;
- and
- features exhibited in the at least one relatively clear modality of the second multi-modal signal.
A method for cross-modal signal denoising, the method comprising using at least one hardware processor for:
- providing correlated features exhibited simultaneously in a relatively clear video modality and in a relatively clear audio modality both belonging to a first multi-modal signal;
- providing a second multi-modal signal comprising at least one relatively noisy modality and at least one relatively clear modality;
- and
- denoising the at least one relatively noisy modality of the second multi-modal signal by associating between (a) features exhibited in the at least one relatively noisy modality of the second multi-modal signal and (b) the correlated features of the first multi-modal signal.
Dependencies are inferred from the claim text.