Content-aware AI-assisted video summarization

A non-transitory computer readable medium, storing instructions for summarizing an infrastructure surface inspection video, the instructions comprising:

• processing frames of the infrastructure surface inspection video through a spatial in-frame process configured to conduct feature and text extraction on the frames on detected observations, the extracted features and text used to generate spatial metadata;
• processing the frames of the infrastructure surface inspection video through a temporal cross-frame process configured to detect changes across sequences of the frames including the spatial metadata, the temporal cross-frame process configured to generate temporal metadata encapsulating the detected changes, the temporal cross-frame process comprising:
• conducting an optical flow estimate process to estimate velocity of a camera moving through a sewage pipe corresponding to the frames indicated in the spatial metadata, the optical flow estimate process configured to provide an optical flow vector representative of the estimated velocity;
• and
• obtaining stationary signals indicative of the frames recorded by the camera when the camera is stationary from executing a band pass filter on a Fourier transform of the optical flow vector;
• and
• processing the frames of the infrastructure surface inspection video through an activity decision process configured to intake the spatial metadata and the temporal metadata to detect and track structural features or defects of interest as activity in the infrastructure surface inspection video and generate summary metadata from the spatial metadata, the temporal metadata, and detected segments of interest.

An apparatus for summarizing an infrastructure surface inspection video, the apparatus comprising:

• a processor, configured to:
• process frames of the infrastructure surface inspection video through a spatial in-frame process configured to conduct feature and text extraction on the frames on detected observations, the extracted features and text used to generate spatial metadata;
• process the frames of the infrastructure surface inspection video through a temporal cross-frame process configured to detect changes across sequences of the frames including the spatial metadata, the temporal cross-frame process configured to generate temporal metadata encapsulating the detected changes, the temporal cross-frame process comprising:
• conducting an optical flow estimate process to estimate velocity of a camera moving through a sewage pipe corresponding to the frames indicated in the spatial metadata, the optical flow estimate process configured to provide an optical flow vector representative of the estimated velocity;
• and
• obtaining stationary signals indicative of the frames recorded by the camera when the camera is stationary from executing a band pass filter on a Fourier transform of the optical flow vector;
• and
• process the frames of the infrastructure surface inspection video through an activity decision process configured to intake the spatial metadata and the temporal metadata to detect and track structural features or defects of interest as activity in the infrastructure surface inspection video and generate summary metadata from the spatial metadata, the temporal metadata, and detected segments of interest.