RGBD video semantic segmentation with temporal and geometric consistency

A non-transitory machine-readable medium having stored thereon at least one program, the at least one program including instructions which, when executed by a processor, cause the processor to perform a method in a processor-based system for RGBD semantic segmentation of video data, comprising:

• determining at least one of semantic segmentation data and depth-related data for less than all classes for images of each frame of a first video comprising at least one of RGB video frames and depth-aware video frames;
• determining at least one of semantic segmentation data and depth-related data for more classes than in the first video for images of each key frame of a second video comprising a synchronous combination of respective frames of the RGB video frames and the depth-aware video frames, (RGBD video), in parallel to the determination of the at least one of the semantic segmentation data and the depth-related data for the images of each frame of the first video;
• temporally and geometrically aligning respective frames of the first video and the second video;
• and
• predicting at least one of semantic segmentation data and depth-related data for images of a subsequent frame of the first video based on the determination of the at least one of semantic segmentation data depth-related data for images of a key frame of the second video;
• wherein the first video comprises an alternating pattern of RGB video frames and depth-aware video frames.

A system for RGBD semantic segmentation of video data, comprising:

• a processor;
• and
• a memory coupled to the processor, the memory having stored therein at least one of programs or instructions executable by the processor to configure the system to:
• determine at least one of semantic segmentation data and depth segmentation data for less than all classes for images of each frame of a first video comprising at least one of RGB video frames and depth-aware video frames;
• determine at least one of semantic segmentation data and depth segmentation data for more classes than in the first video for images of each key frame of a second video comprising a synchronous combination of respective frames of the RGB video frames and the depth-aware video frames, (RGBD video), in parallel to the determination of the at least one of the semantic segmentation data and the depth segmentation data for each frame of the first video;
• temporally and geometrically align respective frames of the first video and the second video;
• and
• predict at least one of semantic segmentation data and depth segmentation data for images of a subsequent frame of the first video based on the determination of the at least one of semantic segmentation data and depth segmentation data for images of a key frame of the second video;
• wherein the first video comprises an alternating pattern of RGB video frames and depth-aware video frames.

A method for RGBD semantic segmentation of video data, comprising:

• determining at least one of semantic segmentation data and depth segmentation data for less than all classes for images of each frame of a first video comprising at least one of RGB video frames and depth-aware video frames;
• determining at least one of semantic segmentation data and depth segmentation data for more classes than in the first video for images of each key frame of a second video comprising a synchronous combination of respective frames of the RGB video frames and the depth-aware video frames, (RGBD video), in parallel to the determination of the at least one of the semantic segmentation data and the depth segmentation data for each frame of the first video;
• temporally and geometrically aligning respective frames of the first video and the second video;
• and
• predicting at least one of semantic segmentation data and depth segmentation data for images of a subsequent frame of the first video based on the determination of the at least one of semantic segmentation data and depth segmentation data for images of a key frame of the second video;
• wherein temporally and geometrically aligning frames of the first video and the second video comprises applying a two-stage process including at least an ego-motion flow process and a projected scene flow process, wherein the projected scene flow process corrects an estimation error introduced by the ego-motion flow process.