Editable semantic map with virtual camera for mobile robot learning

A system comprising:

• one or more processors;
• one or more computer memory in data communication with the one or more processors, wherein the one or more computer memory being non-transitory and having computer readable data stored thereon, the computer readable data including instructions that, when executed by one or more processors, causes the one or more processors to perform a method, the method including
• receiving a first semantic map of an environment, the first semantic map being three-dimensional (3D) and including semantic data;
• generating a background scene by filtering out foreground components from the first semantic map using the semantic data, the background scene including background components;
• generating a foreground scene by filtering out the background components from the first semantic map using the semantic data, the foreground scene including the foreground components;
• generating, via a first machine learning model, an enhanced background view using the background scene, the first machine learning model generating map data for incomplete regions of the background scene, the incomplete regions including at least corresponding parts of the background components occluded by the foreground components in the first semantic map, the map data including image data and depth data;
• receiving input data to edit the background components, the foreground components, or both the background components and the foreground components;
• generating a second semantic map using at least the enhanced background view and the input data, the second semantic map being 3D and a modified version of the first semantic map with respect to the background components and the foreground components;
• and
• generating virtual camera data using the second semantic map, the virtual camera data including at least new image data and new depth data of the second semantic map.

One or more non-transitory computer readable mediums having computer readable data stored thereon, the computer readable data including instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

• receiving a first semantic map of an environment, the first semantic map being three-dimensional (3D) and including semantic data;
• generating a background scene by filtering out foreground components from the first semantic map using the semantic data, the background scene including background components;
• generating a foreground scene by filtering out the background components from the first semantic map using the semantic data, the foreground scene including the foreground components;
• generating, via a first machine learning model, an enhanced background view using the background scene, the first machine learning model generating map data for incomplete regions of the background scene, the incomplete regions including at least corresponding parts of the background components occluded by the foreground components in the first semantic map, the map data including image data and depth data;
• receiving input data to edit the background components, the foreground components, or both the background components and the foreground components;
• generating a second semantic map using at least the enhanced background view and the input data, the second semantic map being 3D and a modified version of the first semantic map with respect to the background components and the foreground components;
• and
• generating virtual camera data using the second semantic map, the virtual camera data including at least new image data and new depth data of the second semantic map.