Real-time remote collaboration and virtual presence using simultaneous localization and mapping to construct a 3D model and update a scene based on sparse data

A computerized method of generating a video stream of a scene including one or more objects, the method comprising:

• capturing, by a sensor device, a plurality of images of one or more objects in a scene;
• generating, by a server computing device coupled to the sensor device, a 3D model for each of the one or more objects in the scene using an image in the plurality of images;
• generating, by the server computing device, a 3D model of the scene using the image in the plurality of images;
• capturing, by the server computing device for each image in the plurality of images, pose information of the sensor device relative to at least one of the scene or one or more of the objects in the scene as the sensor device moves in relation to the scene;
• generating, by the server computing for each image in the plurality of images, one or more changes to at least one of the 3D models for each of the one or more objects in the scene based upon a difference between (i) the pose information of the sensor device captured using the image and (ii) pose information of the sensor device captured using a previous image in the plurality of images;
• receiving, by a viewing device coupled to the server computing device, the 3D models of each of the one or more objects in the scene and the 3D model of the scene from the server computing device;
• receiving, by the viewing device for each image in the plurality of images, the captured pose information of the sensor device relative to at least one of the scene or one or more of the objects in the scene;
• and
• rendering, by the viewing device, a video stream of at least one of the one or more objects in the scene and the scene on a display element of the viewing device by:
• receiving, for each image in the plurality of images, the generated one or more changes to at least one of the 3D models for each of the one or more objects in the scene,
• updating, for each image in the plurality of images, one or more of the 3D models for each of the one or more objects in the scene using the generated one or more changes, and
• rendering, for each image in the plurality of images, a portion of the video stream using the updated 3D models and at least one of (i) the captured pose information of the sensor device relative to at least one of the scene or one or more of the objects in the scene or (ii) pose information of a current viewing perspective of the viewing device relative to at least one of the scene or one or more of the objects in the scene.