Method and system for object segmentation in a mixed reality environment

An apparatus comprising:

• a display device;
• a camera configured to capture a first image of a user's appendage and a physical peripheral input device, wherein the user's appendage covers at least a part of the physical peripheral input device;
• and
• an image processor coupled with the camera and with the display device and configured to:
• receive the first image from the camera;
• perform a segmentation of the first image into a first region of pixels corresponding to the user's appendage, wherein a second region of pixels corresponds to the physical peripheral input device;
• generate a composite image from the first image based on rendering, in place of the second region of pixels, pixels representing a virtual peripheral input device, such that the virtual peripheral input device appears in the composite image as at a same spatial location as the physical peripheral input device and covered at least partially by the user's appendage;
• and
• provide the composite image to the display device for displaying to the user.

A non-transitory computer readable medium storing instructions that, when executed by a hardware processor, causes the hardware processor to:

• receive, from a camera, a first image of a user's appendage and a physical peripheral input device, wherein the user's appendage covers at least a part of the physical peripheral input device;
• perform a segmentation of the first image into a first region of pixels corresponding to the user's appendage, wherein a second region of pixels corresponds to the physical peripheral input device;
• generate a composite image from the first image based on rendering, in place of the second region of pixels, pixels representing a virtual peripheral input device, such that the virtual peripheral input device appears in the composite image as at a same spatial location as the physical peripheral input device and covered at least partially by the user's appendage;
• and
• provide the composite image to a display device for displaying to the user.