Virtual reality head-mounted devices having reduced numbers of cameras, and methods of operating the same

A head-mounted device (HMD) comprising:

• an imaging sensor to capture an image using color-sensing pixels and infrared (IR)-sensing pixels located amongst the color-sensing pixels, the captured image having a color portion and an IR portion;
• a reconstructor configured to remove at least some of the IR portion from the image, and form an IR image from the removed IR portion;
• a first tracker configured to perform first tracking using the IR image;
• an image modifier to form a color image by replacing the removed IR portion with color pixels determined based on the color portion of the image and locations of the removed IR-sensing pixels in the image;
• a second tracker configured to perform a second tracking based on the color image;
• and
• a third tracker configured to perform tracking based on the IR image and the color image.

A system comprising:

• a first imaging sensor included in a head-mounted device (HMD) configured to capture a first image having a color portion captured by color-sensing pixels of the first imaging sensor and an infrared (IR) portion captured by IR-sensing pixels of the first imaging sensor;
• and
• a second imaging sensor configured to capture a second image having a color portion captured by color-sensing pixels of the second imaging sensor and an IR portion captured by IR-sensing pixels of the second imaging sensor;
• at least one memory including instructions;
• and
• at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the system to:
• form a first IR image using the IR portion of the first image;
• form a second IR image using the IR portion of the second image;
• perform an optical tracking based on at least one of the first IR image or the second IR image;
• form a first color image by replacing the IR portion of the first image with color data determined based on the color portion of the first image and a location the IR portion of the first image;
• form a second color image by replacing the IR portion of the second image with color data determined based on the color portion of the second image and a location of the IR portion of the second image;
• perform a six degrees of freedom tracking based on one or more of the first color image and the second color image;
• and
• perform a third tracking based on a stereoscopic image formed from the first IR image, the second IR image, the first color image, and the second color image.