Systems and methods for context-aware visibility improvements in low-visibility conditions

A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

• at least one visible-light camera;
• at least one depth camera;
• a display unit;
• an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle;
• and
• at least one processor configured to:
• obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;
• control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;
• for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;
• identify at least one corresponding image segment in the given visible-light image whose visual information is any one of:
• occluded, degraded, based on:
• (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;
• generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image;
• and
• display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;
• wherein when identifying the at least one image segment, the at least one processor is further configured to:
• process, by utilising a movement filtering technique, the corresponding depth image;
• and
• identify the at least one object as a moving object present in the region of the real-world environment, wherein when generating the image, the at least one processor is configured to generate a virtual element corresponding to the moving object.

A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

• at least one visible-light camera;
• at least one depth camera;
• a display unit;
• an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle;
• and
• at least one processor configured to:
• obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;
• control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;
• for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;
• identify at least one corresponding image segment in the given visible-light image whose visual information is any one of:
• occluded, degraded, based on:
• (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;
• and
• generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image;
• and
• display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;
• the system further comprising a tracker, wherein the at least one processor is further configured to:
• determine gaze directions of the eyes of the at least one user, by utilising the tracker;
• determine a focus depth at which the at least one user is gazing, based on the gaze directions of the eyes;
• detect when a difference between the focus depth and an optical depth at which the at least one virtual element is being presented via the synthetic light field is greater than a predefined threshold difference;
• and
• when it is detected that said difference is greater than the predefined threshold difference, skip generating the image and displaying the image.

A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

• at least one visible-light camera;
• at least one depth camera;
• a display unit;
• an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle;
• and
• at least one processor configured to:
• obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;
• control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;
• for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;
• identify at least one corresponding image segment in the given visible-light image whose visual information is any one of:
• occluded, degraded, based on:
• (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;
• generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image;
• and
• display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;
• the system further comprising a tracker, wherein the at least one virtual element comprises a plurality of virtual elements, and wherein the at least one processor is further configured to:
• determine gaze directions of the eyes of the at least one user, by utilising the tracker;
• determine a focus depth at which the at least one user is gazing, based on the gaze directions of the eyes;
• detect a difference between the focus depth and an optical depth at which respective ones of the plurality of virtual elements are to be presented via the synthetic light field;
• and
• perform digital superimposition of only those virtual elements on the at least one corresponding image segment for which said difference is less than a predefined threshold difference, when generating the image.

A system for improving visibility in low-visibility conditions, the system being implemented in a vehicle, the system comprising:

• at least one visible-light camera;
• at least one depth camera;
• a display unit;
• an optical combiner arranged on an optical path of the display unit and on an optical path of a real-world light field of a real-world environment, wherein the real-world light field is incoming via a windshield of the vehicle;
• and
• at least one processor configured to:
• obtain information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of the optical combiner;
• control the at least one visible-light camera and the at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;
• for a given visible-light image from amongst the plurality of visible-light images, identify, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;
• identify at least one corresponding image segment in the given visible-light image whose visual information is any one of:
• occluded, degraded, based on:
• (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;
• generate an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image;
• and
• display the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;
• wherein prior to identifying the at least one image segment in the corresponding depth image, and identifying the at least one corresponding image segment in the given visible-light image, the at least one processor is further configured to:
• reproject the corresponding depth image, from a camera pose of the at least one depth camera to a head pose of the at least one user or an eye pose of a given eye of the at least one user;
• and
• reproject the given visible-light image, from a camera pose of the at least one visible-light camera to the head pose of the at least one user or the eye pose of the given eye of the at least one user, based on depth information in the corresponding depth image.

A method for improving visibility in low-visibility conditions, the method being implemented in a vehicle, the method comprising:

• obtaining information indicative of a relative position of a head or eyes of at least one user with respect to a semi-reflective surface of an optical combiner, the optical combiner being arranged on an optical path of a display unit and on an optical path of a real-world light field of a real-world environment, the real-world light field incoming via a windshield of the vehicle;
• controlling at least one visible-light camera and at least one depth camera to capture, contemporaneously, a plurality of visible-light images and a plurality of depth images, of a region of the real-world environment that is front of the vehicle, respectively;
• for a given visible-light image from amongst the plurality of visible-light images, identifying, in a corresponding depth image from amongst the plurality of depth images, at least one image segment that represents at least one object present in the region of the real-world environment;
• identifying at least one corresponding image segment in the given visible-light image whose visual information is any one of:
• occluded, degraded, based on:
• (i) the at least one image segment that is identified in the corresponding depth image, and (ii) the relative position of the head or the eyes of the at least one user with respect to the semi-reflective surface of the optical combiner;
• and
• generating an image by digitally superimposing at least one virtual element on the at least one corresponding image segment of the given visible-light image;
• and
• displaying the image via the display unit to produce a synthetic light field, wherein the optical combiner is employed to reflect the synthetic light field towards the eyes of the at least one user, whilst optically combining the synthetic light field with the real-world light field;
• wherein the image is a light field image, and the display unit is a light field display unit comprising a multiscopic optical element configured to direct a first part and a second part of the synthetic light field towards a first eye and a second eye of an individual one of the at least one user, via the optical combiner, presenting a first image and a second image to the first eye and the second eye, respectively, and the light field image is generated by utilising the first image and the second image.