Digital mirror monitoring system using microsensors

A digital mirror (DM) monitoring system for a vehicle, the system comprising:

• one or more optical sensors disposed on the vehicle, the one or more sensors capturing optical information about vehicle surroundings;
• one or more digital mirrors disposed on the vehicle, the one or more digital mirrors each having a display device with a microLED array for displaying images, and a microsensor array disposed within the microLED array for measuring images displayed by the microLED array, wherein each of the one or more digital mirrors further comprises:
• a microLED array having a plurality of microLEDs disposed thereon;
• and
• a microsensor array disposed within the microLED array, wherein a quantity of microsensors in the microsensor array is significantly smaller than a quantity of microLEDs in the microLED array;
• and
• wherein the microsensor array monitors the optical information displayed on the one or more digital mirrors;
• one or more controllers, each of the one or more controllers having a processor, a memory, and one or more input/output (I/O) ports, the memory storing programmatic logic including a digital mirror monitoring application (DMM application), the DMM application comprising:
• a first control logic that causes the optical sensors to obtain images of the surroundings of the vehicle;
• a second control logic causes the optical sensors to send the images of the surroundings of the vehicle to the controller, wherein the images are received by the controller;
• a third control logic that causes the controller to transmit a display command to the microLED arrays of the one or more digital mirrors via a source driver;
• a fourth control logic that causes the source driver to pass the display command to individual microLEDs in the microLED array;
• a fifth control logic that utilizes the microsensors to monitor light output of the microLEDs;
• a sixth control logic that ascertains a level of similarity between the optical information captured by the one or more optical sensors and light output image data captured by the microsensors, including:
• control logic that includes an algorithm that defines when the display device is properly displaying the optical information captured by the one or more sensors by comparing a source signal from the one or more sensors to a display signal from the microsensor array, wherein the algorithm further comprises:
• control logic that determines whether the source signal and the display signal are semantically identical by applying the following equation to the source and display signals:
• M
• ⁢
• SE
• ⁡
• (
• I
• A
• ,
• I
• B
• )
• =
• 1
• m
• ⁢
• n
• ⁢
• ∑
• i
• =
• 0
• m
• -
• 1
• ∑
• j
• =
• 0
• n
• -
• 1
• |
• I
• A
• (
• i
• ,
• j
• )
• -
• I
• B
• (
• i
• ,
• j
• )
• |
• 2
• where MSE(IA, IB) is a mean-square error of the source signal (IA), and the display signal from the microsensor array (IB), wherein semantically identical source and display signals include identical objects within the optical information, and wherein semantically identical source and display signals result in a mean-square error having a value that approaches or is zero;
• utilizing twin networks with shared weights to compare the source and display signals (IA, IB) in real-time using a contrastive loss function L defined as:
• L
• =
• (
• 1
• -
• y
• )
• *
• ||
• x
• i
• -
• x
• j
• ||
• 2
• +
• y
• *
• max
• ⁡
• (
• 0
• ,
• m
• -
• ||
• x
• i
• -
• x
• j
• ||
• 2
• )
• where the contrastive loss function L defines a level of similarity between feature vectors of objects detected within the source and display signals (IA, IB);
• and
• wherein the twin networks further comprise:
• two identical neural networks (NNs) using precisely the same parameters and weights, wherein training inputs to the network include training pairs of images that have similar contents, and training pairs of images that have dissimilar contents, wherein while training the twin networks, the training pairs of images are passed through the identical NNs and feature vectors are extracted for each image of the training pairs of images and when training pairs of images are similar, the feature vectors are similar, and wherein when training pairs of images are not similar, the feature vectors are also not similar;
• and
• a seventh control logic that determines whether the microLED array is operating accurately and in unison with the optical sensor, upon determining that the one or more display devices are functioning properly, the seventh control logic causes the microsensors to continue to monitor the one or more display devices;
• and upon determining that the one or more display devices are not functioning properly, the seventh control logic generates a notification and transmits the notification to a vehicle operator, the notification indicating that one or more DMs of the vehicle is not functioning properly, the seventh control logic forwarding the notification to a vehicle manufacturer;
• and causing the vehicle manufacturer to schedule the vehicle for service, causing the vehicle manufacturer to send an over-the-air (OTA) update to realign or otherwise address DM performance issues identified in the notification.

A method for digital mirror (DM) monitoring in a vehicle, the method comprising:

• capturing optical information about the vehicle's surroundings with one or more optical sensors disposed on the vehicle;
• displaying images on one or more digital mirrors disposed on the vehicle, the one or more digital mirrors each defining a display device having a microLED array for displaying the images and a microsensor array disposed within the microLED array for measuring the images displayed on the microLED array, wherein each of the one or more display devices further comprises:
• a microLED array having a plurality of microLEDs disposed thereon;
• and
• a microsensor array disposed within the microLED array, wherein a quantity of microsensors in the microsensor array is significantly smaller than a quantity of microLEDs in the microLED array;
• and
• wherein the microsensor array monitors the optical information displayed on the display devices;
• executing programmatic control logic stored memory of one or more controllers, each of the one or more controllers having a processor, the memory, and one or more input/output (I/O) ports, the programmatic control logic including a digital mirror monitoring application (DMM application) comprising control logic for:
• causing the optical sensors to obtain images of the surroundings of the vehicle;
• sending, from the optical sensors to the controller, the images of the surroundings of the vehicle and receiving the images within the controller;
• transmitting, via the controller, a display command to the microLED arrays of the one or more digital mirrors via a source driver;
• causing the source driver to pass the display command to individual microLEDs in the microLED array;
• monitoring, via the microsensors, light output of the microLEDs;
• ascertaining a level of similarity between the optical information captured by the one or more optical sensors and light output image data captured by the microsensors;
• and
• determining whether the microLED array is operating accurately and in unison with the optical sensor, upon determining that the one or more display devices are functioning properly, including:
• determining whether the source signal and the display signal are semantically identical by applying the following equation to the source and display signals:
• MSE
• ⁡
• (
• I
• A
• ,
• I
• B
• )
• =
• 1
• m
• ⁢
• n
• ⁢
• ∑
• i
• =
• 0
• m
• -
• 1
• ∑
• j
• =
• 0
• n
• -
• 1
• |
• I
• A
• (
• i
• ,
• j
• )
• -
• I
• B
• (
• i
• ,
• j
• )
• |
• 2
• where MSE(IA, IB) is a mean-square error of the source signal (IA), and the display signal from the microsensor array (IB), wherein semantically identical source and display signals include identical objects within the optical information, and wherein semantically identical source and display signals result in a mean-square error having a value that approaches or is zero;
• comparing the source and display signals (IA, IB) in real-time with twin networks having shared weights using a contrastive loss function L defined as:
• L
• =
• (
• 1
• -
• y
• )
• *
• ||
• x
• i
• -
• x
• j
• ||
• 2
• +
• y
• *
• max
• ⁡
• (
• 0
• ,
• m
• -
• ||
• x
• i
• -
• x
• j
• ||
• 2
• )
• where the contrastive loss function L defines a level of similarity between feature vectors of objects detected within the source and display signals (IA, IB), wherein the twin networks comprise:
• two identical neural networks (NNs) using precisely the same parameters and weights, wherein training inputs to the network include training pairs of images that have similar contents, and training pairs of images that have dissimilar contents, wherein while training the twin networks, the training pairs of images are passed through the identical NNs and feature vectors are extracted for each image of the training pairs of images and when training pairs of images are similar, the feature vectors are similar, and wherein when training pairs of images are not similar, the feature vectors are also not similar;
• continuing to monitor, via the microsensors, the display devices;
• and
• upon determining that the one or more display devices are not functioning properly, generating a notification and transmitting the notification to a vehicle operator, wherein the notification indicates that one or more DMs of the vehicle is not functioning properly;
• forwarding the notification to a vehicle manufacturer;
• and
• causing the vehicle manufacturer to schedule the vehicle for service, causing the vehicle manufacturer to send an over-the-air (OTA) update to realign or otherwise address DM performance issues identified in the notification.