# US 12,759,463 B2 · Digital mirror monitoring system using microsensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,759,463 B2 - Document ID: US12759463B2 - Status: Granted - Kind: B2 - Application number: 18/828,280 - Filed: 2024-09-09 - Published: 2026-03-12 - Granted: 2026-10-13 - Earliest claimed: 2024-09-09 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Jonglee Park; Manoj Kumar Sharma; Julien P. Mourou; Charles R. Quinn - Web page: https://patentgrep.com/patent/US12759463B2 ## CPC - B60R 1/26: to the rear of the vehicle - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G09G 3/32: semiconductive, e.g. using light-emitting diodes [LED] - H04N 7/183: {for receiving images from a single remote source} - B60R 2300/10: characterised by the type of camera system used - B60R 2300/20: characterised by the type of display used - B60R 2300/8066: for monitoring rearward traffic - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G09G 2380/10: Automotive applications - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A digital mirror monitoring system includes one or more sensors, the one or more sensors capturing optical information about an environment of the one or more sensors. The system further includes one or more display devices in electronic communication with the one or more sensors and displaying the optical information about the environment of the one or more sensors, and a monitoring system that determines that the one or more display devices are accurately portraying the optical information from the one or more sensors. Upon determining that the one or more display devices are functioning properly, the monitoring system continues to monitor the one or more display devices, and upon determining that the one or more display devices are not functioning properly, the monitoring system generates a notification indicating that one or more of the DMs is not functioning properly and schedules the one or more display devices for service. ## Claims ### Claim 1 A digital mirror (DM) monitoring system, the system comprising: one or more sensors, the one or more sensors capturing optical information about an environment of the one or more sensors; one or more display devices in electronic communication with the one or more sensors and displaying the optical information about the environment of the one or more sensors, wherein each of the one or more display devices further comprises: a microLED array having a plurality of microLEDs disposed thereon; and a microsensor array disposed within the microLED array, wherein a quantity of microsensors in the microsensor array is significantly smaller than a quantity of microLEDs in the microLED array; and wherein the microsensor array monitors the optical information displayed on the display devices; and a monitoring system that determines that the one or more display devices are accurately portraying the optical information from the one or more sensors, and upon determining that the one or more display devices are functioning properly, the monitoring system continues to monitor the one or more display devices, the monitoring system further comprising: one or more controllers, each of the one or more controllers having a processor, a memory, and one or more input/output (I/O) ports; the one or more sensors are in electronic communication with the one or more display devices via the I/O ports; the one or more display devices displaying the optical information about the environment of the one or more sensors from the sensors; the memory storing programmatic control logic including an algorithm that defines when the display device is properly displaying the optical information captured by the one or more sensors by comparing a source signal from the one or more sensors to a display signal from the microsensor array, wherein the algorithm further comprises: control logic that determines whether the source signal and the display signal are semantically identical by applying the following equation to the source and display signals: M ⁢ SE ⁡ ( I A , I B ) = 1 m ⁢ n ⁢ ∑ i = 0 m - 1 ∑ j = 0 n - 1 | I A ( i , j ) - I B ( i , j ) | 2 where MSE(IA, IB) is a mean-square error of the source signal (IA), and the display signal from the microsensor array (IB), wherein semantically identical source and display signals include identical objects within the optical information, and wherein semantically identical source and display signals result in a mean-square error having a value that approaches or is zero, wherein the algorithm further comprises: utilizing twin networks with shared weights to compare the source and display signals (IA, IB) in real-time using a contrastive loss function L defined as: L = ( 1 - y ) * || x i - x j || 2 + y * max ⁡ ( 0 , m - || x i - x j || 2 ) where the contrastive loss function L defines a level of similarity between feature vectors of objects detected within the source and display signals (IA, IB); and a notification is transmitted to a system operator and forwarded to a system manufacturer, wherein in response to receiving the notification, the system manufacturer schedules the one or more display devices for service or causes an over-the-air (OTA) update to be applied to the DM to realign or update the DM to address performance issues identified in the notification; and upon determining that the one or more display devices are not functioning properly, the monitoring system generates a notification indicating that one or more DMs of the system is not functioning properly and schedules the one or more display devices for service. ### Claim 2 (depends on 1) The system of claim 1, wherein the twin networks further comprise: two identical neural networks (NNs) using precisely the same parameters and weights, wherein training inputs to the network include training pairs of images that have similar contents, and training pairs of images that have dissimilar contents, wherein while training the twin networks, the training pairs of images are passed through the identical NNs and feature vectors are extracted for each image of the training pairs of images and when training pairs of images are similar, the feature vectors are similar, and wherein when training pairs of images are not similar, the feature vectors are also not similar. ### Claim 3 (depends on 1) The system of claim 1, wherein a collimator is disposed overtop of at least a portion of the microLED array and a portion of the microsensor array, wherein the collimator increases an effectiveness of microsensor light collection from the microLEDs on the display device from a first level to a second level greater than the first level. ### Claim 4 (depends on 1) The system of claim 1, wherein the microLED array is disposed on a transparent material, the microsensor array is installed behind the microLED array, relative to an exterior surface of the microLED array, and wherein the microsensors detect light emitted from the microLED array through the transparent material. ### Claim 5 (depends on 1) The system of claim 1, wherein the microLED array receives the optical information about the environment of the one or more sensors from one or more controllers via a source driver; the source driver transmits a display command to each individual microLED of the microLED array; and light emitted by the microLEDs of the microLED array is measured by the microsensors, the microsensors transmit microsensor data to a gate driver via gate bus lines and to a vehicle via sensor bus lines. ### Claim 6 A digital mirror (DM) monitoring system for a vehicle, the system comprising: one or more optical sensors disposed on the vehicle, the one or more sensors capturing optical information about vehicle surroundings; one or more digital mirrors disposed on the vehicle, the one or more digital mirrors each having a display device with a microLED array for displaying images, and a microsensor array disposed within the microLED array for measuring images displayed by the microLED array, wherein each of the one or more digital mirrors further comprises: a microLED array having a plurality of microLEDs disposed thereon; and a microsensor array disposed within the microLED array, wherein a quantity of microsensors in the microsensor array is significantly smaller than a quantity of microLEDs in the microLED array; and wherein the microsensor array monitors the optical information displayed on the one or more digital mirrors; one or more controllers, each of the one or more controllers having a processor, a memory, and one or more input/output (I/O) ports, the memory storing programmatic logic including a digital mirror monitoring application (DMM application), the DMM application comprising: a first control logic that causes the optical sensors to obtain images of the surroundings of the vehicle; a second control logic causes the optical sensors to send the images of the surroundings of the vehicle to the controller, wherein the images are received by the controller; a third control logic that causes the controller to transmit a display command to the microLED arrays of the one or more digital mirrors via a source driver; a fourth control logic that causes the source driver to pass the display command to individual microLEDs in the microLED array; a fifth control logic that utilizes the microsensors to monitor light output of the microLEDs; a sixth control logic that ascertains a level of similarity between the optical information captured by the one or more optical sensors and light output image data captured by the microsensors, including: control logic that includes an algorithm that defines when the display device is properly displaying the optical information captured by the one or more sensors by comparing a source signal from the one or more sensors to a display signal from the microsensor array, wherein the algorithm further comprises: control logic that determines whether the source signal and the display signal are semantically identical by applying the following equation to the source and display signals: M ⁢ SE ⁡ ( I A , I B ) = 1 m ⁢ n ⁢ ∑ i = 0 m - 1 ∑ j = 0 n - 1 | I A ( i , j ) - I B ( i , j ) | 2 where MSE(IA, IB) is a mean-square error of the source signal (IA), and the display signal from the microsensor array (IB), wherein semantically identical source and display signals include identical objects within the optical information, and wherein semantically identical source and display signals result in a mean-square error having a value that approaches or is zero; utilizing twin networks with shared weights to compare the source and display signals (IA, IB) in real-time using a contrastive loss function L defined as: L = ( 1 - y ) * || x i - x j || 2 + y * max ⁡ ( 0 , m - || x i - x j || 2 ) where the contrastive loss function L defines a level of similarity between feature vectors of objects detected within the source and display signals (IA, IB); and wherein the twin networks further comprise: two identical neural networks (NNs) using precisely the same parameters and weights, wherein training inputs to the network include training pairs of images that have similar contents, and training pairs of images that have dissimilar contents, wherein while training the twin networks, the training pairs of images are passed through the identical NNs and feature vectors are extracted for each image of the training pairs of images and when training pairs of images are similar, the feature vectors are similar, and wherein when training pairs of images are not similar, the feature vectors are also not similar; and a seventh control logic that determines whether the microLED array is operating accurately and in unison with the optical sensor, upon determining that the one or more display devices are functioning properly, the seventh control logic causes the microsensors to continue to monitor the one or more display devices; and upon determining that the one or more display devices are not functioning properly, the seventh control logic generates a notification and transmits the notification to a vehicle operator, the notification indicating that one or more DMs of the vehicle is not functioning properly, the seventh control logic forwarding the notification to a vehicle manufacturer; and causing the vehicle manufacturer to schedule the vehicle for service, causing the vehicle manufacturer to send an over-the-air (OTA) update to realign or otherwise address DM performance issues identified in the notification. ### Claim 7 (depends on 6) The system of claim 6, wherein a collimator is disposed overtop of at least a portion of the microLED array and a portion of the microsensor array, wherein the collimator increases an effectiveness of microsensor light collection from the microLEDs on the display device from a first level to a second level greater than the first level. ### Claim 8 (depends on 6) The system of claim 6, wherein the microLED array is disposed on a transparent material, the microsensor array is installed behind the microLED array, relative to an exterior surface of the microLED array, and wherein the microsensors detect light emitted from the microLED array through the transparent material. ### Claim 9 (depends on 6) The system of claim 6, wherein the microLED array receives the optical information about the surroundings of the vehicle from the controller via a source driver; the source driver transmits a display command to each individual microLED of the microLED array; and light emitted by the microLEDs of the microLED array is measured by the microsensors, the microsensors transmit microsensor data to a gate driver via gate bus lines and to the vehicle via sensor bus lines. ### Claim 10 A method for digital mirror (DM) monitoring in a vehicle, the method comprising: capturing optical information about the vehicle's surroundings with one or more optical sensors disposed on the vehicle; displaying images on one or more digital mirrors disposed on the vehicle, the one or more digital mirrors each defining a display device having a microLED array for displaying the images and a microsensor array disposed within the microLED array for measuring the images displayed on the microLED array, wherein each of the one or more display devices further comprises: a microLED array having a plurality of microLEDs disposed thereon; and a microsensor array disposed within the microLED array, wherein a quantity of microsensors in the microsensor array is significantly smaller than a quantity of microLEDs in the microLED array; and wherein the microsensor array monitors the optical information displayed on the display devices; executing programmatic control logic stored memory of one or more controllers, each of the one or more controllers having a processor, the memory, and one or more input/output (I/O) ports, the programmatic control logic including a digital mirror monitoring application (DMM application) comprising control logic for: causing the optical sensors to obtain images of the surroundings of the vehicle; sending, from the optical sensors to the controller, the images of the surroundings of the vehicle and receiving the images within the controller; transmitting, via the controller, a display command to the microLED arrays of the one or more digital mirrors via a source driver; causing the source driver to pass the display command to individual microLEDs in the microLED array; monitoring, via the microsensors, light output of the microLEDs; ascertaining a level of similarity between the optical information captured by the one or more optical sensors and light output image data captured by the microsensors; and determining whether the microLED array is operating accurately and in unison with the optical sensor, upon determining that the one or more display devices are functioning properly, including: determining whether the source signal and the display signal are semantically identical by applying the following equation to the source and display signals: MSE ⁡ ( I A , I B ) = 1 m ⁢ n ⁢ ∑ i = 0 m - 1 ∑ j = 0 n - 1 | I A ( i , j ) - I B ( i , j ) | 2 where MSE(IA, IB) is a mean-square error of the source signal (IA), and the display signal from the microsensor array (IB), wherein semantically identical source and display signals include identical objects within the optical information, and wherein semantically identical source and display signals result in a mean-square error having a value that approaches or is zero; comparing the source and display signals (IA, IB) in real-time with twin networks having shared weights using a contrastive loss function L defined as: L = ( 1 - y ) * || x i - x j || 2 + y * max ⁡ ( 0 , m - || x i - x j || 2 ) where the contrastive loss function L defines a level of similarity between feature vectors of objects detected within the source and display signals (IA, IB), wherein the twin networks comprise: two identical neural networks (NNs) using precisely the same parameters and weights, wherein training inputs to the network include training pairs of images that have similar contents, and training pairs of images that have dissimilar contents, wherein while training the twin networks, the training pairs of images are passed through the identical NNs and feature vectors are extracted for each image of the training pairs of images and when training pairs of images are similar, the feature vectors are similar, and wherein when training pairs of images are not similar, the feature vectors are also not similar; continuing to monitor, via the microsensors, the display devices; and upon determining that the one or more display devices are not functioning properly, generating a notification and transmitting the notification to a vehicle operator, wherein the notification indicates that one or more DMs of the vehicle is not functioning properly; forwarding the notification to a vehicle manufacturer; and causing the vehicle manufacturer to schedule the vehicle for service, causing the vehicle manufacturer to send an over-the-air (OTA) update to realign or otherwise address DM performance issues identified in the notification. ### Claim 11 (depends on 10) The method of claim 10, further comprising: defining when the display device is properly displaying the optical information captured by the one or more sensors by comparing a source signal from the one or more sensors to a display signal from the microsensor array. ### Claim 12 (depends on 10) The method of claim 10, further comprising: increasing an effectiveness of microsensor light collection from the microLEDs on the display device from a first level to a second level greater than the first level with a collimator disposed overtop of at least a portion of the microLED array and a portion of the microsensor array. ### Claim 13 (depends on 10) The method of claim 10, further comprising: placing the microLED array on a transparent material; installing the microsensor array behind the microLED array, relative to an exterior surface of the microLED array; and detecting, via the microsensors, light emitted from the microLED array through the transparent material.