# US 11,994,615 B2 · Compensating for a sensor deficiency in a heterogeneous sensor array > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,994,615 B2 - Document ID: US11994615B2 - Status: Granted - Kind: B2 - Application number: 17/892,862 - Filed: 2022-08-22 - Published: 2023-04-20 - Granted: 2024-05-28 - Earliest claimed: 2022-08-22 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Ignacio Alvarez; David Arditti Ilitzky; Patrick Andrew Mead; Javier Felip Leon; David Gonzalez Aguirre - Web page: https://patentgrep.com/patent/US11994615B2 ## CPC - G01S 7/4004: {of parts of a radar system} - G01S 13/86: Combinations of radar systems with non-radar systems, e.g. sonar, direction finder - G01S 7/40: Means for monitoring or calibrating - G01S 7/497: Means for monitoring or calibrating - G01S 7/52004: {Means for monitoring or calibrating (short-range imaging G01S7/5205)} - G01S 13/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 13/865: {Combination of radar systems with lidar systems} - G01S 13/87: Combinations of radar systems, e.g. primary radar and secondary radar - G01S 13/931: of land vehicles - G01S 15/931: of land vehicles - G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/931: of land vehicles - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G01S 2013/9318: {Controlling the steering} - G01S 2013/93185: {Controlling the brakes} - G01S 2013/9323: {Alternative operation using light waves} - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Apparatuses, methods and storage medium associated with compensating for a sensor deficiency in a heterogeneous sensor array are disclosed herein. In embodiments, an apparatus may include a compute device to aggregate perception data from individual perception pipelines, each of which is associated with respective one of different types of sensors of a heterogeneous sensor set, to identify a characteristic associated with a space to be monitored by the heterogeneous sensor set; detect a sensor deficiency associated with a first sensor of the sensors; and in response to a detection of the sensor deficiency, derive next perception data for more than one of the individual perception pipelines from sensor data originating from at least one second sensor of the sensors. Other embodiments may be disclosed or claimed. ## Claims ### Claim 1 An apparatus, comprising: a heterogeneous sensor array to monitor a physical space, the heterogeneous sensor array including a plurality of sensors, wherein at least one sensor of the plurality of sensors is a different type of sensor than another sensor of the plurality of sensors; and perception circuitry connected to the heterogeneous sensor array, wherein the perception circuitry is to: feed sensor data generated by respective sensors of the plurality of sensors through corresponding perception pipelines to detect objects; detect a disruption to first sensor data associated with a first sensor of the plurality of sensors; and in response to the detection of the disruption, use second sensor data associated with a second sensor of the plurality of sensors to compensate for the disruption to the first sensor data. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein each of the corresponding perception pipelines is to: detect one or more objects based on the sensor data generated by the respective sensors. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the perception circuitry is to fuse detection data output by each of the corresponding perception pipelines into a single representation of a surrounding environment. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein at least one perception pipeline of the corresponding perception pipelines includes a long short-term memory network (LSTM) model to associate objects across sensors and through time. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein at least one perception pipeline of the corresponding perception pipelines includes a deep learning model comprising an encoder network, an attention network, and a decoder network. ### Claim 6 (depends on 1) The apparatus of claim 1, wherein the perception circuitry is to, responsive to the detection of the disruption, access a sensor translation model to convert the second sensor data into synthetic first sensor data, wherein the synthetic first sensor data is to compensate for the disruption to the first sensor data. ### Claim 7 (depends on 6) The apparatus of claim 6, wherein the sensor translation model is a deep learning model comprising an encoder network, an attention network, and a decoder network. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the apparatus includes at least one processor to operate the corresponding perception pipelines or the sensor translation model. ### Claim 9 (depends on 1) The apparatus of claim 1, wherein the disruption to the first sensor data is based on at least one of a missing portion of the first sensor data, a corrupted portion of the first sensor data, noisy data among the first sensor data, portions of the first sensor data being unreliable for object detection, or the first sensor data being determined to be unusable for perception. ### Claim 10 (depends on 1) The apparatus of claim 1, wherein the disruption to the first sensor data is based on at least one of inactivity of the first sensor, a malfunction of the first sensor, failure of the first sensor, a reset of the first sensor, or security event associated with the first sensor. ### Claim 11 (depends on 1) The apparatus of claim 1, wherein the apparatus is disposed in an autonomous or semi-autonomous driving vehicle, a robot, or a mobile device. ### Claim 12 One or more non-transitory computer readable media (NTCRM) comprising instructions, wherein execution of the instructions by one or more processors of a computing device is to cause the computing device to: facilitate sensor data generated by respective sensors of a plurality of sensors to flow to corresponding object detectors, wherein at least one sensor of the plurality of sensors is a different type of sensor than at least one other sensor of the plurality of sensors; detect a disruption to first sensor data associated with a first sensor of the plurality of sensors; and in response to the detection of the disruption, use second sensor data associated with a second sensor of the plurality of sensors to compensate for the disruption to the first sensor data. ### Claim 13 (depends on 12) The one or more NTCRM of claim 12, wherein execution of the instructions is to cause the computing device to operate each of the corresponding object detectors to: detect one or more objects based on the sensor data generated by the respective sensors. ### Claim 14 (depends on 12) The one or more NTCRM of claim 12, wherein execution of the instructions is to cause the computing device to operate each of the corresponding object detectors to: perform data fusion on detection data output by each of the corresponding object detectors into a single representation of a surrounding environment and through time. ### Claim 15 (depends on 12) The one or more NTCRM of claim 12, wherein the disruption to the first sensor data is based on at least one of a missing portion of the first sensor data, a corrupted portion of the first sensor data, noisy data among the first sensor data, portions of the first sensor data being unreliable for object detection, the first sensor data being determined to be unusable for perception, inactivity of the first sensor, a malfunction of the first sensor, failure of the first sensor, a reset of the first sensor, or a security event associated with the first sensor. ### Claim 16 (depends on 12) The one or more NTCRM of claim 12, wherein execution of the instructions is to cause the computing device to: operate a long short-term memory network (LSTM) model to associate objects across sensors and through time. ### Claim 17 (depends on 12) The one or more NTCRM of claim 12, wherein execution of the instructions is to cause the computing device to: operate a sensor translation model to convert the second sensor data into synthetic first sensor data, wherein the synthetic first sensor data is to compensate for the disruption to the first sensor data. ### Claim 18 (depends on 17) The one or more NTCRM of claim 17, wherein the sensor translation model is a deep learning model comprising an encoder network, an attention network, and a decoder network. ### Claim 19 A computing system, comprising: interface circuitry to connect the computing system to one or more mechanical components, and connect the computing system to a heterogeneous sensor array to monitor a physical space surrounding an autonomous or semi-autonomous system, wherein the heterogeneous sensor array includes a plurality of sensors, wherein at least one sensor of the plurality of sensors is a different type of sensor than other sensors of the plurality of sensors; and perception circuitry connected to the heterogeneous sensor array via the interface circuitry, wherein the perception circuitry is to: feed sensor data generated by respective sensors of the plurality of sensors through corresponding perception pipelines to corresponding object detectors; detect a disruption to first sensor data associated with a first sensor of the plurality of sensors; in response to the detection of the disruption, use second sensor data associated with a second sensor of the plurality of sensors to compensate for the disruption to the first sensor data; determine a trajectory based on perception data output by the corresponding object detectors, wherein the perception data is based on the compensation for the disrupted first sensor data; and control the one or more mechanical components based on the determined trajectory. ### Claim 20 (depends on 19) The computing system of claim 19, wherein the one or more mechanical components include one or more of a propulsion system, a steering mechanism, and a braking mechanism. ### Claim 21 (depends on 19) The computing system of claim 19, wherein each of the corresponding object detectors is to: detect one or more objects based on the sensor data generated by the respective sensors. ### Claim 22 (depends on 19) The computing system of claim 19, wherein the perception circuitry is to feed the perception data output by each of the corresponding object detectors to a data fusion module to fuse the perception data into a single representation of a surrounding environment and through time. ### Claim 23 (depends on 19) The computing system of claim 19, wherein the perception circuitry includes a long short-term memory network (LSTM) model to associate objects across sensors and through time. ### Claim 24 (depends on 19) The computing system of claim 19, wherein the perception circuitry is to operate a deep learning model, wherein the deep learning model includes an encoder network, an attention network, and a decoder network. ### Claim 25 (depends on 24) The computing system of claim 24, wherein the deep learning model is a sensor translation model, and the perception circuitry is to operate the sensor translation model to convert the second sensor data into synthetic first sensor data, wherein the synthetic first sensor data is to compensate for the disruption to the first sensor data.