# US 12,223,738 B2 · Machine-learned explainable object detection system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,738 B2 - Document ID: US12223738B2 - Status: Granted - Kind: B2 - Application number: 17/590,264 - Filed: 2022-02-01 - Published: 2022-08-04 - Granted: 2025-02-11 - Earliest claimed: 2021-02-01 - Assignees: Hitachi Rail GTS Canada Inc. - Applicants: Hitachi Rail GTS Canada Inc. - Inventors: Veronica Marin; Evgeny Nuger; Jaewook Jung; William Wang; David Beach - Web page: https://patentgrep.com/patent/US12223738B2 ## CPC - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - B61L 23/041: {Obstacle detection} - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G06V 10/82: using neural networks - G06V 20/582: {of traffic signs} - G06V 20/584: {of vehicle lights or traffic lights} - B60L 2200/26: Rail vehicles - B60L 2240/62: Vehicle position - B60L 2240/70: Interactions with external data bases, e.g. traffic centres - B61L 2205/04: Satellite based navigation systems, e.g. global positioning system [GPS] - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B61L: GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC (brakes or auxiliary equipment B61H, B61K; point or crossing construction E01B) - B60L: PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES (arrangements or mounting of electrical propulsion units or of plural diverse prime-movers for mutual or common propulsion in vehicles B60K1/00, B60K6/20; arrangements or mounting of electrical gearing in vehicles B60K17/12, B60K17/14; preventing wheel slip by reducing power in rail vehicles B61C15/08; dynamo-electric machines H02K; control or regulation of electric motors H02P); SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES (electric coupling devices combined with mechanical couplings of vehicles B60D1/64; electric heating for vehicles B60H1/00); ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL (control or regulation of electric motors H02P); MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES ## Abstract A method of detecting a way-side object includes receiving data frames from sensors mounted on a vehicle. Position data corresponding to the position of the vehicle is received. Object-of-interest data is retrieved from a database. The sensor data frames and the object-of-interest data are processed to determine a region-of-interest in the sensor data frame. A portion of the sensor data frames corresponding to the region of interest is processed using machine-learned object detection to identify a first object-of-interest. The portion of the sensor data frames corresponding to the region of interest is processed using computer vision methods to detect features of the expected object-of-interest and identifying the detected object-of-interest as explained when the features of the expected object-of-interest are detected. Explained object-of-interest data corresponding to the explained detected object-of-interest is output to a navigation system of the vehicle. ## Claims ### Claim 1 A method of detecting a way-side object comprising; receiving sensor data frames corresponding to a first time from sensors mounted on a vehicle, receiving position data corresponding to a first position of the vehicle at the first time; retrieving object-of-interest data from a database, wherein the object-of-interest data corresponds to an expected object-of-interest detectable by the sensors at the first position of the vehicle; processing the sensor data frames and the object-of-interest data to determine a region-of-interest in at least one frame of the sensor data frames; processing a portion of the sensor data frames corresponding to the region of interest using machine-learned object detection to identify a first object-of-interest; processing the portion of the sensor data frames corresponding to the region of interest using computer vision methods to detect features of the expected object-of-interest and identifying the detected object-of-interest as explained when the features of the expected object-of-interest are detected; and outputting explained object-of-interest data corresponding to the explained detected object-of-interest to a navigation system of the vehicle. ### Claim 2 (depends on 1) The method of claim 1, further comprising receiving and processing sensor data frames corresponding to a series of times to generate a series of explained object-of-interest data. ### Claim 3 (depends on 2) The method of claim 2, wherein the series of explained object-of-interest is compared to validate the explained object-of-interest. ### Claim 4 (depends on 1) The method of claim 1, wherein the machine-learned object detection is a deep convolutional neural network. ### Claim 5 (depends on 3) The method of claim 3, further comprising comparing the validated explained object-of-interest with expected object-of-interest data to identify detection failures. ### Claim 6 An explained machine learning object detection system, comprising: an input module connected to sensors and a data server, wherein the input module receives data frames from the sensors corresponding to a first time and wherein the input module receives data corresponding to the sensors position and objects-of-interest detectable from the sensors position; a region-of-interest module connected to the input module, wherein the region-of-interest module determines, using object-of-interest data, regions-of-interest of the data frames; a machine-learned object detection module connected to the region-of-interest module, wherein the machine-learned object detection module uses the regions-of-interest of the data frames to detect objects in the regions of interest of the data frames; and an explainable detection module connected to the machine-learned detection module, wherein the explainable detection module uses the regions of interest of the data frames where objects have been detected to identify objects of interest using computer vision processing and outputting explained detected object data. ### Claim 7 (depends on 6) The system of claim 6, wherein the sensors are at least one of cameras, RADAR or LIDAR. ### Claim 8 (depends on 6) The system of claim 6, further comprising a tracking module, connected to the explainable detection module and receiving explained detected object data, storing the explained detected object data in memory and validating an explained detected object. ### Claim 9 (depends on 8) The system of claim 8, further comprising a tracking supervision module, connected to the tracking module and receiving validated explained detected object data, comparing the validated explained detected object data with expected object-of-interest data to identify detection failures. ### Claim 10 (depends on 6) The system of claim 6, further comprising a detection supervision module connected to the machine-learned module and receiving object detections from the machine-learned module and comparing the object detection with expected object-of-interest data to identify detection failures. ### Claim 11 A method of detecting a way-side object comprising; receiving a sensor data frames corresponding to a first time from sensors mounted on a vehicle, receiving position data corresponding to the first position of the vehicle at the first time; retrieving object-of-interest data from a database, wherein the object-of-interest data corresponds to an expected object-of-interest detectable by the sensors at the first position of the vehicle; processing the sensor data frames and the object-of-interest data to determine a region-of-interest in the sensor data frame; processing a portion of the sensor data frames corresponding to the region of interest using machine-learned object detection to identify a first object-of-interest; processing the portion of the sensor data frames corresponding to the region of interest using computer vision methods to detect features of the expected object-of-interest and identifying the detected object-of-interest as explained when the features of the expected object-of-interest are detected; processing sensor data frames corresponding to a series of times to generate a series of explained object-of-interest data; validating an object detection using the series of explained object-of-interest data; and outputting explained object-of-interest data corresponding to the validated object detection to a navigation system of the vehicle. ### Claim 12 (depends on 11) The method of claim 11, further comprising comparing the validated object detections with expected object-of-interest data to identify object detection failures. ### Claim 13 (depends on 12) The method of claim 12, further comprising calibrating the sensors based on the object detection failures. ### Claim 14 (depends on 12) The method of claim 12, further comprising calibrating the machine-learned object detection based on the object detection failures. ### Claim 15 (depends on 12) The method of claim 12, wherein the expected object-of-interest data corresponds to a calibration object. ### Claim 16 (depends on 13) The method of claim 13 wherein the calibrating is performed by modifying one or more of an aperture, a shutter speed or a focal length. ### Claim 17 (depends on 14) The method of claim 14 wherein the calibrating is performed by modifying neural-network detection probability thresholds. ### Claim 18 A method of detecting a way-side object comprising; receiving sensor data frames corresponding to a first time from sensors mounted on a vehicle; retrieving object-of-interest data from a database, wherein the object-of-interest data corresponds to an expected object-of-interest detectable by the sensors at a first position of the vehicle; processing the sensor data frames and the object-of-interest data to determine a region-of-interest in at least one frame of the sensor data frames; processing a portion of the sensor data frames corresponding to the region of interest using machine-learned object detection to identify a first object-of-interest; processing the portion of the sensor data frames corresponding to the region of interest using computer vision methods to detect features of the expected object-of-interest and identifying the detected object-of-interest as explained when the features of the expected object-of-interest are detected; and outputting explained object-of-interest data corresponding to the explained detected object-of-interest to a navigation system of the vehicle. ### Claim 19 (depends on 18) The method of detecting a way-side object of claim 18, wherein the region of interest determined for the at least one frame of the sensor data frames is the entirety of at least one frame. ### Claim 20 (depends on 18) The method of detecting a way-side object of claim 18, further comprising comparing the expected object-of-interest and detected object-of-interest to determine a failure rate and wherein the failure rate is used to identify a safety level.