# US 2024/0203084 A1 · System and method for augmented decision making by detecting occlusions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0203084 A1 - Document ID: US20240203084A1 - Status: Published application - Kind: A1 - Application number: 18/066,008 - Filed: 2022-12-14 - Published: 2024-06-20 - Earliest claimed: 2022-12-14 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Manoj Sharma; Jacob Alan Bond; Joseph F. Szczerba - Web page: https://patentgrep.com/patent/US20240203084A1 ## CPC - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06T 5/77: Retouching; Inpainting; Scratch removal - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/582: {of traffic signs} - G06T 2207/30256: Lane; Road marking - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and method for selectively augmenting information in a vehicle includes receiving a scene information from at least one sensor. identifying a region of interest from the scene information, analyzing the scene information to determine whether the region of interest is occluded, and augmenting the region of interest using a display system when the region of interest is occluded. The region of interest includes an object identified within the scene information that is classified as related to vehicle safety. ## Claims ### Claim 1 A method for selectively augmenting information in a vehicle, the method comprising: receiving a scene information from at least one sensor; identifying a region of interest from the scene information; analyzing the scene information to determine whether the region of interest is occluded; and augmenting the region of interest using a display system when the region of interest is occluded. ### Claim 2 (depends on 1) The method of claim 1, wherein the scene information includes data pertaining to an environment surrounding the vehicle. ### Claim 3 (depends on 1) The method of claim 1, wherein the region of interest includes an object identified within the scene information that is classified as related to vehicle safety. ### Claim 4 (depends on 3) The method of claim 3, wherein related to vehicle safety includes a road sign, a brake light, a pedestrian, a traffic light, and a traffic sign. ### Claim 5 (depends on 1) The method of claim 1, wherein the region of interest includes an object identified by an occupant demand for augmentation. ### Claim 6 (depends on 5) The method of claim 5, wherein identified by an occupant demand for augmentation includes preferences for augmentation set by the occupant. ### Claim 7 (depends on 1) The method of claim 1, wherein identifying a region of interest includes passing the scene information through a neural network. ### Claim 8 (depends on 1) The method of claim 1, wherein analyzing the scene information includes passing the scene information through a neural network to determine whether the region of interest is occluded. ### Claim 9 (depends on 8) The method of claim 8, wherein the region of interest is occluded when the region of interest is blocked, missing, or is not viewable by an occupant of the vehicle due to direct sunlight. ### Claim 10 (depends on 8) The method of claim 8, wherein the region of interest is occluded when the region of interest is blocked or missing greater than a predetermined occlusion threshold. ### Claim 11 (depends on 8) The method of claim 8, wherein analyzing the scene information further comprises: identifying whether the region of interest contains a traffic light; determining the traffic light state; converting a red, green, or red light of the traffic light to an intensity image; calculating contrast values of the of the intensity image; determining whether the contrast values are less than a contrast threshold; and augmenting the traffic light state, using the display system, when the contrast is less than the contrast threshold; and not augmenting the traffic light state, using the display system, when the contrast is greater than the contrast threshold. ### Claim 12 (depends on 8) The method of claim 8, wherein determining the traffic light state comprises passing the region of interest containing the traffic light through a neural network. ### Claim 13 (depends on 8) The method of claim 8, wherein analyzing the scene further comprises: identifying whether the region of interest includes a pedestrian; determining whether the pedestrian is occluded; augmenting the region of interest, using the display system, when the pedestrian is occluded; and not augmenting the region of interest, using the display system, when the pedestrian is not occluded. ### Claim 14 (depends on 8) The method of claim 8, wherein analyzing the scene information further comprises: identifying whether the region of interest includes a traffic sign; determining a type of traffic sign by passing the scene information through a neural network; determining whether the traffic sign is occluded; augmenting the type of traffic sign, using the display system, when the traffic sign is occluded; and not augmenting the traffic sign type, using the display system, when the traffic sign is not occluded. ### Claim 15 (depends on 8) The method of claim 8, wherein analyzing the scene information further comprises: identifying a road marker in the region of interest; determining whether the road marker is occluded; augmenting the road marker, using the display system, when the road marker is occluded; and not augmenting the road marker, using the display system, when the road marker is not occluded. ### Claim 16 (depends on 8) The method of claim 8, wherein analyzing the scene information further comprises: identifying whether the region of interest contains a remote vehicle in front of the vehicle; detecting whether the remote vehicle is decelerating; detecting whether brake lights on the remote vehicle are occluded; augmenting the brake lights based on the determination that the remote vehicle is decelerating and the brake lights are occluded; and not augmenting the brake lights based on the determination that the remote vehicle is either not decelerating and or the brake lights are not occluded. ### Claim 17 (depends on 1) The method of claim 1, augmenting the region of interest includes graphically removing the occlusion or displaying a graphic on or near the region of interest. ### Claim 18 A system for selectively augmenting information in a vehicle, the system comprising: at least one sensor, wherein the at least one sensor is used to acquire information about an environment surrounding the vehicle; a display system located in view of an occupant; and a controller in electrical communication with the at least one sensor and the display system, wherein the controller is programed to: receive a scene information from the at least one sensor; identify a region of interest from the scene information; analyze the scene information to determine whether the region of interest is occluded; and augment the region of interest, using the display system, when the region of interest is occluded. ### Claim 19 (depends on 18) The system of claim 18, wherein the at least one sensor includes a camera, a RADAR, a LIDAR, or a V2X and wherein the display system includes an AR-HUD, AR-cluster, TWD system, or a direct view display. ### Claim 20 A computer for a selective augmentation system of a vehicle, the system notifying an occupant of the vehicle about an occluded region of interest, the system further including an at least one sensor for acquiring information about an environment surrounding the vehicle, the system further including a display system, located in view of an occupant, for displaying augmented regions of interested to the occupant, the system further including a controller in electrical communication with the at least one sensor and the display system, for augmenting the region of interest, the controller comprising: at least one processor electrically communicating with the at least one sensor and the display system; and a non-transitory computer readable storage medium storing instructions such that the at least one processor is programmed to: receive a scene information from the at least one sensor; identify a region of interest from the scene information; analyze the scene information to determine whether the region of interest is occluded; and augment the region of interest using the display system when the region of interest is occluded.