# US 12,761,672 B2 · Image-based method for simplifying a vehicle-external takeover of control of a motor vehicle, assistance device, and motor vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,761,672 B2 - Document ID: US12761672B2 - Status: Granted - Kind: B2 - Application number: 18/273,589 - Filed: 2022-01-10 - Published: 2024-03-28 - Granted: 2026-10-13 - Earliest claimed: 2021-01-22 - Assignees: Bayerische Motoren Werke Aktiengesellschaft - Applicants: Bayerische Motoren Werke Aktiengesellschaft - Inventors: Gerhard Graf; Christopher Kuhn; Zhenxin Zhang - Web page: https://patentgrep.com/patent/US12761672B2 ## CPC - G05D 1/80: Arrangements for reacting to or preventing system or operator failure (handing over between remote control and on-board control, or handing over between remote control arrangements G05D1/227) - G05D 1/2247: {providing the operator with simple or augmented images from one or more cameras} - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04W 4/40: for vehicles, e.g. vehicle-to-pedestrians [V2P] - G05D 2109/10: Land vehicles - G05D 2111/30: Radio signals - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20112: Image segmentation details - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A method is provided for simplifying a takeover of control of a motor vehicle by a vehicle-external operator. In the method, images of the surroundings of the vehicle are captured from the vehicle and semantically segmented. Errors in a corresponding segmentation model are predicted on the basis of at least one such image each. If a corresponding error prediction triggering a request for the takeover of control is made, an image-based visualization is automatically generated in which exactly one region corresponding to the error prediction is visually highlighted. The request and the visualization are then sent to the vehicle-external operator. ## Claims ### Claim 1 A method for simplifying a takeover of control of a motor vehicle by a vehicle-external operator, the method comprising: in a conditionally automated operation of the motor vehicle, acquiring and semantically segmenting images of an environment of the motor vehicle by way of a predetermined trained segmentation model, based on at least one of the images in each case, predicting errors of the segmentation model, for an error prediction, which, according to a predetermined criterion, triggers an automatic output of a request for the takeover of control by the vehicle-external operator, automatically generating an image-based visualization that includes an image and visual highligting of an area of the image corresponding to the error prediction, wherein the visual highlighting is superimposed on the image, and sending the request and the visualization to the vehicle-external operator, wherein: the errors of the segmentation model are predicted pixel by pixel, an average error is determined based on the errors of the segmentation model for the respective image, and it is checked as at least a part of the predetermined criterion whether the average error is greater than a first predetermined error threshold value. ### Claim 2 (depends on 1) The method according to claim 1, wherein: a number of the predicted errors is determined based on the errors of the segmentation model for the respective image, and it is checked as part of the predetermined criterion whether the number of the errors is greater than a second predetermined error threshold value. ### Claim 3 (depends on 1) The method according to claim 1, wherein: a size of a coherent area of error pixels is determined, and whether the size corresponds at least to a predetermined size threshold value is checked as part of the predetermiend criterion. ### Claim 4 (depends on 1) The method according to claim 1, wherein: by way of a predetermined reconstruction model, from a semantic segmentation, the image underlying the semantic segmentation is approximated by generating a corresponding reconstruction image and the respective visualization is generated based on the reconstruction image. ### Claim 5 (depends on 4) The method according to claim 4, wherein: the reconstruction model comprises generative adversarial networks. ### Claim 6 (depends on 4) The method according to claim 4, wherein: to predict the errors, the reconstruction image is compared to the respective underlying acquired image and the errors are predicted based on detected differences. ### Claim 7 (depends on 1) The method according to claim 1, wherein: the visualization is generated in a form of a heat map. ### Claim 8 (depends on 1) The method according to claim 1, further comprising: determining which functionality is affected by the errors, and sending the functionality with the request to the vehicle-external operator. ### Claim 9 (depends on 1) An assistance unit for the motor vehicle, the assistance unit comprising: an input interface for acquiring the images, a data storage unit, a processor unit, and an output interface for outputting the request for the takeover of control by the vehicle-external operator and the visualization, wherein the assistance unit is configured to carry out the method according to claim 1. ### Claim 10 (depends on 9) A motor vehicle comprising: a camera for recording the images, the assistance unit according to claim 9, wherein the assistance unit is connected to the camera, and a communication unit for wirelessly sending the request for the takeover of control and the visualization and for wirelessly receiving control signals for control of the motor vehicle. ### Claim 11 (depends on 1) The method according to claim 1, wherein: the average error is determined as an average error probability, confidence, or degree of severity over all pixels of the respective image.