# US 12,007,765 B2 · Method for assisting a user in the remote control of a motor vehicle, computer program product, remote-control device and driver assistance system for a motor vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,007,765 B2 - Document ID: US12007765B2 - Status: Granted - Kind: B2 - Application number: 17/311,185 - Filed: 2019-11-21 - Published: 2022-02-03 - Granted: 2024-06-11 - Earliest claimed: 2018-12-06 - Assignees: Valeo Schalter und Sensoren GmbH - Applicants: Valeo Schalter und Sensoren GmbH - Inventors: Johannes Petzold; Joerg Schrepfer; Stefan Milz - Web page: https://patentgrep.com/patent/US12007765B2 ## CPC - B60W 60/005: {Handover processes (Handing over between remote control and on-board control or handing over between remote control arrangements G05D1/227)} - G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)} - B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor - B60K 35/22: Display screens - B60K 35/80: Arrangements for controlling instruments - B60W 30/06: Automatic manoeuvring for parking - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B62D 15/0285: {Parking performed automatically} - G05D 1/0044: {by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005)} - G05D 1/224: Output arrangements on the remote controller, e.g. displays, haptics or speakers - G05D 1/225: operated by off-board computers - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60K 2360/1523: Matrix displays - B60K 2360/171: Vehicle or relevant part thereof displayed - B60K 2360/176: Camera images - B60K 2360/55: Remote control arrangements - B60K 2360/566: Mobile devices displaying vehicle information - B60K 2360/573: Mobile devices controlling vehicle functions - B60W 2050/143: {Alarm means} - B60W 2050/146: {Display means} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2554/00: Input parameters relating to objects - B62D 1/00: Steering controls, i.e. means for initiating a change of direction of the vehicle - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) ## Abstract A method for assisting a user in the remote control of a motor vehicle by a remote-control device is disclosed. Depending on sensor data provided by means of at least one environment sensor of the motor vehicle, a first surroundings image of at least one part of the surroundings of the motor vehicle from a determined perspective is provided and displayed by means of the remote-control device. In this case, the determined perspective is selected from a number of selectable perspectives as a perspective that is independent of a position of the at least one environment sensor, depending on a user input effected by the user and detected by means of the remote-control device. ## Claims ### Claim 1 A method for assisting a user in the remote control of a motor vehicle by a remote-control device, comprising: receiving, by the remote-control device from the user in proximity of the remote-control device and remote from the motor vehicle, a user input that specifies a point in a three-dimensional (3D) vehicle model of the motor vehicle, wherein the point in the 3D vehicle model corresponds to an interior location within the motor vehicle; depending on sensor data provided by at least one environment sensor of the motor vehicle, providing a first surroundings image of at least one part of the surroundings of the motor vehicle viewed from the interior location of the motor vehicle as a determined perspective, wherein the determined perspective is represented by using the point in the 3D vehicle model as a virtual viewpoint, wherein the first surroundings image comprises the sensor data and an interior portion of the 3D vehicle model; displaying, while the remote-control device and the user are remote from the motor vehicle, the first surroundings image by the remote-control device to be viewed by the user as if the user is at the interior location within the motor vehicle; and selecting the determined perspective from a plurality of selectable perspectives as a perspective that is independent of a position of the at least one environment sensor, depending on the user input effected by the user and detected by the remote-control device, wherein the remote control of the motor vehicle is carried out independently of a distance between the remote-control device and the motor vehicle. ### Claim 2 (depends on 1) The method according to claim 1, wherein depending on the selected perspective a view window on a defined projective surface is determined and the first surroundings image is calculated as part of a 3D all-round view around the motor vehicle, said 3D all-round view being able to be provided from the sensor data, depending on the determined view window on the defined projective surface, such that the first surroundings image represents the part of the surroundings that lies within the view window from the selected perspective. ### Claim 3 (depends on 2) The method according to claim 2, wherein the projective surface constitutes a curved surface. ### Claim 4 (depends on 1) The method according to claim 1, wherein a second virtual viewpoint assigned to a second perspective is freely selectable on one or a plurality of predetermined lines or in three-dimensional space. ### Claim 5 (depends on 4) The method according to claim 4, wherein the second virtual viewpoint assigned to the second perspective is selectable inside the motor vehicle and outside the motor vehicle. ### Claim 6 (depends on 1) The method according to claim 1, wherein in the first surroundings image at least one part of a motor vehicle representation of the motor vehicle is represented from the selected perspective. ### Claim 7 (depends on 1) The method according to claim 1, wherein the first surroundings image is calculated from the sensor data by the motor vehicle and communicated to the remote-control device. ### Claim 8 (depends on 1) The method according to claim 1, wherein the sensor data are communicated from the motor vehicle to the remote-control device and the first surroundings image is calculated from the sensor data by the remote-control device. ### Claim 9 (depends on 1) The method according to claim 1, wherein the fact of whether the first surroundings image is calculated from the sensor data by the motor vehicle or by the remote-control device is determined depending on a type of the communication connection between motor vehicle and remote-control device and/or a bandwidth and/or a data transfer rate of the data transfer between motor vehicle and remote-control device. ### Claim 10 (depends on 8) The method according to claim 8, wherein when the bandwidth of the communication connection and/or the data transfer rate of the communication connection between motor vehicle and remote-control device fall(s) below a predetermined limit value, the first surroundings image is calculated from the sensor data by the motor vehicle and communicated to the remote-control device, and when the bandwidth of the communication connection and/or the data transfer rate of the communication connection between motor vehicle and remote-control device do(es) not fall below the predetermined limit value, the sensor data are communicated to the remote-control device and the first surroundings image is calculated from the sensor data by the remote-control device. ### Claim 11 (depends on 1) The method according to claim 1, wherein depending on the sensor data at least one second surroundings image from a second perspective is provided and displayed simultaneously with the first surroundings image by the remote-control device. ### Claim 12 (depends on 1) A computer program product, comprising instructions which, when executed by a processor of an electronic control device, cause said processor to carry out a method according to claim 1. ### Claim 13 A remote-control device for assisting a user in the remote control of a motor vehicle, wherein the remote-control device is configured to receive, from the user in proximity of the remote-control device and remote from the motor vehicle, a user input that specifies a point in a three-dimensional (3D) vehicle model of the motor vehicle, wherein the point in the 3D vehicle model corresponds to an interior location within the motor vehicle; depending on sensor data provided by at least one environment sensor of the motor vehicle, obtain a first surroundings image of at least one part of the surroundings of the motor vehicle viewed from the interior location of the motor vehicle as a determined perspective, wherein the determined perspective is represented by using the point in the 3D vehicle model as a virtual viewpoint, wherein the first surroundings image comprises the sensor data and an interior portion of the 3D vehicle model; displaying, while the remote-control device and the user are remote from the motor vehicle, the first surroundings image by the remote-control device to be viewed by the user as if the user is at the interior location within the motor vehicle; and select the determined perspective from a plurality of selectable perspectives as a perspective that is independent of a position of the at least one environment sensor, depending on the user input effected by the user and detected by the remote-control device, wherein the remote control of the motor vehicle is carried out independently of a distance between the remote-control device and the motor vehicle. ### Claim 14 A driver assistance system for a motor vehicle for assisting a user in the remote control of the motor vehicle by a remote-control device, wherein the driver assistance system is configured, depending on sensor data provided by at least one environment sensor of the motor vehicle, to: provide a surroundings image of at least one part of the surroundings of the motor vehicle from a determined perspective and to communicate the surroundings image to the remote-control device for displaying, while the remote-control device and the user are remote from the motor vehicle, the surroundings image to be viewed by the user as if the user is at an interior location within the motor vehicle; receive from the remote-control device selection information dependent on a user input effected by the user and detected by means of the remote-control device and, depending on the received selection information, to select the determined perspective from a plurality of selectable perspectives as a perspective that is independent of a position of the at least one environment sensor, wherein the user input is received by the remote-control device from the user in proximity of the remote-control device and remote from the motor vehicle, wherein the user input specifies a point in a three-dimensional (3D) vehicle model of the motor vehicle, wherein the point in the 3D vehicle model corresponds to the interior location within the motor vehicle, wherein the surroundings image corresponds to the at least one part of the surroundings of the motor vehicle viewed from the interior location of the motor vehicle as the determined perspective, wherein the determined perspective is represented by using the point in the 3D vehicle model as a virtual viewpoint, wherein the surroundings image comprises the sensor data and an interior portion of the 3D vehicle model, and wherein the remote control of the motor vehicle is carried out independently of a distance between the remote-control device and the motor vehicle.