Remote operations of a vehicles
AssigneeNVIDIA Corporation
InventorsJen-Hsun Huang, Prajakta Gudadhe, Justin Ebert, Dane Johnston
Life of this patent
Estimating the term…
Abstract
In various examples, at least partial control of a vehicle may be transferred to a control system remote from the vehicle. Sensor data may be received from a sensor(s) of the vehicle and the sensor data may be encoded to generate encoded sensor data. The encoded sensor data may be transmitted to the control system for display on a virtual reality headset of the control system. Control data may be received by the vehicle and from the control system that may be representative of a control input(s) from the control system, and actuation by an actuation component(s) of the vehicle may be caused based on the control input.
Claims20
A method for controlling a machine, the method comprising:
- initiating a remote control session, the remote control session including a transfer of at least partial control of a machine to a control system remote from the machine;
- generating, using at least one camera of the machine, sensor data representative of a field of view of a physical environment;
- generating, using one or more sensors of the machine, state data representative of a spatial state of one or more actuation components of the machine;
- transmitting the sensor data and the state data to the remote control system to cause the remote control system to generate a virtual simulation for display in a virtual environment, wherein the virtual simulation includes a display of the field of view of the physical environment and a generated representation of the one or more actuation components at a current state based at least in part on the state data;
- receiving, from the remote control system, control data representative of at least one control input generated by a remote operator;
- and
- causing actuation by the one or more actuation components of the machine to navigate the physical environment based at least in part on the control data.
A system for controlling a machine, the system comprising:
- a machine including:
- one or more cameras configured to generate sensor data representative of a field of view of a physical environment;
- one or more actuation components;
- one or more sensors configured to generate state data representative of a spatial state of the one or more actuation components;
- and
- a communication interface;
- and
- a remote control system located remotely from the machine and including at least one processor, at least one display device, one or more remote control components, and a communication interface,
- wherein the at least one processor is configured to:
- receive, via the communication interface, the sensor data and the state data from the machine;
- generate, based at least in part on the sensor data and the state data, a virtual simulation for display in a virtual environment on the at least one display device, the virtual simulation including a display of the field of view of the physical environment and a generated representation of the one or more actuation components at a current state;
- generate control data representative of at least one control input received via the one or more remote control components during a remote control session in which at least partial control of the machine is transferred to the remote control system;
- and
- transmit the control data to the machine to cause actuation by the one or more actuation components of the machine to navigate the physical environment based at least in part on the control data.
A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a system comprising a machine and a remote control system, cause the one or more processors to perform operations comprising:
- initiating a remote control session including transferring at least partial control of the machine to the remote control system;
- receiving sensor data generated by at least one camera of the machine and representative of a field of view of a physical environment;
- receiving state data generated by one or more sensors of the machine and representative of a spatial state of one or more actuation components of the machine;
- generating, based at least in part on the sensor data and the state data, a virtual simulation for display in a virtual environment, the virtual simulation including a display of the field of view of the physical environment and a generated representation of the one or more actuation components at a current state;
- receiving, via one or more remote control components, at least one control input from a remote operator;
- generating control data representative of the at least one control input;
- and causing actuation by the one or more actuation components of the machine to navigate the physical environment based at least in part on the control data.
Dependencies are inferred from the claim text.