Augmented reality control for machine
AssigneeDeere & Company
InventorMark J. Cherney
Life of this patent
Estimating the term…
Abstract
Systems and methods are described for controlling an operation of machinery using augmented reality or virtual reality. A controller is configured to detect an object visible via a display based at least in part on sensor data received from an object sensor. The controller then generates and outputs on the display a graphical element corresponding to the detected object. The controller then monitors for interactions between an operator's hand and the graphical element and transmits control signals to a machinery actuator to control an operation of a tool of the machinery based at least in part on one or more detected interactions between the operator's hand and the graphical element.
Claims15
A system for operating machinery, the system comprising:
- a display;
- an object sensor;
- and
- a controller configured to
- detect an object visible via the display based at least in part on sensor data from the object sensor, wherein the detected object includes an operable tool of the machinery,
- generate a graphical element corresponding to the detected object, wherein the graphical element includes a virtual handle, wherein the virtual handle has an appearance of a hand-manipulable control and is not a graphical representation of any part of the operable tool,
- output the graphical element on the display,
- detect an interaction of an operator's hand with the virtual handle, wherein the operator's hand is visible via the display and is detected based at least in part on the sensor data from the object sensor, and wherein detecting the interaction of the operator's hand with the virtual handle includes detecting the operator's hand positioned at a location corresponding to the virtual handle and moving while contacting the virtual handle, and
- transmit a control signal to an actuator corresponding to the operable tool, wherein the transmitted control signal is configured to cause the actuator to adjust a position of the operable tool in a direction corresponding to the detected movement of the operator's hand while contacting the virtual handle.
A system for operating machinery, the system comprising:
- a display;
- an object sensor;
- and
- a controller configured to
- detect an object visible via the display based at least in part on sensor data from the object sensor, wherein the detected object includes an operable tool of the machinery,
- generate a graphical element highlighting a location of the operable tool visible via the display, and
- output the graphical element on the display,
- detect an interaction of an operator's hand with the graphical element, wherein the operator's hand is detected based at least in part on the sensor data from the object sensor, and wherein detecting the interaction of the operator's hand with the graphical element includes detecting a tapping motion towards a location on the operable tool visible via the display, and
- transmit a control signal to an actuator corresponding to the operable tool, wherein the transmitted control signal is configured to cause the actuator to adjust a position of the operable tool in a direction corresponding to at least one selected from a group consisting of the location of the detected tapping motion and a direction of the detected tapping motion.
A system for adjusting a position of a blade of a motor grader using augmented reality, the system comprising:
- a headset including a display and an object sensor,
- wherein the display includes an at least partially transparent graphical display screen configured to output graphical elements in a field of view of an operator wearing the headset while real world objects are visible to the operator through the at least partially transparent graphical display screen, and
- wherein the object sensor is positioned on the headset with a field of view that at least partially overlaps with the field of view of the operator wearing the headset;
- and
- a controller configured to
- detect a location of the blade of the motor grader in the field of view of the operator based at least in part on sensor data received from the object sensor,
- generate and output on the display a virtual handle for the blade of the motor grader,
- detect a movement of the operator's hand while contacting the virtual handle based at least in part on sensor data from the object sensor, and
- transmit a control signal to a motor grader actuator, wherein the control signal is configured to cause the motor grader actuator to adjust a position of the motor grader blade in a direction and a magnitude corresponding to the detected movement of the operator's hand while contacting the virtual handle.
Dependencies are inferred from the claim text.