# US 10,871,784 B2 · Method and system for providing remote robotic control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,871,784 B2
- Document ID: US10871784B2
- Status: Granted
- Kind: B2
- Application number: 16/865,261
- Filed: 2020-05-01
- Published: 2020-08-13
- Granted: 2020-12-22
- Earliest claimed: 2018-10-10
- Assignees: Midea Group Co., Ltd.
- Applicants: Midea Group Co., Ltd.
- Inventors: Yuan Tian; Suresh Mani
- Web page: https://patentgrep.com/patent/US10871784B2

## CPC

- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- B25J 5/007: {mounted on wheels}
- B25J 9/1689: {Teleoperation}
- B25J 13/025: {comprising haptic means}
- B25J 13/085: {Force or torque sensors (B25J13/082, B25J13/084 take precedence)}
- B25J 19/023: {including video camera means}
- G01C 21/206: {specially adapted for indoor navigation}
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- G05D 1/005: {by providing the operator with signals other than visual, e.g. acoustic, haptic}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 2203/0383: Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A virtualized environment corresponding to a physical environment currently surrounding a robot is displayed. The virtualized environment is updated in accordance with streaming environment data received from sensors collocated with the robot. A first user input inserting a first virtual object at a first location in the virtualized environment is detected. The virtualized environment is modified in accordance with the insertion of the first virtual object at the first location. The first virtual object at the first location causes the robot to execute a first navigation path in the physical environment. A second user input is detected that moves the first virtual object along a movement path to a second location in the virtualized environment. The movement path is constrained by simulated surfaces in the virtualized environment, and the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment.

## Claims

### Claim 1

A method of providing mixed-initiative robotic control, including:
at a computing device having one or more processors and memory:
displaying a control user interface via a display generation component in communication with the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding a robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot, wherein the virtualized environment includes a first virtual object located at a first location within the virtualized environment, and wherein the robot executes a first navigation path in the physical environment that is generated in accordance with a first set of preprogrammed path-planning instructions and the first location of the first virtual object;
while the robot is executing the first navigation path in the physical environment, detecting a first user input, including detecting a first movement input directed to the first virtual object; and
in response to detecting the first user input:
moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, and wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions, and wherein:
the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object, or,
the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object.

### Claim 2 (depends on 1)

The method of claim 1, including:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated friction forces between the first virtual object and a simulated surface at a current location of the first virtual object.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first virtual object has a predefined size and shape, and wherein the method includes:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated reactive forces from a first simulated surface adjacent to a current location of the first virtual object to the first virtual object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first virtual object has a predefined size and shape, and wherein the method includes:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated reactive forces from a second virtual object adjacent to a current location of the first virtual object to the first virtual object.

### Claim 5

A computing device for providing mixed-initiative robotic control, including:
one or more processors; and
memory storing instructions, wherein:
the instructions, when executed by the one or more processors, cause the processors, to perform operations comprising:
displaying a control user interface via a display generation component in communication with the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding a robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot, wherein the virtualized environment includes a first virtual object located at a first location within the virtualized environment, and wherein the robot executes a first navigation path in the physical environment that is generated in accordance with a first set of preprogrammed path-planning instructions and the first location of the first virtual object;
while the robot is executing the first navigation path in the physical environment, detecting a first user input, including detecting a first movement input directed to the first virtual object; and
in response to detecting the first user input:
moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions, and wherein:
the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object, or,
the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object.

### Claim 6 (depends on 5)

The computing device of claim 5, wherein the operations include:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated friction forces between the first virtual object and a simulated surface at a current location of the first virtual object.

### Claim 7 (depends on 5)

The computing device of claim 5, wherein the first virtual object has a predefined size and shape, and wherein the method includes:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated reactive forces from a first simulated surface adjacent to a current location of the first virtual object to the first virtual object.

### Claim 8 (depends on 5)

The computing device of claim 5, wherein the first virtual object has a predefined size and shape, and wherein the operations include:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated reactive forces from a second virtual object adjacent to a current location of the first virtual object to the first virtual object.

### Claim 9

A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors of a computing device, cause the computing device to perform operations, wherein:
the operations include:
displaying a control user interface via a display generation component in communication with the computing device, including displaying a virtualized environment corresponding to a first physical environment currently surrounding the robot, wherein the virtualized environment is generated and updated in accordance with streaming environment data received from a first set of sensors collocated with the robot, wherein the virtualized environment includes a first virtual object located at a first location within the virtualized environment, and wherein the robot executes a first navigation path in the physical environment that is generated in accordance with a first set of preprogrammed path-planning instructions and the first location of the first virtual object;
while the robot is executing the first navigation path in the physical environment, detecting a first user input, including detecting a first movement input directed to the first virtual object; and
in response to detecting the first user input:
moving the first virtual object along a first movement path to a second location in the virtualized environment in accordance with the first movement input, wherein the first movement path is constrained by one or more simulated surfaces in the virtualized environment, wherein the first virtual object at the second location causes the robot to execute a modified navigation path in the physical environment that is generated in accordance with the first set of pre-programmed path-planning instructions, and wherein:
the first virtual object is marked as an obstacle and the robot avoids the first virtual object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object, or,
the first virtual object is marked as a target and the robot approaches the first labeled object during navigation on a path generated in accordance with the first set of pre-programmed instructions and the first location of the first virtual object.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the operations include:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated friction forces between the first virtual object and a simulated surface at a current location of the first virtual object.

### Claim 11 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the first virtual object has a predefined size and shape, and wherein the operations include:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated reactive forces from a first simulated surface to the first virtual object adjacent to a current location of the first virtual object.

### Claim 12 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the first virtual object has a predefined size and shape, and wherein the method includes:
during movement of the first virtual object along the first movement path to the second location in the virtualized environment, providing haptic feedback via a haptic-enabled input device in accordance with simulated reactive forces from a second virtual object to the first virtual object adjacent to a current location of the first virtual object.