# US 12,179,790 B2 · Hybrid autonomy system for autonomous and automated delivery vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,179,790 B2 - Document ID: US12179790B2 - Status: Granted - Kind: B2 - Application number: 16/917,827 - Filed: 2020-06-30 - Published: 2021-12-30 - Granted: 2024-12-31 - Earliest claimed: 2020-06-30 - Assignees: DoorDash, Inc. - Applicants: DoorDash, Inc. - Inventors: Harrison Shih; Mishel George - Web page: https://patentgrep.com/patent/US12179790B2 ## CPC - B60W 60/001: {Planning or execution of driving tasks} - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60W 60/00256: {Delivery operations} - G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements - G05D 1/243: Means capturing signals occurring naturally from the environment, e.g. ambient optical, acoustic, gravitational or magnetic signals (using passive navigation aids external to the vehicle G05D1/244; using signals from positioning sensors located off-board the vehicle G05D1/249) - G06Q 10/0833: Tracking - B60W 2050/146: {Display means} - B60W 2554/4041: Position - B60W 2554/4044: Direction of movement, e.g. backwards - B60W 2554/4045: Intention, e.g. lane change or imminent movement - B60W 2710/20: Steering systems - G05D 2105/285: {postal packages} - G05D 2107/17: Spaces with priority for humans, e.g. populated areas, pedestrian ways, parks or beaches - G05D 2109/10: Land vehicles - G05D 2111/10: Optical signals - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Provided are various systems and processes for improving last-mile delivery of real-time, on-demand orders for perishable goods. In one aspect, a method for operating an autonomous vehicle comprises receiving real-time perception data at a planner module located on the autonomous vehicle, and generating an initial motion plan based on the real-time perception data. The autonomous vehicle is maneuvered along a constrained route based on the initial motion plan without user input. An alert condition is detected from the real-time perception data, and a notification is displayed at an operator device. The notification includes a request for user input to adjust the initial motion plan. User input is received at the planner module and a modified motion plan is generated by adjusting the initial motion plan based on the user input. A virtual representation of objects identified by the AV is locally rendered for display at the operator device. ## Claims ### Claim 1 A method, comprising: generating, at an autonomous vehicle operating in a hybrid autonomy mode, real-time perception data using a machine learning model, the real-time perception data indicating objects that have been detected in a vicinity of the autonomous vehicle; generating, at the autonomous vehicle operating in the hybrid autonomy mode, an initial motion plan based, at least in part, on the real-time perception data; maneuvering the autonomous vehicle operating in the hybrid autonomy mode along a constrained route based on the initial motion plan without user input; receiving, at the autonomous vehicle operating in the hybrid autonomy mode, input including a lateral offset and an angular offset; responsive to the input, modifying a route via which the autonomous vehicle is traveling; detecting, at the autonomous vehicle operating in the hybrid autonomy mode, an alert condition from the real-time perception data, the alert condition indicating that a classification of one of the objects is not possible or probability of certainty of identification of the one of the objects is below a threshold percentage; transmitting, by the autonomous vehicle operating in the hybrid autonomy mode to a client device, an image of the one of the objects, wherein during an alert condition, an operator can control only a subset of control axes; transmitting, by the autonomous vehicle operating in the hybrid autonomy mode to a client device, a notification corresponding to the alert condition, wherein the notification indicates a request for user input to adjust the initial motion plan; receiving the user input at the autonomous vehicle operating in the hybrid autonomy mode; generating, by the autonomous vehicle operating in the hybrid autonomy mode, a modified motion plan by adjusting the initial motion plan based on the user input; updating the machine learning model based, at least in part, on the user input; responsive to determining that a detected network latency is greater than a predefined threshold, prohibiting, by the autonomous vehicle operating in the hybrid autonomy mode, activation of a direct driving mode and continuing-arbitrating between autonomous control of the autonomous vehicle and additional user input; responsive to a disruption in network connectivity, reducing a frame rate at which video data is transmitted to the client device. ### Claim 2 (depends on 1) The method of claim 1, further comprising: converting the modified motion plan into power output signals for a steering mechanism and a motor of the autonomous vehicle; moving the steering mechanism based on the power output signals; and generating wheel motion by the motor based on the power output signals. ### Claim 3 (depends on 1) The method of claim 1, wherein the user input includes one or more of the following: longitudinal velocity adjustments, lateral offset adjustments, or an obstacle disregard selection. ### Claim 4 (depends on 1) The method of claim 1, wherein the real-time perception data is obtained from a sensor array of the autonomous vehicle. ### Claim 5 (depends on 1) The method of claim 1, wherein the notification corresponding to the alert condition includes a notification of an unidentified obstacle. ### Claim 6 (depends on 1) The method of claim 1, wherein the real-time perception data includes: video data captured by an image capture device located on the autonomous vehicle. ### Claim 7 (depends on 6) The method of claim 6, wherein the real-time perception data further includes one or more of: geolocation data, object recognition data, object proximity data, or audio data. ### Claim 8 (depends on 6) The method of claim 6, further comprising: transmitting object recognition data to the client device, the object recognition data corresponding to identified objects in the video data; locally rendering a virtual representation of the identified objects for display at the client device; tracking positions of a subset of pixels corresponding to the identified objects; transmitting changes in the positions of the subset of pixels to the client device; and displaying, at the client device, movement of the identified objects in the virtual representation based on the changes in the positions of the subset of pixels. ### Claim 9 (depends on 8) The method of claim 8, further comprising alternating from the hybrid autonomy mode to a direct driving mode, wherein the autonomous vehicle is fully controlled by user input in the direct driving mode, wherein alternating from the hybrid autonomy mode to the direct driving mode is initiated responsive to override input at the device. ### Claim 10 (depends on 9) The method of claim 9, wherein alternating from the hybrid autonomy mode to the direct driving mode is prohibited if the detected network latency is greater than the predefined threshold. ### Claim 11 (depends on 1) The method of claim 1, wherein the constrained route leads from a first aggregation depot to a second aggregation depot. ### Claim 12 A system comprising: an autonomous vehicle including one or more processors configurable to cause: generating, at the autonomous vehicle while operating in a hybrid autonomy mode, real-time perception data using a machine learning model, the real-time perception data indicating objects that have been detected in a vicinity of the autonomous vehicle; generating, at the autonomous vehicle operating in the hybrid autonomy mode, an initial motion plan based, at least in part, on the real-time perception data; maneuvering the autonomous vehicle operating in the hybrid autonomy mode along a constrained route based on the initial motion plan without user input; receiving, at the autonomous vehicle operating in the hybrid autonomy mode, input including a lateral offset and an angular offset; responsive to the input, modifying a route via which the autonomous vehicle is traveling; detecting, at the autonomous vehicle operating in the hybrid autonomy mode, an alert condition from the real-time perception data, the alert condition indicating that a classification of one of the objects is not possible or probability of certainty of identification of the one of the objects is below a threshold percentage; transmitting, by the autonomous vehicle operating in the hybrid autonomy mode to a client device, an image of the one of the objects, wherein during an alert condition, an operator can control only a subset of control axes; transmitting, by the autonomous vehicle operating in the hybrid autonomy mode to a client device, a notification corresponding to the alert condition, wherein the notification indicates a request for user input to adjust the initial motion plan; receiving the user input at the autonomous vehicle operating in the hybrid autonomy mode; generating, by the autonomous vehicle operating in the hybrid autonomy mode, a modified motion plan by adjusting the initial motion plan based on the user input; updating the machine learning model based, at least in part, on the user input; responsive to determining that a detected network latency is greater than a predefined threshold, prohibiting, by the autonomous vehicle operating in the hybrid autonomy mode, activation of a direct driving mode and continuing-arbitrating between autonomous control of the autonomous vehicle and additional user input; and responsive to a disruption in network connectivity, reducing a frame rate at which video data is transmitted to the client device. ### Claim 13 (depends on 12) The system of claim 12, the processors further configurable to cause: converting the modified motion plan into power output signals for a steering mechanism and a motor of the autonomous vehicle. ### Claim 14 (depends on 12) The system of claim 12, wherein the user input includes one or more of the following: longitudinal velocity adjustments, lateral offset adjustments, or an obstacle disregard selection. ### Claim 15 (depends on 12) The system of claim 12, wherein the notification corresponding to the alert condition includes a notification of an unidentified obstacle. ### Claim 16 (depends on 12) The system of claim 12, wherein the real-time perception data includes one or more of the following: video data captured by an image capture device of the sensory array, geolocation data, object recognition data, object proximity data, or audio data. ### Claim 17 (depends on 16) The system of claim 16, the processors further configurable to cause: identifying objects in the video data, tracking pixel changes to determine movement of objects in the video data, and transmitting the pixel changes to the operator device. ### Claim 18 (depends on 17) The system of claim 17, the processors further configurable to cause: activating a direct driving mode responsive to override input received from the client device, wherein the autonomous vehicle is fully controlled by user input in the direct driving mode. ### Claim 19 (depends on 18) The system of claim 18, wherein activating the direct driving mode is prohibited if the detected network latency is greater than the predefined threshold. ### Claim 20 One or more non-transitory computer readable media storing instructions that, when executed, cause an autonomous vehicle to execute a method, comprising: generating, at the autonomous vehicle while operating in a hybrid autonomy mode, real-time perception data using a machine learning model, the real-time perception data indicating objects that have been detected in a vicinity of the autonomous vehicle; generating, at the autonomous vehicle operating in the hybrid autonomy mode, an initial motion plan based, at least in part, on the real-time perception data; maneuvering the autonomous vehicle operating in the hybrid autonomy mode along a constrained route based on the initial motion plan without user input; receiving, at the autonomous vehicle operating in the hybrid autonomy mode, input including a lateral offset and an angular offset; responsive to the input, modifying a route via which the autonomous vehicle is traveling; detecting, at the autonomous vehicle operating in the hybrid autonomy mode, an alert condition from the real-time perception data, the alert condition indicating that a classification of one of the objects is not possible or probability of certainty of identification of the one of the objects is below a threshold percentage; transmitting, by the autonomous vehicle operating in the hybrid autonomy mode to a client device, an image of the one of the objects, wherein during an alert condition, an operator can control only a subset of control axes; transmitting, by the autonomous vehicle operating in the hybrid autonomy mode to a client device, a notification corresponding to the alert condition, wherein the notification indicates a request for user input to adjust the initial motion plan; receiving the user input at the autonomous vehicle operating in the hybrid autonomy mode; generating, by the autonomous vehicle operating in the hybrid autonomy mode, a modified motion plan by adjusting the initial motion plan based on the user input; updating the machine learning model based, at least in part, on the user input; responsive to determining that a detected network latency is greater than a predefined threshold, prohibiting, by the autonomous vehicle operating in the hybrid autonomy mode, activation of a direct driving mode and continuing-arbitrating between autonomous control of the autonomous vehicle and additional user input; and responsive to a disruption in network connectivity, reducing a frame rate at which video data is transmitted to the client device.