# US 2020/0401150 A1 · Autonomous transportation vehicle image augmentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401150 A1 - Document ID: US20200401150A1 - Status: Published application - Kind: A1 - Application number: 16/448,497 - Filed: 2019-06-21 - Published: 2020-12-24 - Earliest claimed: 2019-06-21 - Assignees: Volkswagen Aktiengesellschaft; Audi AG; Porsche AG - Applicants: Volkswagen AG; Audi AG; Porsche AG - Inventors: Xinwei GONG; Sai Prasad NOOKA - Web page: https://patentgrep.com/patent/US20200401150A1 ## CPC - G05D 1/0221: {involving a learning process} - B60W 60/001: {Planning or execution of driving tasks} - G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself} - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - B60W 2420/403: Image sensing, e.g. optical camera - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Devices, systems, and methods related to autonomous transportation vehicle operation may include image augmentation arrangements for training and/or evaluating autonomous operations. Such augmentations may include artificial impressions of driving conditions which can prompt recovery operations. ## Claims ### Claim 1 An autonomous transportation vehicle, comprising: a vehicle base including a chassis and a drive system, the drive system comprising a steering system for providing lateral steering adjustment of the vehicle base and a drive train coupled with the chassis to provide propulsion of the vehicle base; an image capture device coupled with the vehicle base to capture images during operation of the autonomous transportation vehicle; and means for conducting shadow mode autonomous guidance of the guidance system including generating simulated autonomous guidance of the drive system while a human driver conducts actual guidance of the drive system, for determining shadow mode autonomous guidance operations, for receiving one or more captured images from the image capture device indicating an actual course of travel of the autonomous transportation vehicle, for augmenting at least one of the one or more captured images to impose an artificial perception of a shadow mode course of travel corresponding to actualization of the shadow mode autonomous guidance of the drive system, for determining a remedial course of travel for returning to the actual course of travel from the shadow mode course of travel, and for determining at least one revision to the instructions based on the remedial course of travel to train the control system according to the artificial perception. ### Claim 2 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the means includes a control system including at least one processor for executing instructions stored on a memory for providing functionality of the means. ### Claim 3 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the augmenting of the at least one captured image includes augmenting the at least one captured image by at least one of lateral shift and yaw shift to impose the artificial perception of the shadow mode course of travel for consideration in conducting shadow mode guidance of the drive system. ### Claim 4 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the augmenting of the at least one captured image includes estimating image augmentation corresponding to actualization of the shadow mode autonomous guidance of the drive system from the perspective of the image capture device according to the at least one captured image. ### Claim 5 (depends on 4) The autonomous transportation vehicle of claim 4, wherein the augmenting of the at least one capture image includes generating augmented image data based on the estimated image augmentation. ### Claim 6 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the instructions form at least a portion of a neural network. ### Claim 7 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the remedial course of travel for returning to the actual course of travel from the shadow mode course of travel includes operation of the drive system to return to the actual course of travel while imposing no more than a threshold amount of g-force to occupants. ### Claim 8 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the means is configured to determine an amount of deviation between the shadow mode course of travel corresponding to actualization of the shadow mode autonomous guidance of the drive system and the actual course of travel conducted by the human driver. ### Claim 9 (depends on 8) The autonomous transportation vehicle of claim 8, wherein the means is configured to determine the amount of deviation based on comparison between each of the captured images and the corresponding augmented captured image, and to determine whether the amount of deviation exceeds a threshold deviation. ### Claim 10 (depends on 9) The autonomous transportation vehicle of claim 9, wherein the means is configured to record each exceedance of the threshold deviation as a disengagement. ### Claim 11 (depends on 10) The autonomous transportation vehicle of claim 10, wherein the means is configured to store at least one of the captured image and the augmented image corresponding to each disengagement. ### Claim 12 (depends on 11) The autonomous transportation vehicle of claim 11, wherein the means is configured to exclude from storage images which do not correspond to disengagement. ### Claim 13 (depends on 9) The autonomous transportation vehicle of claim 9, wherein the threshold deviation is a deviation of 2 inches between the shadow mode course of travel and the actual course of travel. ### Claim 14 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the image capture device includes a video capture device and the one or more images include video frames. ### Claim 15 (depends on 1) The autonomous transportation vehicle of claim 1, wherein the one or more captured images includes a first captured image corresponding to a first time and a second captured image corresponding to a second time, and configuration to augment at least one of the one or more captured images includes augmenting the first captured image to impose an artificial perception of the shadow mode course of travel corresponding to actualization of the shadow mode autonomous guidance of the drive system at the second time. ### Claim 16 (depends on 15) The autonomous transportation vehicle of claim 15, wherein configuration to augment at least one of the one or more captured images includes augmenting the second captured image to impose another artificial perception of the shadow mode course of travel corresponding to actualization of the shadow mode autonomous guidance of the drive system at a third time. ### Claim 17 (depends on 15) The autonomous transportation vehicle of claim 15, wherein the remedial course of travel includes a course of travel for returning to the actual course of travel from the artificial perception of the shadow mode course of travel corresponding to actualization of the shadow mode course of guidance of the drive system at the second time. ### Claim 18 A method of operating an autonomous transportation vehicle, the method comprising: operating a control system to conduct shadow mode autonomous guidance of the autonomous transportation vehicle as a simulated autonomous guidance of the autonomous transportation vehicle while a human driver conducts actual guidance of the autonomous transportation vehicle; capturing one or more images during actual guidance of the autonomous transportation vehicle indicating an actual course of travel of the autonomous transportation vehicle; augmenting at least one of the captured images to impose an artificial perception of a shadow mode course of travel of the autonomous transportation vehicle corresponding to actualization of the shadow mode autonomous guidance of the autonomous transportation vehicle; determining a remedial course of guidance for returning to the actual course of travel from the shadow mode course of travel; and developing at least one revision of instructions stored on a memory for execution by a processor to conduct shadow mode autonomous guidance of the autonomous transportation vehicle based on the remedial course of guidance to train the control system according to the artificial perception. ### Claim 19 (depends on 18) The method of claim 18, wherein augmenting the at least one captured image includes augmenting the at least one captured image by at least one of lateral shift and yaw shift to impose the artificial perception of the shadow mode course of travel for consideration in conducting shadow mode guidance operations. ### Claim 20 (depends on 18) The method of claim 18, wherein augmenting the at least one captured image includes estimating image augmentation corresponding to actualization of the shadow mode autonomous guidance operations from the perspective of the image capture device according to the at least one captured image. ### Claim 21 (depends on 20) The method of claim 20, wherein augmenting the at least one capture image includes generating augmented image data based on the estimated image augmentation. ### Claim 22 (depends on 18) The method of claim 18, wherein the instructions form at least a portion of a neural network and revising the instructions includes training the neural network. ### Claim 23 An autonomous transportation vehicle, comprising: a vehicle base including a chassis and a drive system, the drive system comprising a lateral guidance apparatus for providing steering of the vehicle base and a drive train coupled with the chassis to provide propulsion to the vehicle base; an image capture device coupled with the vehicle base to capture images during operation of the autonomous transportation vehicle; and a control system for providing autonomous guidance of the drive system, the control system including at least one processor for executing instructions stored on a memory for conducting drive system operations autonomously, wherein the control system is configured to receive one or more images from the image capture device indicating an actual course of travel of the autonomous transportation vehicle, to augment at least one of the one or more images to impose an artificial perception of an augmented course of travel of the autonomous transportation vehicle, to determine a remedial course of travel for returning to the actual course of travel from the augmented course of travel, and to determine at least revision of the instructions based on the remedial course of travel to train the control system according to the artificial perception. ### Claim 24 (depends on 23) The autonomous transportation vehicle of claim 23, wherein configuration to augment the at least one image includes augmenting the at least one image by at least one of lateral shift and yaw shift. ### Claim 25 (depends on 23) The autonomous transportation vehicle of claim 23, wherein configuration to augment the at least one image includes augmenting the at least one image to have hazardous conditions including steering to guide the autonomous transportation vehicle beyond the lane markings of a roadway on which the autonomous transportation vehicle operates as within the artificial perception. ### Claim 26 (depends on 23) The autonomous transportation vehicle of claim 23, wherein configuration to augment the at least one image includes augmenting the at least one image to have hazardous conditions including steering to guide the autonomous transportation vehicle to have yaw out of parallel with lane markings of a roadway on which the autonomous transportation vehicle operates as within the artificial perception. ### Claim 27 (depends on 23) The autonomous transportation vehicle of claim 23, wherein the instructions form at least a portion of a neural network. ### Claim 28 A method of training an autonomous transportation vehicle control system, comprising: capturing one or more images by an image capture apparatus, augmenting at least one of the captured images to impose an artificial perception, and determining at least one revision of a set of instructions for conducting autonomous operations according to the artificial perception. ### Claim 29 (depends on 28) The method of claim 28, further comprising determining a remedial course of travel for returning to an actual course of travel corresponding to the one or more images from an augmented course of travel corresponding to the augmented at least one image. ### Claim 30 (depends on 29) The method of claim 29, wherein revising includes revising the set of instructions based on the remedial course of travel to train the autonomous transportation vehicle control system according to the artificial perception. ### Claim 31 (depends on 29) The method of claim 29, wherein revising the set of instructions comprises training a neural network formed at least in part by the set of instructions.