# US 11,299,169 B2 · Vehicle neural network training > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,299,169 B2 - Document ID: US11299169B2 - Status: Granted - Kind: B2 - Application number: 16/751,842 - Filed: 2020-01-24 - Published: 2021-07-29 - Granted: 2022-04-12 - Earliest claimed: 2020-01-24 - Assignees: Ford Global Technologies, LLC - Applicants: Ford Global Technologies, LLC - Inventors: Punarjay Chakravarty; Ashley Elizabeth Micks - Web page: https://patentgrep.com/patent/US11299169B2 ## CPC - H04W 4/44: for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H] - B60W 10/04: including control of propulsion units - B60W 10/18: including control of braking systems - B60W 10/20: including control of steering systems - B60W 50/00: Details of control systems for road vehicle drive control not related to the control of a particular sub-unit {, e.g. process diagnostic or vehicle driver interfaces} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/74: {involving reference images or patches} - G06T 7/75: {involving models} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04W 4/023: {using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds} - B60W 60/00: Drive control systems specially adapted for autonomous road vehicles - B60W 2050/0075: {Automatic parameter input, automatic initialising or calibrating means} - B60W 2050/0083: {Setting, resetting, calibration} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2520/14: Yaw - B60W 2520/16: Pitch - B60W 2520/18: Roll - B60W 2556/45: External transmission of data to or from the vehicle - G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20104: Interactive definition of region of interest [ROI] - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES ## Abstract A computer, including a processor and a memory, the memory including instructions to be executed by the processor to determine six degree of freedom (DoF) data for a first object in a first video image and generate a synthetic video image corresponding to the first video image including a synthetic object and a synthetic object label based on the six DoF data. The instructions can include further instructions to train a generative adversarial network (GAN) based on a paired first video image and a synthetic video image to generate a modified synthetic image and train a deep neural network to locate the synthetic object in the modified synthetic video image based on the synthetic object. The instructions can include further instructions to download the trained deep neural network to a computing device in a vehicle. ## Claims ### Claim 1 A computer, comprising a processor; and a memory, the memory including instructions executable by the processor to: determine six degree of freedom (DoF) data for a first object in a first video image; generate a synthetic video image corresponding to the first video image including a synthetic object and a synthetic object label based on the six DoF data; train a generative adversarial network (GAN) based on a paired first video image and a synthetic video image to generate a modified synthetic image; train a deep neural network to locate the synthetic object in the modified synthetic video image based on the synthetic object label; and download the trained deep neural network to a computing device in a vehicle. ### Claim 2 (depends on 1) The computer of claim 1, wherein the synthetic video image is generated by photorealistic rendering to correspond with the first video image including rendering the synthetic object to correspond with an identity and location of the first object in the first video image based on the six DoF data. ### Claim 3 (depends on 1) The computer of claim 1, wherein the six DoF data includes a location in three-dimensional (3-D) space and an orientation in 3-D space, both determined with respect to orthogonal three-dimensional axes, wherein the orthogonal three-dimensional axes are determined with respect to a global coordinate system. ### Claim 4 (depends on 3) The computer of claim 3, wherein the global coordinate system is based on latitude, longitude, and altitude, location is measured in x, y, and z coordinates and orientation is measured by roll, pitch, and yaw specified with respect to the global coordinate system. ### Claim 5 (depends on 1) The computer of claim 1, the instructions including further instructions to determine the six DoF data by processing the first video image using machine vision techniques to determine correspondence between locations of first data points on the first object in the first video image, and locations of measured data points determined by measuring locations of data points on one or more of the first object or engineering drawings of the first object. ### Claim 6 (depends on 5) The computer of claim 5, the instructions including further instructions to determine the six DoF data by minimizing differences between locations of the first data points and the measured data points by non-linear curve fitting. ### Claim 7 (depends on 6) The computer of claim 6, the instructions including further instructions to determine the six DoF data based on engineering drawings of the first object and virtual camera point of view. ### Claim 8 (depends on 7) The computer of claim 7, wherein the virtual camera point of view includes six DoF data for an optical axis of the virtual camera and magnification data for the virtual camera. ### Claim 9 (depends on 1) The computer of claim 1, wherein the GAN includes a generative network and an adversarial network. ### Claim 10 (depends on 1) The computer of claim 1, wherein the GAN adds image feature data from the first video image to the synthetic video image to generate the modified synthetic video image. ### Claim 11 (depends on 1) The computer of claim 1, wherein the deep neural network includes convolutional layers and fully-connected layers. ### Claim 12 (depends on 11) The computer of claim 11, the instructions including further instructions to train the deep neural network to locate the first object by comparing back propagated output states with the synthetic object label. ### Claim 13 (depends on 1) The computer of claim 1, the instructions including further instructions to operate the vehicle by processing acquired video data with the downloaded deep neural network to locate one or more objects in the acquired video data. ### Claim 14 (depends on 13) The computer of claim 13, the instructions including further instructions to operate the vehicle by controlling vehicle powertrain, steering and brakes based on locations of the one or more objects in the acquired video data. ### Claim 15 A method, comprising: determining six degree of freedom (DoF) data for a first object in a first video image; generating a synthetic video image corresponding to the first video image including a synthetic object and a synthetic object label based on the six DoF data; training a generative adversarial network (GAN) based on a paired first video image and a synthetic video image to generate a modified synthetic image; training a deep neural network to locate an object in the modified synthetic image based on the synthetic object label; and download the trained deep neural network to a computing device in a vehicle. ### Claim 16 (depends on 15) The method of claim 15, wherein the synthetic video image is generated by a photorealistic rendering process to correspond with the first video image including rendering the synthetic object to correspond with a location of the first object in the first video image based on the six DoF data. ### Claim 17 (depends on 15) The method of claim 15, wherein the six DoF data includes a location in three-dimensional (3-D) space and an orientation in 3-D space, both determined with respect to orthogonal three-dimensional axes, wherein the orthogonal three-dimensional axes are determined with respect to a global coordinate system. ### Claim 18 (depends on 17) The method of claim 17, wherein the global coordinate system is based on latitude, longitude, and altitude, location is measured in x, y, and z coordinates and orientation is measured by roll, pitch, and yaw specified with respect to the global coordinate system. ### Claim 19 (depends on 15) The method of claim 15, further comprising determining the six DoF data by processing the first video image using machine vision techniques to determine correspondence between locations of first data points on the first object in the first video image and locations of measured data points determined by measuring locations of data points on one or more of the first object or engineering drawings of the first object. ### Claim 20 (depends on 19) The method of claim 19, wherein the instructions including further instructions to determine six DoF data by minimizing differences between locations of the first data points and the measured data points by non-linear curve fitting.