# US 12,117,839 B2 · Deep learning-based autonomous vehicle control device, system including the same, and method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,117,839 B2 - Document ID: US12117839B2 - Status: Granted - Kind: B2 - Application number: 17/073,808 - Filed: 2020-10-19 - Published: 2021-02-18 - Granted: 2024-10-15 - Earliest claimed: 2017-03-27 - Assignees: Hyundai Motor Company; Kia Motors Corporation - Applicants: Hyundai Motor Company; Kia Motors Corporation - Inventors: Byung Yong You - Web page: https://patentgrep.com/patent/US12117839B2 ## CPC - B60W 60/001: {Planning or execution of driving tasks} - G05B 23/0229: {knowledge based, e.g. expert systems; genetic algorithms} - G05B 23/0294: {Optimizing process, e.g. process efficiency, product quality} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - B60W 50/0205: {Diagnosing or detecting failures; Failure detection models} - G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A deep learning-based autonomous vehicle control system includes: a processor determining an autonomous driving control based on deep learning, correcting an error in determination of the deep learning-based autonomous driving control based on determination of an autonomous driving control based on a predetermined expert rule, and controlling an autonomous vehicle; and a non-transitory computer-readable storage medium storing data for the determination of the deep learning-based autonomous driving control, data for the determination of the expert rule-based autonomous driving control, and information about the error in the determination of the deep learning-based autonomous driving control. ## Claims ### Claim 1 A deep learning-based autonomous vehicle control system, comprising: a processor configured to determine an autonomous driving control based on deep learning, to correct an error in determination of a deep learning-based autonomous driving control based on determination of an autonomous driving control based on a predetermined expert rule, and to control an autonomous vehicle; and a non-transitory computer-readable storage medium storing data for the determination of the deep learning-based autonomous driving control, data for the determination of the expert rule-based autonomous driving control, and information about the error in the determination of the deep learning-based autonomous driving control, wherein the processor is further configured to: output a deep learning-based autonomous driving control output value for the deep learning-based autonomous driving control; output an expert rule-based autonomous driving control output value based on the expert rule; and compare the deep learning-based autonomous driving control output value with the expert rule-based autonomous driving control output value and outputting a final autonomous driving control output value depending on a comparison result, wherein when the deep learning-based an autonomous driving control output value corresponds to a steering or acceleration/deceleration output control less than a predetermined minimum time to collision (TTC), the processor stops the steering or acceleration/deceleration output control less than the predetermined minimum TTC, and when the deep learning-based autonomous driving control output value corresponds to a steering value greater than a predetermined steering reference value, the processor adjusts the steering value to be less than the steering reference value. ### Claim 2 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein the processor outputs the deep learning-based autonomous driving control output value as the final autonomous driving control output value when the deep learning-based autonomous driving control output value and the expert rule-based autonomous driving control output value match. ### Claim 3 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein the processor outputs the expert rule-based autonomous driving control output value as the final autonomous driving control output value unless the deep learning-based autonomous driving control output value and the expert rule-based autonomous driving control output value match. ### Claim 4 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein the predetermined expert rule includes at least one of limitations on a steering direction with respect to a free space, a time to collision (TTC), a degree of change in steering, a degree of change in acceleration/deceleration, and a lane departure. ### Claim 5 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein when the deep learning-based autonomous driving control output value corresponds to an acceleration/deceleration value greater than a acceleration/deceleration reference value, the processor adjusts the acceleration/deceleration value to be less than the acceleration/deceleration reference value. ### Claim 6 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein when the deep learning-based autonomous driving control output value corresponds to an output value for a steering control in a direction departing from a lane, the processor stops the steering control. ### Claim 7 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein the processor further configured to control the autonomous vehicle using the final autonomous driving control output value. ### Claim 8 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein the deep learning-based autonomous driving control output value includes at least one of a relative speed between a preceding vehicle and a subject vehicle, a relative distance between the preceding vehicle and the subject vehicle, a free space on a neighboring lane, a distance to a left lane, a distance to a right lane, a lane number of a lane on which the subject vehicle is currently driving, and an angle between a lane and the subject vehicle. ### Claim 9 (depends on 1) The deep learning-based autonomous vehicle control system according to claim 1, wherein the non-transitory computer-readable storage medium comprises: a deep learning storage storing a deep learning-based output control parameter for the determination of the deep learning-based autonomous driving control; an expert rule storage storing the predetermined expert rule; and an error storage storing the information about the error that is determined and corrected by the processor. ### Claim 10 A deep learning-based autonomous vehicle control device, comprising a processor configured to: output a deep learning-based autonomous driving control output value for a deep learning-based autonomous driving control; output an expert rule-based autonomous driving control output value based on an expert rule; compare the deep learning-based autonomous driving control output value with the expert rule-based autonomous driving control output value and outputting a final autonomous driving control output value depending on a comparison result; and control an autonomous vehicle using the final autonomous driving control output value, wherein when the deep learning-based an autonomous driving control output value corresponds to a steering or acceleration/deceleration output control less than a predetermined minimum time to collision (TTC), the processor stops the steering or acceleration/deceleration output control less than the predetermined minimum TTC, and when the deep learning-based autonomous driving control output value corresponds to a steering value greater than a predetermined steering reference value, the processor adjusts the steering value to be less than the steering reference value. ### Claim 11 (depends on 10) The deep learning-based autonomous vehicle control device according to claim 10, wherein the processor outputs the deep learning-based autonomous driving control output value as the final autonomous driving control output value when the deep learning-based autonomous driving control output value and the expert rule-based autonomous driving control output value match. ### Claim 12 (depends on 11) The deep learning-based autonomous vehicle control device according to claim 11, wherein the processor outputs the expert rule-based autonomous driving control output value as the final autonomous driving control output value unless the deep learning-based autonomous driving control output value and the expert rule-based autonomous driving control output value match. ### Claim 13 A deep learning-based autonomous vehicle control method, comprising steps of: outputting, by a processor, a deep learning-based autonomous driving control output value for a deep learning-based autonomous driving control; outputting, by the processor, an expert rule-based autonomous driving control output value based on an expert rule; comparing, by the processor, the deep learning-based autonomous driving control output value with the expert rule-based autonomous driving control output value and outputting a final autonomous driving control output value depending on a comparison result; and controlling, by the processor, an autonomous vehicle using the final autonomous driving control output value, wherein the step of outputting a final autonomous driving control output value comprises: stopping, by the processor, the steering or acceleration/deceleration output control less than the predetermined minimum TTC, when the deep learning-based an autonomous driving control output value corresponds to a steering or acceleration/deceleration output control less than a predetermined minimum time to collision (TTC), and adjusting, by the processor, the steering value to be less than the steering reference value, when the deep learning-based autonomous driving control output value corresponds to a steering value greater than a predetermined steering reference value. ### Claim 14 (depends on 13) The deep learning-based autonomous vehicle control method according to claim 13, wherein the step of outputting the final autonomous driving control output value comprises outputting the deep learning-based autonomous driving control output value as the final autonomous driving control output value when the deep learning-based autonomous driving control output value and the expert rule-based autonomous driving control output value match. ### Claim 15 (depends on 14) The deep learning-based autonomous vehicle control method according to claim 14, wherein the step of outputting the final autonomous driving control output value comprises outputting the expert rule-based autonomous driving control output value as the final autonomous driving control output value unless the deep learning-based autonomous driving control output value and the expert rule-based autonomous driving control output value match. ### Claim 16 (depends on 13) The deep learning-based autonomous vehicle control method according to claim 13, wherein the expert rule includes at least one of limitations on steering direction with respect to a free space, a time to collision (TTC), a degree of change in steering, a degree of change in acceleration/deceleration, and a lane departure.