# US 10,769,463 B2 · Training of vehicles to improve autonomous capabilities > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,463 B2 - Document ID: US10769463B2 - Status: Granted - Kind: B2 - Application number: 16/679,363 - Filed: 2019-11-11 - Published: 2020-03-12 - Granted: 2020-09-08 - Earliest claimed: 2017-11-20 - Applicants: Ashok Krishnan - Inventors: Ashok Krishnan - Web page: https://patentgrep.com/patent/US10769463B2 ## CPC - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - A61B 3/113: for determining or recording eye movement - A61B 5/225: {of the fingers, e.g. by monitoring hand-grip force} - A61B 5/6893: {Cars} - B60W 40/09: Driving style or behaviour - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions - G06V 40/18: Eye characteristics, e.g. of the iris - G06V 40/70: Multimodal biometrics, e.g. combining information from different biometric modalities - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods of training vehicles to improve performance, reliability and autonomy of vehicles. Sensors capture a vehicle driver's eye movements, hand grip and contact area on steering wheel, distance of the accelerator and brake pedals from the foot, depression of these pedals by the foot, and ambient sounds. Data from these sensors is used as training data to improve autonomy of vehicles. Detection of events occurring outside a vehicle is made by comparing human sensor data with an associated road map to determine correlation. Signatures corresponding to human reactions and actions are extracted, which also incorporate data acquired using vehicle and outside environment sensors. By scoring driver responses to events outside the vehicle as well as during non-events, expert drivers are identified, whose responses are used as a part of vehicle training data. ## Claims ### Claim 1 A system comprising: at least one imaging device configured to acquire images of at least one eye of a human subject operating a vehicle; one or more processing units configured to process the acquired images to extract eye data from said images, the eye data corresponding to one or both of: (a) movement of the at least one eye, (b) position of the at least one eye; the vehicle having a brake pedal and an accelerator pedal controlled by a foot of the human; a sensing system configured to acquire foot data relating to one or more of: (i) distance of the foot from the brake pedal, (ii) distance of the foot from the accelerator pedal, (iii) amount of displacement of the brake pedal when depressed by the foot, (iv) amount of displacement of the accelerator pedal when depressed by the foot; the one or more processing units configured to process the eye data and foot data to extract eye events and foot events; the one or more processing units configured to make a determination that an event outside the vehicle has occurred if: (i) both an eye event as well as a foot event has occurred, and (ii) neither the eye event that has occurred nor the foot event that has occurred correlates with an associated map; wherein the data related to the event that has occurred outside the vehicle is used to train vehicles to become partially or fully autonomous or to improve their autonomous functioning. ### Claim 2 (depends on 1) The system of claim 1, wherein the vehicle has a steering wheel controlled by at least one hand of the human subject; the system further comprising a sensing system configured to acquire hand data relating to: (i) grip force of the at least one hand on the steering wheel, and (ii) contact area of the at least one hand on the steering wheel; the one or more processing units configured to process the hand data to extract hand events; the one or more processing units configured to make the determination that an event outside the vehicle has occurred if, additionally, a hand event has occurred; wherein the hand event that occurred does not correlate with said associated map. ### Claim 3 (depends on 1) The system of claim 1, further configured to: (i) score the human subject's responses to one or more events occurring outside the vehicle to obtain a total event score, and (ii) score the human subject's performance when there are no events occurring outside the vehicle to obtain a total non-event score; the scoring of each of the events and non-events at least partly based on the location of the vehicle and the associated map; the system further configured to combine the total event score and the total non-event score to obtain a combined score, and incorporate the human subject's responses into the training data if the combined score is higher than a threshold score. ### Claim 4 (depends on 1) The system of claim 1, further configured to: acquire a signature of an event outside the vehicle, the signature comprising data relating to the eye data and the foot data, the training data further comprising the signature. ### Claim 5 (depends on 4) The system of the claim 4, the signature further comprising hand data. ### Claim 6 (depends on 1) The system of claim 1, wherein: the movement of the at least one eye is associated with one or more of: (i) saccades, (ii) glissades, (iii) square-wave jerks, (iv) smooth pursuits; the position of the at least one eye is associated with one or more of: (i) fixations, (ii) tremors, (iii) drift, (iv) microsaccades. ### Claim 7 (depends on 1) The system of claim 1, wherein: the at least one imaging device is placed on one or more of: (a) a frame that is mounted on the head of the human subject, (b) dashpad of the vehicle, (c) dashboard of the vehicle; and the at least one imaging device is a standalone device or incorporated into a smartphone, and the at least one imaging device is configured to image one or both of: (a) the at least one eye, (b) the road ahead. ### Claim 8 (depends on 1) The system of claim 1, additionally comprising a device to determine the plane of focus of the at least one eye. ### Claim 9 A system comprising: at least one imaging device configured to acquire images of at least one eye of a human subject operating a vehicle; one or more processing units configured to process the acquired images to extract eye data from said images, the eye data corresponding to movement of the at least one eye or the position of the at least one eye; the vehicle having a brake pedal, an accelerator pedal, and a steering wheel; both said pedals controlled by a foot of the human subject, said steering wheel controlled by at least one hand of the human subject; the system further comprising one or both of: (a) a hand sensing system, (b) a foot sensing system; wherein the hand sensing system is configured to acquire hand data relating to: (i) grip force of the at least one hand on the steering wheel, and (ii) contact area of the at least one hand on the steering wheel; the foot sensing system is configured to acquire foot data relating to one or more of: (i) distance of the foot from the brake pedal, (ii) distance of the foot from the accelerator pedal, (iii) amount of displacement of the brake pedal when depressed by the foot, (iv) amount of displacement of the accelerator pedal when depressed by the foot; the one or more processing units configured to process: the eye data, and one or both of: (a) the hand data or (b) the foot data to extract eye events, and one or both of: (a) hand events or (b) foot events; the one or more processing units further configured to make a determination that an event outside the vehicle has occurred if: (i) an eye event, as well as one or both of: (a) a hand event or (b) a foot event has occurred, and (ii) none of the eye event, the hand event or foot event correlates with an associated map. ### Claim 10 (depends on 9) The system of claim 9, wherein the movement of the at least one eye is associated with one or more of (i) saccades, (ii) glissades, (iii) square-wave jerks, (iv) smooth pursuits; the position of the at least one eye is associated with one or more of: (i) fixations, (ii) tremors, (iii) drift, (iv) microsaccades. ### Claim 11 (depends on 9) The system of claim 9, wherein: the at least one imaging device is placed in one or more of: (i) a frame that is mounted on the head of the human subject, (ii) dashpad or dashboard of the vehicle, and the at least one imaging device is a standalone device or incorporated into a smartphone, and the at least one imaging device is configured to image one or both of: (a) the at least one eye, or (a) the road ahead. ### Claim 12 (depends on 9) The system of claim 9, additionally comprising a device to determine the plane of focus of the at least one eye. ### Claim 13 A system comprising: at least one aural sensor configured to acquire sounds inside a vehicle operated by a human subject; one or more processing units configured to process the acquired sounds to extract aural data from said sounds to form training data, the aural data corresponding to sounds produced outside the vehicle: the vehicle having a brake pedal, an accelerator pedal, and a steering wheel; both said pedals controlled by a foot of the human subject, said steering wheel controlled by at least one hand of the human subject; the system further comprising one or both of: (a) a hand sensing system, (b) a foot sensing system, wherein the hand sensing system is configured to acquire hand data relating to: (i) grip force of the at least one hand on the steering wheel, and (ii) contact area of the at least one hand on the steering wheel, and the foot sensing system is configured to acquire foot data relating to one or more of: (i) distance of the foot from the brake pedal, (ii) distance of the foot from the accelerator pedal, (iii) amount of displacement of the brake pedal when depressed by the foot, (iv) amount of displacement of the accelerator pedal when depressed by the foot; the one or more processing units configured to process the aural data and one or both of: (a) the hand data, or (b) the foot data to extract aural events and one or both of: (a) hand events, or (b) foot events; the one or more processing units configured to make a determination that an event outside the vehicle has occurred if: (i) an aural event as well as one or both of: (a) a hand event, or (b) a foot event has occurred, and (ii) none of the aural event, the hand event or foot event correlates with an associated map. ### Claim 14 (depends on 13) The system of claim 13, further configured to acquire a signature of the event outside the vehicle, the signature comprising data relating to the eye data and one or both of the hand data or foot data. ### Claim 15 (depends on 13) The system of claim 13, further configured to: (i) score the human subject's responses to one or more events occurring outside the vehicle to obtain a total event score, and (ii) score the human subject's performance when there are no events occurring outside the vehicle to obtain a total non-event score; the scoring of each of the events and non-events at least partly based on the location of the vehicle and the associated map; the system further configured to combine the total event score and the total non-event score to obtain a combined score and incorporate the human subject's responses into the training data if the combined score is higher than a threshold score. ### Claim 16 (depends on 13) The system of claim 13, wherein the at least one aural sensor is any one of a: microphone, smartphone, a set of binaural microphones; further wherein the at least one aural sensor is mounted on one or more of the following: the vehicle's ceiling, the vehicle's dashpad, the vehicle's dashboard, the vehicle's driver's seat headrest, the vehicle's driver's seat back, on a frame worn on the head of the human subject. ### Claim 17 (depends on 16) The system of claim 16, wherein the set of binaural microphones is embedded in a full mannequin head or a truncated mannequin head.