# US 10,481,602 B2 · Sensor fusion for autonomous driving transition control > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,481,602 B2 - Document ID: US10481602B2 - Status: Granted - Kind: B2 - Application number: 15/295,039 - Filed: 2016-10-17 - Published: 2018-04-19 - Granted: 2019-11-19 - Earliest claimed: 2016-10-17 - Assignees: Steering Solutions IP Holding Corporation - Applicants: Steering Solutions IP Holding Corporation - Inventors: Ashok Chandy - Web page: https://patentgrep.com/patent/US10481602B2 ## CPC - B60W 60/0059: {Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity} - B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle - B60W 30/10: Path keeping {(cruise control for automatically following a preceding vehicle B60W30/165)} - B60W 40/08: related to drivers or passengers - B60W 60/0053: {from vehicle to occupant} - B62D 1/286: {Systems for interrupting non-mechanical steering due to driver intervention} - B62D 5/0424: {the axes of motor and final driven element of steering gear, e.g. rack, being parallel} - B62D 6/10: characterised by means for sensing {or determining} torque - B62D 15/025: {Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation (B62D1/28 takes precedence)} - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver - B60K 2360/175: Autonomous driving - B60W 2040/0818: {Inactivity or incapacity of driver} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2510/20: Steering systems - B60W 2510/202: Steering torque - B60W 2510/205: Steering speed - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - B62D: MOTOR VEHICLES; TRAILERS (steering, or guiding on a desired track, of agricultural machines or implements A01B69/00; wheels, castors, axles, increasing wheel adhesion B60B; vehicle tyres, tyre inflation or tyre changing B60C; connections between vehicles of a train or the like B60D; vehicles for use on rail and road, amphibious or convertible vehicles B60F; suspension arrangements B60G; heating, cooling, ventilating or other air treating devices B60H; windows, windscreens, non-fixed roofs, doors or similar devices, protective coverings for vehicles not in use B60J; propulsion plant arrangements, auxiliary drives, transmissions, controls, instrumentation or dashboards B60K; electric equipment or propulsion of electrically-propelled vehicles B60L; power supply for electrically-propelled vehicles B60M; passenger accommodation not otherwise provided for B60N; adaptations for load transportation or to carry special loads or objects B60P; arrangement of signalling or lighting devices, the mounting or supporting thereof or circuits therefor, for vehicles in general B60Q; vehicles, vehicle fittings or vehicle parts not otherwise provided for B60R; servicing, cleaning, repairing, supporting, lifting, or manoeuvring, not otherwise provided for, B60S; brake arrangements, brake control systems or parts thereof B60T; air-cushion vehicles B60V; motor-cycles, accessories therefor B62J, B62K; testing of vehicles G01M) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES ## Abstract A system for sensor fusion for autonomous driving transition control includes a sensor fusion module and a decision making module. The sensor fusion module fuses a plurality of steering sensor data from one or more sensors of a steering system with a plurality of driver state data from a plurality of driver state sensors as a plurality of fused state data aligned in time. The decision making module determines whether to transition from an autonomous driving mode to a manual driving mode based on the fused state data. ## Claims ### Claim 1 A system for sensor fusion for autonomous driving transition control, the system comprising: a processor operable to receive a plurality of steering sensor data from one or more sensors of a steering system and receive a plurality of driver state data from a plurality of driver state sensors, the processor configured to: fuse the steering sensor data with the driver state data as a plurality of fused state data aligned in time; determine whether to transition from an autonomous driving mode to a manual driving mode based on the fused state data; command a steering motor actuator of the steering system autonomously in the autonomous driving mode; command the steering motor actuator responsive to a driver input in the manual driving mode; determine an attentive state of a driver based on time-averaging of the fused state data; continue monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated to determine an attentive state of a driver based on time-averaging of the fused state data; and revert from the manual driving mode to the autonomous driving mode based on determining that the attentive state of the driver no longer meets one or more transition validity criteria. ### Claim 2 (depends on 1) The system of claim 1, wherein the steering sensor data comprise sensed or derived data from one or more of a handwheel torque sensor and a handwheel position/velocity sensor. ### Claim 3 (depends on 1) The system of claim 1, wherein the driver state data comprise sensed or derived data from one or more of handwheel touch sensor and a camera. ### Claim 4 (depends on 3) The system of claim 3, wherein the driver state data sensed or derived from the handwheel touch sensor comprises one or more of: an area of contact with a handwheel, a width of contact with the handwheel, a force of contact with the handwheel, and a position of contact with the handwheel, and wherein the driver state data sensed or derived from the camera comprises one or more of: a driver body posture, a driver head pose, a driver eye gaze, and a driver hand position. ### Claim 5 (depends on 1) The system of claim 1, further comprising a recent data store of the fused state data and a history data store of the fused state data, wherein the history data store is operable to collect older values of the fused state data than collected in the recent data store. ### Claim 6 (depends on 5) The system of claim 5, wherein the processor is configured to average multiple samples of the fused state data from the history data store as time-averaged fused state data and determine whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data from the recent data store in combination with the time-averaged fused state data. ### Claim 7 (depends on 1) The system of claim 1, wherein the processor is further configured to revert from the manual driving mode to the autonomous driving mode based on determining that an input from one or more of the driver state sensors is obstructed while monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated. ### Claim 8 A steering system comprising: one or more sensors operable to produce a plurality of steering sensor data; a plurality of driver state sensors operable to produce a plurality of driver state data; a steering actuator motor; and a control module comprising a processor operable to command the steering actuator motor in an autonomous driving mode and in a manual driving mode, the control module further operable to fuse the steering sensor data with the driver state data as a plurality of fused state data aligned in time, determine whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data, determine an attentive state of a driver based on time-averaging of the fused state data, continue monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated to determine an attentive state of a driver based on time-averaging of the fused state data, and revert from the manual driving mode to the autonomous driving mode based on determining that the attentive state of the driver no longer meets one or more transition validity criteria. ### Claim 9 (depends on 8) The steering system of claim 8, wherein the steering sensor data comprise sensed or derived data from one or more of a handwheel torque sensor and a handwheel position/velocity sensor. ### Claim 10 (depends on 8) The steering system of claim 8, wherein the driver state data comprise sensed or derived data from one or more of handwheel touch sensor and a camera. ### Claim 11 (depends on 10) The steering system of claim 10, wherein the driver state data sensed or derived from the handwheel touch sensor comprises one or more of: an area of contact with a handwheel, a width of contact with the handwheel, a force of contact with the handwheel, and a position of contact with the handwheel, and wherein the driver state data sensed or derived from the camera comprises one or more of: a driver body posture, a driver head pose, a driver eye gaze, and a driver hand position. ### Claim 12 (depends on 8) The steering system of claim 8, wherein the control module comprises a recent data store of the fused state data and a history data store of the fused state data, wherein the history data store is operable to collect older values of the fused state data than collected in the recent data store. ### Claim 13 (depends on 12) The steering system of claim 12, wherein the control module is operable to average multiple samples of the fused state data from the history data store as time-averaged fused state data and determine whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data from the recent data store in combination with the time-averaged fused state data. ### Claim 14 (depends on 8) The steering system of claim 8, wherein the processor is further configured to revert from the manual driving mode to the autonomous driving mode based on determining that an input from one or more of the driver state sensors is obstructed while monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated. ### Claim 15 A method for sensor fusion for autonomous driving transition control, the method comprising: acquiring, by a processor of a control module, a plurality of steering sensor data from one or more sensors of a steering system; acquiring, by the processor of the control module, a plurality of driver state data from a plurality of driver state sensors; fusing, by the processor of the control module, the steering sensor data with the driver state data as a plurality of fused state data aligned in time; determining, by the processor of the control module, whether to transition from an autonomous driving mode to a manual driving mode based on the fused state data; commanding, by the processor of the control module, a steering motor actuator of the steering system autonomously in the autonomous driving mode; commanding, by the processor of the control module, the steering motor actuator responsive to a driver input in the manual driving mode; determining, by the processor of the control module, an attentive state of a driver based on time-averaging of the fused state data; continuing monitoring, by the processor of the control module, the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated to determine an attentive state of a driver based on time-averaging of the fused state data; and reverting, by the processor of the control module, from the manual driving mode to the autonomous driving mode based on determining that the attentive state of the driver no longer meets one or more transition validity criteria. ### Claim 16 (depends on 15) The method of claim 15, wherein the steering sensor data comprise sensed or derived data from one or more of a handwheel torque sensor and a handwheel position/velocity sensor. ### Claim 17 (depends on 15) The method of claim 15, wherein the driver state data comprise sensed or derived data from one or more of handwheel touch sensor and a camera, the driver state data sensed or derived from the handwheel touch sensor comprises one or more of: an area of contact with a handwheel, a width of contact with the handwheel, a force of contact with the handwheel, and a position of contact with the handwheel, and wherein the driver state data sensed or derived from the camera comprises one or more of: a driver body posture, a driver head pose, a driver eye gaze, and a driver hand position. ### Claim 18 (depends on 15) The method of claim 15, further comprising: storing the fused state data in a recent data store; and storing the fused state data in the history data store, wherein the history data store collects older values of the fused state data than collected in the recent data store. ### Claim 19 (depends on 18) The method of claim 18, further comprising: averaging multiple samples of the fused state data from the history data store as time-averaged fused state data; and determining whether to transition from the autonomous driving mode to the manual driving mode based on the fused state data from the recent data store in combination with the time-averaged fused state data. ### Claim 20 (depends on 15) The method of claim 15, further comprising: reverting from the manual driving mode to the autonomous driving mode based on determining that an input from one or more of the driver state sensors is obstructed while monitoring the fused state data after the transition from the autonomous driving mode to the manual driving mode has been initiated.