# US 11,440,554 B2 · Method and system for determining driver emotions in conjuction with driving environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,440,554 B2 - Document ID: US11440554B2 - Status: Granted - Kind: B2 - Application number: 17/084,041 - Filed: 2020-10-29 - Published: 2022-02-03 - Granted: 2022-09-13 - Earliest claimed: 2020-07-29 - Assignees: Hyundai Motor Company; Kia Motors Corporation - Applicants: Hyundai Motor Company; Kia Motors Corporation - Inventors: Jin Mo Lee; Young Bin Min - Web page: https://patentgrep.com/patent/US11440554B2 ## CPC - B60W 40/08: related to drivers or passengers - B60W 40/02: related to ambient conditions - G10L 25/63: for estimating an emotional state - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2540/21: Voice - B60W 2540/22: Psychological state; Stress level or workload - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method of determining driver emotions in conjunction with a driving environment includes determining a basic emotion of a driver, acquiring driving environment information, determining the driver emotion based on the driving environment information and the basic emotion, and providing a service of displaying the driver emotion or a driver-emotion-based service. ## Claims ### Claim 1 A method of determining driver emotions in conjunction with a driving environment, the method comprising: determining a basic emotion of a driver; acquiring driving environment information; determining the driver emotion based on the driving environment information and the basic emotion; and providing a service of displaying the driver emotion or a driver-emotion-based service, wherein the determining the driver emotion comprises: simultaneously receiving the basic emotion and the driving environment information, which are generated in real time, and recognizing whether the driver emotion occurs, and matching the basic emotion and the driving environment information with each other according to a preset matching table and determining whether a preset driver emotion occurs. ### Claim 2 (depends on 1) The method of claim 1, wherein the driving environment information comprises at least one of a vehicle speed, an acceleration, global positioning system (GPS) information, passenger riding information, voice utterance information, or information device manipulation information. ### Claim 3 (depends on 1) The method of claim 1, further comprising: determining positive and negative levels of the basic emotion. ### Claim 4 (depends on 3) The method of claim 3, wherein the determining the driver emotion when the basic emotion is a positive emotion, determining that the basic emotion is the driver emotion while traveling; and when the basic emotion is a negative emotion, classifying the basic emotion into a subdivided negative emotion based on the negative emotion and the driving environment information. ### Claim 5 (depends on 1) The method of claim 1, further comprising: detecting a driving environment event; and when the detected driving environment event indicates an emotion-associated event, activating a basic emotion recognition system. ### Claim 6 (depends on 5) The method of claim 5, further comprising: after the basic emotion recognition system is activated, determining whether the basic emotion occurs within a threshold time. ### Claim 7 (depends on 6) The method of claim 6, further comprising: when the basic emotion does not occur within the threshold time, deactivating the basic emotion recognition system; and when the basic emotion occurs within the threshold time, determining the driver emotion based on the driving environment information and the basic emotion. ### Claim 8 (depends on 5) The method of claim 5, wherein, when the driving environment event is detected, the activating the basic emotion recognition system comprises classifying a type of the driving environment event into an event that induces a change in emotion and an event that takes time to induce a change in emotion and changing a time of activating the emotion recognition system. ### Claim 9 (depends on 1) A non-transitory computer-readable recording medium having recorded thereon a program for executing the method of claim 1. ### Claim 10 A system for determining driver emotions in conjunction with a driving environment, the system comprising: a camera; a basic emotion recognition system configured to classify a basic emotion of a driver using face recognition information of the camera; a driving environment information acquirer configured to acquire driving environment information; and a driver emotion classification system configured to classify the basic emotion of the driver using the driving information and a result of recognition of the basic emotion and to classify a driver emotion, wherein the system for determining driver emotions is configured to provide a service of displaying the driver emotion or a driver-emotion-based service, wherein the driver emotion classification system simultaneously receives the basic emotion and the driving environment information, which are generated in real time, and recognizes whether the driver emotion occurs, and matches the basic emotion and the driving environment information with each other according to a preset matching table and determines that a preset driver emotion occurs. ### Claim 11 (depends on 10) The system of claim 10, wherein the driving environment information comprises at least one of a vehicle speed, an acceleration, global positioning system (GPS) information, passenger riding information, voice utterance information, or information device manipulation information. ### Claim 12 (depends on 10) The system of claim 10, wherein the driver emotion classification system determines positive and negative levels of the basic emotion. ### Claim 13 (depends on 12) The system of claim 12, wherein, when the basic emotion is a positive emotion, the driver emotion classification system determines that the basic emotion is the driver emotion while traveling, and when the basic emotion is a negative emotion, the driver emotion classification system classifies the basic emotion into a subdivided negative emotion based on the negative emotion and the driving environment information. ### Claim 14 (depends on 10) The system of claim 10, wherein, when a driving environment event is detected, and when the detected driving environment event indicates an emotion-associated event, the basic emotion recognition system is activated. ### Claim 15 (depends on 14) The system of claim 14, wherein, after the basic emotion recognition system is activated, the basic emotion recognition system determines whether basic emotion occurs within a threshold time. ### Claim 16 (depends on 15) The system of claim 15, wherein, when the basic emotion does not occur within the threshold time, the basic emotion recognition system is deactivated, and when the basic emotion occurs within the threshold time, the driver emotion classification system determines the driver emotion based on the driving environment information and the basic emotion. ### Claim 17 (depends on 14) The system of claim 14, wherein, when the driving environment event is detected, if the basic emotion recognition system is activated, the system for determining driver emotions classifies a type of the driving environment event into an event that induces a change in emotion and an event that takes time to induce a change in emotion, and changes a time of activating the emotion recognition system. ### Claim 18 A method of determining driver emotions in conjunction with a driving environment, the method comprising: determining a basic emotion of a driver; acquiring driving environment information; determining the driver emotion based on the driving environment information and the basic emotion; providing a service of displaying the driver emotion or a driver-emotion-based service; detecting a driving environment event; and when the detected driving environment event indicates an emotion-associated event, activating a basic emotion recognition system. ### Claim 19 (depends on 18) The method of claim 18, further comprising: after the basic emotion recognition system is activated, determining whether the basic emotion occurs within a threshold time. ### Claim 20 (depends on 19) The method of claim 19, further comprising: when the basic emotion does not occur within the threshold time, deactivating the basic emotion recognition system; and when the basic emotion occurs within the threshold time, determining the driver emotion based on the driving environment information and the basic emotion. ### Claim 21 (depends on 18) The method of claim 18, wherein, when the driving environment event is detected, the activating the basic emotion recognition system comprises classifying a type of the driving environment event into an event that induces a change in emotion and an event that takes time to induce a change in emotion and changing a time of activating the emotion recognition system. ### Claim 22 A system for determining driver emotions in conjunction with a driving environment, the system comprising: a camera; a basic emotion recognition system configured to classify a basic emotion of a driver using face recognition information of the camera; a driving environment information acquirer configured to acquire driving environment information; and a driver emotion classification system configured to classify the basic emotion of the driver using the driving information and a result of recognition of the basic emotion and to classify a driver emotion, wherein the system for determining driver emotions is configured to provide a service of displaying the driver emotion or a driver-emotion-based service, wherein, when a driving environment event is detected, and when the detected driving environment event indicates an emotion-associated event, the basic emotion recognition system is activated. ### Claim 23 (depends on 22) The system of claim 22, wherein, after the basic emotion recognition system is activated, the basic emotion recognition system determines whether basic emotion occurs within a threshold time. ### Claim 24 (depends on 23) The system of claim 23, wherein, when the basic emotion does not occur within the threshold time, the basic emotion recognition system is deactivated, and when the basic emotion occurs within the threshold time, the driver emotion classification system determines the driver emotion based on the driving environment information and the basic emotion. ### Claim 25 (depends on 22) The system of claim 22, wherein, when the driving environment event is detected, if the basic emotion recognition system is activated, the system for determining driver emotions classifies a type of the driving environment event into an event that induces a change in emotion and an event that takes time to induce a change in emotion, and changes a time of activating the emotion recognition system.