# US 11,897,331 B2 · In-vehicle acoustic monitoring system for driver and passenger > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,897,331 B2 - Document ID: US11897331B2 - Status: Granted - Kind: B2 - Application number: 17/149,585 - Filed: 2021-01-14 - Published: 2022-07-14 - Granted: 2024-02-13 - Earliest claimed: 2021-01-14 - Assignees: Baidu USA LLC - Applicants: Baidu USA LLC - Inventors: Wei Wang; Qi Luo; Kecheng Xu; Hongyi Sun; Wesley Reynolds; Zejun Lin - Web page: https://patentgrep.com/patent/US11897331B2 ## CPC - B60K 28/02: responsive to conditions relating to the driver - B60W 60/0059: {Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity} - B60W 30/182: Selecting between different operative modes, e.g. comfort and performance modes - B60W 40/08: related to drivers or passengers - B60W 40/09: Driving style or behaviour - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60W 60/0013: {specially adapted for occupant comfort} - B60W 60/0015: {specially adapted for safety} - B60W 60/0018: {by employing degraded modes, e.g. reducing speed, in response to suboptimal conditions} - B60W 60/0051: {from occupants to vehicle} - G01S 7/539: using analysis of echo signal for target characterisation; Target signature; Target cross-section - G01S 15/32: using transmission of continuous waves, whether amplitude-, frequency-, or phase-modulated, or unmodulated - G01S 15/52: Discriminating between fixed and moving objects or between objects moving at different speeds - G01S 15/586: {using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves and based upon the Doppler effect resulting from movement of targets} - G10L 25/51: for comparison or discrimination - H04R 1/02: Casings; Cabinets {; Supports therefor;} Mountings therein (H04R1/28 takes precedence {; attachments for microphones H04R1/08; mounting of transducers in earpieces H04R1/1075}) - H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor - B60W 2040/0818: {Inactivity or incapacity of driver} - B60W 2050/0057: {Frequency analysis, spectral techniques or transforms} - B60W 2050/0071: {Controller overrides driver automatically} - B60W 2050/143: {Alarm means} - B60W 2420/54: Audio sensitive means, e.g. ultrasound - B60W 2540/01: Occupants other than the driver - B60W 2540/223: Posture, e.g. hand, foot, or seat position, turned or inclined - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - B60W 2720/10: Longitudinal speed - B60W 2720/12: Lateral speed - H04R 2499/13: Acoustic transducers and sound field adaptation in vehicles - B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Sound signals that are not audible are generated by one or more speakers disposed in the vehicle. Reflected sound signals from a driver or a passenger of the vehicle that are not audible are received by one or more microphones disposed in the vehicle. A behavior-induced acoustic pattern is detected based on the reflected ultrasound signals. The behavior-induced acoustic pattern is analyzed to recognize a behavior of the driver or the passenger of the vehicle. A response or an alert is generated according to the recognized behavior of the driver or the passenger in the vehicle. ## Claims ### Claim 1 A computer-implemented method for operating a vehicle, the method comprising: generating sound signals using one or more speakers disposed within the vehicle, wherein the sound signals are not audible; receiving reflected sound signals from a driver or a passenger of the vehicle by one or more microphones disposed within the vehicle; detecting a behavior-induced acoustic pattern based on the reflected sound signals received by the one or more microphones; analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle; and generating an alert according to the behavior of the driver or the passenger of the vehicle in view of a set of one or more rules. ### Claim 2 (depends on 1) The method of claim 1, wherein the sound signals comprise ultrasound signals. ### Claim 3 (depends on 1) The method of claim 1, wherein the one or more microphones are disposed at an upper front left corner, an upper front right corner, between a driver seat and a front passenger seat, a back of the driver seat or the front passenger seat, or between two back passenger seats in the vehicle. ### Claim 4 (depends on 1) The method of claim 1, wherein analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle comprises matching the behavior-induced acoustic pattern with a list of predetermined acoustic patterns representing a list of predetermined behaviors. ### Claim 5 (depends on 1) The method of claim 1, wherein detecting a behavior-induced acoustic pattern based on the reflected sound signals includes extracting a power spectrogram based on the reflected sound signals. ### Claim 6 (depends on 5) The method of claim 5, wherein analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle comprises comparing the power spectrogram with a predetermined power spectrogram associated with a particular behavior. ### Claim 7 (depends on 1) The method of claim 1, further comprising determining a risk level of the behavior of the driver or the passenger, and wherein the alert is further generated based on the risk level of the behavior of the driver or the passenger. ### Claim 8 (depends on 7) The method of claim 7, wherein generating the alert according to the risk level of the driver or the passenger includes, when the behavior is from the driver, generating an alarm and activating an automatic driving mode to slow down to stop at a road side in response to the determined risk level being higher than a predetermined risk threshold. ### Claim 9 (depends on 7) The method of claim 7, wherein generating the alert according to the risk level of the driver or the passenger includes, when the behavior is from the passenger and the vehicle is in an autonomous driving mode, reducing a speed of the vehicle or switching to a comfort driving mode of the autonomous driving mode. ### Claim 10 A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of operating a vehicle, the operations comprising: generating sound signals using one or more speakers disposed within the vehicle, wherein the sound signals are not audible; receiving reflected sound signals from a driver or a passenger of the vehicle by one or more microphones disposed within the vehicle; detecting a behavior-induced acoustic pattern based on the reflected sound signals received by the microphones; analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle; and generating an alert according to the behavior of the driver or the passenger of the vehicle in view of a set of one or more rules. ### Claim 11 (depends on 10) The machine-readable medium of claim 10, wherein the sound signals comprise ultrasound signals. ### Claim 12 (depends on 10) The machine-readable medium of claim 10, wherein the one or more microphones are disposed at an upper front left corner, an upper front right corner, between a driver seat and a front passenger seat, a back of the driver seat or the front passenger seat, or between two back passenger seats in the vehicle. ### Claim 13 (depends on 10) The machine-readable medium of claim 10, wherein analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle comprises matching the behavior-induced acoustic pattern with a list of predetermined acoustic patterns representing a list of predetermined behaviors. ### Claim 14 (depends on 10) The machine-readable medium of claim 10, wherein detecting a behavior-induced acoustic pattern based on the reflected sound signals includes extracting a power spectrogram based on the sound signals. ### Claim 15 (depends on 14) The machine-readable medium of claim 14, wherein analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle comprises comparing the power spectrogram with a predetermined power spectrogram associated with a particular behavior. ### Claim 16 (depends on 10) The machine-readable medium of claim 10, wherein the operations further comprise determining a risk level of the behavior of the driver or the passenger, and wherein the alert is further generated based on the risk level of the behavior of the driver or the passenger. ### Claim 17 (depends on 16) The machine-readable medium of claim 16, wherein generating the alert according to the risk level of the driver or the passenger includes, when the behavior is from the driver, generating an alarm and activating an automatic driving mode to slow down to stop at a road side in response to the risk level being higher than a predetermined risk threshold. ### Claim 18 (depends on 16) The machine-readable medium of claim 16, wherein generating the alert according to the risk level of the driver or the passenger includes, when the behavior is from the passenger and the vehicle is in an autonomous driving mode, reducing a speed of the vehicle or switching to a comfort driving mode of the autonomous driving mode. ### Claim 19 A data processing system, comprising: a processor; and a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations of operating a vehicle, the operations including generating sound signals using one or more speakers disposed within the vehicle, wherein the sound signals are not audible, receiving reflected sound signals from a driver or a passenger of the vehicle by one or more microphones disposed within the vehicle, detecting a behavior-induced acoustic pattern based on the reflected sound signals received by the microphones, analyzing the behavior-induced acoustic pattern to determine a behavior of the driver or the passenger of the vehicle, and generating an alert according to the behavior of the driver or the passenger of the vehicle in view of a set of one or more rules. ### Claim 20 (depends on 19) The system of claim 19, wherein the sound signals comprise ultrasound signals.