# US 2023/0391344 A1 · In-vehicle multimodal violence detection for shared vehicles and self-driving shuttles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0391344 A1
- Document ID: US20230391344A1
- Status: Published application
- Kind: A1
- Application number: 17/829,643
- Filed: 2022-06-01
- Published: 2023-12-07
- Earliest claimed: 2022-06-01
- Assignees: GM Global Technology Operations LLC
- Applicants: GM Global Technology Operations LLC
- Inventors: Alaa M. Khamis; Wei Tong; Sai Vishnu Aluru; Upali P. Mudalige; Sravan Daruri
- Web page: https://patentgrep.com/patent/US20230391344A1

## CPC

- B60W 40/08: related to drivers or passengers
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- A61B 5/18: for vehicle drivers {or machine operators}
- B60W 30/18109: {Braking}
- G06V 10/12: Details of acquisition arrangements; Constructional details thereof
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 40/70: Multimodal biometrics, e.g. combining information from different biometric modalities
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/63: for estimating an emotional state
- A61B 5/02: Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
- A61B 5/0205: Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/374: Detecting the frequency distribution of signals, e.g. detecting delta, theta, alpha, beta or gamma waves
- A61B 5/6893: {Cars}
- A61B 5/746: {Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms}
- A61B 2503/22: Motor vehicles operators, e.g. drivers, pilots, captains
- B60W 2040/089: {Driver voice}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/51: for comparison or discrimination
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An in-vehicle violence detection system includes a speech and non-speech audio recognition module capturing occupant threat words and non-speech audio events. In-vehicle accelerometers generate data analyzed in a shaking movement recognition module. A heart rate and breathing rate detection module measures physiological changes in occupant heart rates and breathing rhythms. An in-vehicle semantic scene recognition module analyzes occupant non-verbal interactions. Occupant threat indicators including an audible threat indicator are generated by the speech and non-speech audio event recognition module. Visual threat indicators are generated by the in-vehicle semantic scene recognition module. Physiological threat indicators are generated by the heart rate and breathing rate detection module. Vibration-based threat indicators are generated by the shaking movement recognition module. The four threat indicators are fused to estimate a threat level, consolidated by incorporating contextual information including a vehicle location and time. Different actions are taken based on the consolidated threat level.

## Claims

### Claim 1

An in-vehicle multimodal violence detection system, comprising:
a speech and non-speech audio event recognition module capturing threat words and non-speech audio events of occupants of an automobile vehicle;
multiple in-vehicle accelerometers generating in-vehicle accelerometer data analyzed in a shaking movement recognition module;
a heart rate and breathing rate detection module measuring physiological changes in heart rates and breathing rhythms of the occupants;
an in-vehicle semantic scene recognition module capturing and analyzing non-verbal interactions between the occupants; and
one or more occupant threat indicators including an audible threat indicator generated by the speech and non-speech audio event recognition module, a visual threat indicator generated by the in-vehicle semantic scene recognition module, a physiological threat indicator generated by the heart rate and breathing rate detection module and a vibration-based threat indicator generated by the shaking movement recognition module.

### Claim 2 (depends on 1)

The in-vehicle multimodal violence detection system of claim 1, including a priori knowledge defining a history record of violence applied to determine a threat level.

### Claim 3 (depends on 2)

The in-vehicle multimodal violence detection system of claim 2, including an outside source providing the a priori information.

### Claim 4 (depends on 2)

The in-vehicle multimodal violence detection system of claim 2, wherein the a priori information includes an area of travel wherein known violence has occurred and a past history record of different “normal” versus “threat” events.

### Claim 5 (depends on 1)

The in-vehicle multimodal violence detection system of claim 1, including a picture compilation module identifying vectors combining an output signal from the speech and non-speech audio event recognition module, the shaking movement recognition module, the heart rate and breathing rate detection module and the in-vehicle semantic scene recognition module and assigning predetermined thresholds distinguishing between a normal event and a threat event.

### Claim 6 (depends on 5)

The in-vehicle multimodal violence detection system of claim 5, including a threat evaluation module analyzing the threat event using audible, visual, physiological and vibration-based indicators, and an a priori knowledge including a history record of violence and contextual information having a location and a time of day to determine a threat level.

### Claim 7 (depends on 6)

The in-vehicle multimodal violence detection system of claim 6, including a threat assessment unit receiving the threat level and confirming if an active threat is present and generating an active threat signal.

### Claim 8 (depends on 7)

The in-vehicle multimodal violence detection system of claim 7, wherein after generating the active threat signal a threat confirmation request is generated and forwarded to a threat timer unit.

### Claim 9 (depends on 8)

The in-vehicle multimodal violence detection system of claim 8, including:
a confirmation request visually and audibly presented to the occupants and a predetermined time interval allowed for at least one of the occupants to confirm if the active threat is present in the automobile vehicle is set; and
a threat level determination unit initiated by a time interval time-out signal if the predetermined time interval is exceeded prior to receiving a response from any one of the occupants.

### Claim 10 (depends on 9)

The in-vehicle multimodal violence detection system of claim 9, including:
an active threat signal generated and forwarded to the threat level determination unit if any one of the occupants confirm that an active threat is present;
the threat level determination unit identifying if the active threat should be categorized as one of a “low threat”, a “medium threat” or a “high threat”, wherein:
identification of the “low threat” activates a vehicle horn and a vehicle warning flasher;
identification of the “medium threat” activates the vehicle horn and the vehicle warning flasher and activates a vehicle brake assist system and notification of the medium threat to an outside source; and
identification of the “high threat” activates the vehicle horn and the vehicle warning flasher, activates the vehicle brake assist system and forwards a request to the outside source to initiate an emergency assistance request.

### Claim 11

A method to perform in-vehicle multimodal violence detection, comprising:
capturing threat words and non-speech audio events of occupants of an automobile vehicle using a speech and non-speech audio event recognition module;
analyzing in-vehicle accelerometer data in a shaking movement recognition module;
identifying sudden physiological changes in heart rates and breathing rhythms of the occupants in a heart rate and breathing rate detection module;
capturing and analyzing non-verbal interactions between the occupants using an in-vehicle semantic scene recognition module; and
generating one or more occupant threat indicators including audible, visual, physiological and vibration-based indicators.

### Claim 12 (depends on 11)

The method of claim 11, further including applying a priori knowledge including a history record of violence to determine a threat level.

### Claim 13 (depends on 12)

The method of claim 12, further including applying contextual information including a vehicle location and a time of day from the a priori knowledge.

### Claim 14 (depends on 11)

The method of claim 11, further including incorporating ride information and occupant information from a booking database and analyzing the ride information and the occupant information to distinguish differences between the occupants.

### Claim 15 (depends on 11)

The method of claim 11, further including identifying shaking movements of the occupants of the automobile vehicle using the in-vehicle accelerometer data.

### Claim 16 (depends on 11)

The method of claim 11, further including applying inarticulate sounds and sounds accompanying threat or abnormal behaviors including shouting, screaming, crying and glass breaking as audio-based threat indicators.

### Claim 17 (depends on 11)

The method of claim 11, further including analyzing the one or more occupant threat indicators in a multimodal threat evaluation module.

### Claim 18

A method to perform in-vehicle multimodal violence detection, comprising:
capturing threat words and non-speech audio events of occupants of an autonomous automobile vehicle using a speech and non-speech audio event recognition module;
analyzing in-vehicle accelerometer data in a shaking movement recognition module;
measuring sudden physiological changes in heart rates and breathing rhythms of the occupants in a heart rate and breathing rate detection module;
capturing and analyzing non-verbal interactions between the occupants using an in-vehicle semantic scene recognition module;
generating one or more occupant threat indicators including audible, visual, physiological and vibration-based indicators; and
identifying vectors combining an output signal from the speech and non-speech audio event recognition module, the shaking movement recognition module, the heart rate and breathing rate detection module and the in-vehicle semantic scene recognition module and assigning predetermined thresholds distinguishing between a normal event and a threat event in a picture compilation module.

### Claim 19 (depends on 18)

The method of claim 18, further including:
collecting output signals generated by multiple in-vehicle cameras positioned in the autonomous automobile vehicle;
identifying if any one of the multiple in-vehicle cameras is covered in a camera assessment unit; and
generating an audio pulse signal by the camera assessment unit to notify the occupants of the autonomous automobile vehicle to uncover a covered one of the any one of the multiple in-vehicle cameras.

### Claim 20 (depends on 18)

The method of claim 18, further including:
identifying by receipt of an occupant status signal forwarded to an occupant confirmation unit if two or more occupants are present in the autonomous automobile vehicle;
actuating operation of a ride type identification unit, wherein:
if a “hailing ride type” signal is generated by the ride type identification unit a first program assumption is made that all of the occupants present in the autonomous automobile vehicle are known to each other and that no threats will later occur; or
if a “sharing ride type” signal is generated by the ride type identification unit a second program assumption is made that at least one of the occupants present in the autonomous automobile vehicle is not previously known to other ones of the occupants and that a potential therefore exists for the threat event to develop during subsequent operation of the autonomous automobile vehicle.
