# US 2023/0065647 A1 · Emergency siren detection in autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0065647 A1
- Document ID: US20230065647A1
- Status: Published application
- Kind: A1
- Application number: 17/822,735
- Filed: 2022-08-26
- Published: 2023-03-02
- Earliest claimed: 2021-08-27
- Assignees: Tusimple, Inc.
- Applicants: Tusimple, Inc.
- Inventors: Scott Douglas FOSTER; Neil M. OVERMON; Erik Orlando PORTILLO; Zhujia SHI; Joyce TAM; Mohammad POORSARTEP; Christopher MILLER; Pengji DUAN; Lingting GE; Navid SARMADNIA; Panqu PANQU WANG; Zhe HUANG
- Web page: https://patentgrep.com/patent/US20230065647A1

## CPC

- G08G 1/0965: responding to signals from another vehicle, e.g. emergency vehicle
- B60W 60/0015: {specially adapted for safety}
- G01S 3/808: using transducers spaced apart and measuring phase or time difference between signals therefrom, i.e. path-difference systems
- G01S 11/14: using ultrasonic, sonic or infrasonic waves
- G01S 15/586: {using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves and based upon the Doppler effect resulting from movement of targets}
- G01S 15/931: of land vehicles
- G06N 3/08: Learning methods
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/584: {of vehicle lights or traffic lights}
- G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]}
- G08G 1/0133: {for classifying traffic situation}
- G08G 1/096725: {where the received information generates an automatic action on the vehicle control}
- B60W 2050/0057: {Frequency analysis, spectral techniques or transforms}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2420/54: Audio sensitive means, e.g. ultrasound
- B60W 2420/60: Doppler effect
- B60W 2554/4041: Position
- B60W 2554/802: Longitudinal distance
- G06N 3/09: Supervised learning
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An autonomous vehicle includes audio sensors configured to detect audio in an environment around the autonomous vehicle and to generate audio signals based on the detected audio. A processor in the autonomous vehicle receives the audio signals and compares a time domain or frequency domain representation of the audio signals to a corresponding representation of a known emergency vehicle siren. The comparison causes the processor to output a first determination indicating whether the audio signals are indicative of an emergency vehicle siren. The processor also applies a trained neural network to the audio signals that causes the processor to output a second determination indicating whether the audio signals are indicative of the emergency vehicle siren. If the first determination or the second determination indicates presence of an emergency vehicle siren in the environment around the autonomous vehicle, the autonomous vehicle is caused to perform an action.

## Claims

### Claim 1

An autonomous vehicle, comprising:
a plurality of audio sensors configured to detect audio in an environment around the autonomous vehicle and to generate audio signals based on the detected audio;
one or more processors; and
a non-transitory computer-readable storage medium storing executable instructions, the instructions when executed by the one or more processors causing the one or more processors to:
compare a time domain or a frequency domain representation of the audio signals to a corresponding representation of a known emergency vehicle siren, wherein the comparison outputs a first determination indicating whether the audio signals are indicative of an emergency vehicle siren in the environment around the autonomous vehicle;
apply a trained neural network to the audio signals, the trained neural network causing the one or more processors to output a second determination indicating whether the audio signals are indicative of the emergency vehicle siren in the environment around the autonomous vehicle; and
based on the first determination and the second determination each indicating the audio signals are indicative of an emergency vehicle siren in the environment around the autonomous vehicle, cause the autonomous vehicle to perform an action.

### Claim 2 (depends on 1)

The autonomous vehicle of claim 1, wherein the plurality of audio sensors comprise:
a first audio sensor on a front side of the autonomous vehicle and configured to generate a first audio signal;
a second audio sensor on a rear side of the autonomous vehicle and configured to generate a second audio signal;
a third audio sensor on a left side of the autonomous vehicle and configured to generate a third audio signal; and
a fourth audio sensor on a right side of the autonomous vehicle and configured to generate a fourth audio signal.

### Claim 3 (depends on 2)

The autonomous vehicle of claim 2, wherein the instructions when executed by the one or more processors further cause the one or more processors to:
generate, based on the third audio signal or the fourth audio signal, a representation of ambient audio in the environment of the autonomous vehicle; and
modify the first audio signal or the second audio signal based on the representation of the ambient audio;
wherein the trained neural network is applied to the modified first audio signal or the modified second audio signal.

### Claim 4 (depends on 2)

The autonomous vehicle of claim 2, wherein the instructions when executed by the one or more processors further cause the one or more processors to, in response to the first determination or the second determination indicating the audio signals are indicative of the emergency vehicle siren in the environment around the autonomous vehicle:
generate a comparison between the third audio signal and the fourth audio signal; and
detect a direction from the autonomous vehicle to an emergency vehicle producing the emergency vehicle siren based on the comparison between the third audio signal and the fourth audio signal.

### Claim 5 (depends on 1)

The autonomous vehicle of claim 1, wherein the plurality of audio sensors comprises an audio sensor array that includes a set of audio sensors distributed along a length of the audio sensor array, and wherein the instructions when executed by the one or more processors further cause the one or more processors to:
receive a first audio signal generated by a first audio sensor in the audio sensor array in response to a sound incident on the audio sensor array;
receive a second audio signal generated by a second audio sensor in the audio sensor array;
determine a phase shift angle between the first audio signal and the second audio signal;
calculate an angle of incidence of the sound on the audio sensor array based on the phase shift angle and a distance between the first audio sensor and the second audio sensor; and
in response to the second determination output by the trained neural network when applied to the first audio signal or the second audio signal indicating presence of the emergency vehicle siren in the environment around the autonomous vehicle, determine an angle between the autonomous vehicle and a source of the emergency vehicle siren based on the angle of incidence of the sound on the audio sensor array.

### Claim 6 (depends on 1)

The autonomous vehicle of claim 1, further comprising:
a camera configured to detect light signals in the environment around the autonomous vehicle;
wherein the instructions when executed by the one or more processors further cause the one or more processors to, in response to the first determination or the second determination indicating the audio signals are indicative of the emergency vehicle siren in the environment around the autonomous vehicle:
process the light signals detected by the camera to identify an emergency vehicle light pattern; and
determine a distance between the autonomous vehicle and an emergency vehicle producing the emergency vehicle siren based on a difference between an arrival time of the emergency vehicle light pattern at the camera and an arrival of the detected audio determined to indicate the emergency vehicle siren.

### Claim 7

A method comprising:
receiving, at a processor associated with a vehicle, one or more audio signals generated by respective audio sensors disposed on or in the vehicle;
comparing, by the processor, a time domain or frequency domain representation of the one or more audio signals to a corresponding representation of a known emergency vehicle siren, wherein the comparison outputs a first determination indicating whether the one or more audio signals are indicative of an emergency vehicle siren in an environment around the vehicle;
applying, by the processor, a trained neural network to the one or more audio signals, the trained neural network causing the processor to output a second determination indicating whether the audio signals are indicative of the emergency vehicle siren in the environment around the vehicle; and
based on the first determination and the second determination indicating the audio signals are indicative of an emergency vehicle siren in the environment around the vehicle, causing the vehicle to perform an action.

### Claim 8 (depends on 7)

The method of claim 7, wherein the trained neural network is trained using a set of training audio signals generated by test audio sensors in response to a plurality of different types of sirens output by different types of emergency vehicles.

### Claim 9 (depends on 8)

The method of claim 8, wherein each of the plurality of different types of sirens has a corresponding periodic siren sound pattern, and wherein the set of training audio signals include a subset of audio signals generated by the test audio sensors in response to a plurality of siren segments that represent different portions of each periodic siren sound pattern.

### Claim 10 (depends on 8)

The method of claim 8, wherein the set of training audio signals include a subset of audio signals generated by the test audio sensors in response to:
a plurality of emergency vehicle sirens produced at different volumes;
a plurality of emergency vehicle sirens produced at different distances from the test audio sensors; or
a plurality of emergency vehicle sirens produced from different angles around the test audio sensors.

### Claim 11 (depends on 7)

The method of claim 7, wherein comparing the time domain or frequency domain representation of the one or more audio signals to the corresponding representation of a known emergency vehicle siren comprises:
computing a frequency spectrum of the one or more audio signals; and
comparing the computed frequency spectrum to one or more frequency spectra of known sirens stored in a data repository available to the processor.

### Claim 12 (depends on 7)

The method of claim 7, wherein comparing the time domain or frequency domain representation of the one or more audio signals to the corresponding representation of a known emergency vehicle siren comprises:
performing a curve fit on a time domain representation of the one or more audio signals; and
comparing features of the curve fit to corresponding curve fit features of known sirens stored in a data repository available to the processor.

### Claim 13 (depends on 7)

The method of claim 7, further comprising, in response to the first determination or the second determination indicating the audio signals are indicative of an emergency vehicle siren in the environment around the vehicle:
detecting a distance between the vehicle and a source of the emergency vehicle siren;
wherein the vehicle is caused to perform the action when the detected distance is less than a threshold distance.

### Claim 14 (depends on 7)

The method of claim 7, further comprising, in response to the first determination or the second determination indicating the audio signals are indicative of an emergency vehicle siren in the environment around the vehicle:
detecting a Doppler effect-based change to the emergency vehicle siren; and
determining whether a source of the emergency vehicle siren is moving towards or away from the vehicle;
wherein the vehicle is caused to perform the action when the source of the emergency vehicle siren is determined to be moving towards the vehicle.

### Claim 15 (depends on 7)

The method of claim 7, wherein causing the vehicle to perform the action comprises causing the vehicle to navigate out of a path of an emergency vehicle producing the emergency vehicle siren.

### Claim 16

A non-transitory computer-readable storage medium storing executable instructions, the instructions when executed by one or more processors associated with a vehicle causing the one or more processors to:
receive one or more audio signals generated by respective audio sensors disposed on or in the vehicle;
compare a time domain or frequency domain representation of the one or more audio signals to a corresponding representation of a known emergency vehicle siren, wherein the comparison outputs a first determination indicating whether the one or more audio signals are indicative of an emergency vehicle siren in an environment around the vehicle;
apply a trained neural network to the one or more audio signals, the trained neural network causing the processor to output a second determination indicating whether the audio signals are indicative of the emergency vehicle siren in the environment around the vehicle; and
based on a weighted sum of the first determination and the second determination, cause the vehicle to perform an action.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed by the one or more processors further cause the one or more processors to:
receive a first audio signal generated by an audio sensor disposed at a front or a rear of the vehicle;
receive a second audio signal generated by an audio sensor disposed at a left side or a right side of the vehicle;
generate, based on the second audio signal, a representation of ambient audio in the environment of the vehicle; and
modify the first audio signal based on the representation of the ambient audio;
wherein the trained neural network is applied to the modified first audio signal.

### Claim 18 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed by the one or more processors further cause the one or more processors to:
receive a first audio signal generated by an audio sensor disposed at a left side of the vehicle;
receive a second audio signal generated by an audio sensor disposed at a right side of the vehicle;
generate a comparison between the first audio signal and the second audio signal; and
detect a direction from the vehicle to an emergency vehicle producing the emergency vehicle siren based on the comparison between the first audio signal and the second audio signal.

### Claim 19 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the trained neural network is trained using a set of training audio signals generated by test audio sensors in response to a plurality of different types of sirens output by different types of emergency vehicles, the set of training audio signals including:
a subset of audio signals generated by the test audio sensors in response to a plurality of siren segments that represent different portions of periodic siren sound patterns;
a subset of audio signals generated by the test audio sensors in response to a plurality of emergency vehicle sirens produced at different volumes;
a subset of audio signals generated by the test audio sensors in response to a plurality of emergency vehicle sirens produced at different distances from the test audio sensors; or
a subset of audio signals generated by the test audio sensors in response to a plurality of emergency vehicle sirens produced from different angles around the test audio sensors.

### Claim 20 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed by the one or more processors further cause the one or more processors to, in response to the first determination or the second determination indicating the audio signals are indicative of an emergency vehicle siren in the environment around the vehicle:
detecting a distance between the vehicle and an emergency vehicle producing the emergency vehicle siren;
wherein causing the vehicle to perform the action comprises causing the vehicle to navigate out of a path of the emergency vehicle when the detected distance is less than a threshold distance.
