Detecting at least one emergency vehicle using a perception algorithm

A perception system for a vehicle, the perception system comprising:

• a data processing apparatus;
• a memory device configured to store a perception algorithm;
• a convolutional recurrent neural network (CRNN) and at least one further artificial neural network (ANN) configured to detect at least one emergency vehicle;
• at least two microphones mounted at different positions on the vehicle configured to generate at least two time-dependent audio datasets;
• at least one camera mounted to the vehicle configured to generate at least one camera image;
• and
• a computing unit configured to:
• generate at least two spectrograms based on the at least two time-dependent audio datasets;
• generate at least one interaural difference map based on the at least two spectrograms, which contains at least one of an interaural phase difference map or an interaural time difference map or an interaural level difference map:
• generate audio source localization data for at least one grid cell of a predefined spatial grid in an environment of the vehicle;
• specify a number of audio sources in the at least one grid cell:
• apply the CRNN to a first input data containing the at least two spectrograms and the at least one interaural difference map;
• predict output data of at least one bounding box for the at least one emergency vehicle by applying the at least one further ANN to a second input data containing the at least one camera image and the at least two spectrograms;
• and
• adapt network parameters of the CRNN and the at least one further ANN by depending on the output data and the audio source localization data.

A non-transitory computer-readable storage medium storing instructions, which when executed on a computer, cause the computer to perform a method for training a perception algorithm of a vehicle, the method comprising:

• providing a convolutional recurrent neural network (CRNN) and at least one further artificial neural network (ANN) to detect at least one emergency vehicle;
• receiving at least two time-dependent audio datasets from two microphones mounted at different positions on the vehicle;
• generating at least two spectrograms based on the at least two time-dependent audio datasets;
• generating at least one interaural difference map based on the at least two spectrograms, which contains at least one of an interaural phase difference map or an interaural time difference map or an interaural level difference map;
• generating audio source localization data for at least one grid cell of a predefined spatial grid in an environment of the vehicle;
• specifying a number of audio sources in the at least one grid cell;
• applying the CRNN to a first input data containing the at least two spectrograms and the at least one interaural difference map;
• receiving at least one camera image from at least one camera mounted to the vehicle;
• predicting output data of at least one bounding box for the at least one emergency vehicle by applying the at least one further ANN to a second input data containing the at least one camera image and the at least two spectrograms;
• and
• adapting network parameters of the CRNN and the at least one further ANN by depending on the output data and the audio source localization data.