Data processing system, data processing method, and information providing system

A method for multi-modal object identification, the method comprising:

• acquiring a position of a vehicle;
• recognizing motion of an occupant of the vehicle pointing outside of the vehicle from image data captured inside the vehicle;
• recognizing, from audio data collected in the vehicle, a word or phrase comprising one or more appearance descriptors spoken by the occupant;
• matching the recognized appearance descriptor(s) to appearance-descriptor words stored for respective objects in a database that stores, for each respective object, a geographic position and appearance-descriptor words;
• searching the database using a geographic position of the vehicle, a direction pointed to by the occupant, and the matched appearance descriptor;
• extracting target object candidates pointed to by the occupant by discarding objects that lack the matched appearance descriptor(s) and filtering to objects that would be visible from the occupant's viewpoint at the time of the utterance based on the geographic position of the vehicle and positions stored in the database;
• identifying a specific target object among the target object candidates based on a dialogue with the occupant that outputs, for each candidate, a feature not included in the recognized word or phrase and receives a confirmation;
• and
• outputting an indication of the specific target object to the occupant.

A system that communicates with a vehicle for performing multi-modal object identification, the system comprising:

• a memory that stores a database of objects, wherein the database of objects includes, for each respective object, words describing an appearance of the respective object and a geographic position of the respective object;
• a communication interface communicatively coupled to the vehicle via a communication network;
• and
• one or more processors communicatively coupled to the memory and the communication interface, wherein the one or more processors are collectively configured to:
• receive, using the communication interface, audio data from a microphone located in the vehicle, wherein the microphone captures sounds in an interior of the vehicle including words spoken by an occupant of the vehicle,
• recognize, from the audio data, a word or phrase comprising one or more appearance descriptors spoken by the occupant,
• match the recognized appearance descriptor(s) to appearance-descriptor words stored for respective objects in the database,
• receive, using the communication interface, image data from an interior camera located in the vehicle, wherein the interior camera captures images of the interior of the vehicle including the occupant of the vehicle,
• recognize a direction of a pointing gesture by the occupant from the image data,
• extract target object candidates pointed at by the occupant by searching the database based on a geographic position of the vehicle, the pointing gesture of the occupant, and the matched appearance descriptor, and discarding objects that lack the matched appearance descriptor,
• identify a specific target object among the target object candidates based on a dialogue with the occupant that outputs, for each candidate, a feature of that candidate that was not included in the word or phrase spoken by the occupant and receives a confirmation, utilizing the appearance of the respective target object, and
• output, using the communication interface, an indication of the specific target object to the occupant.