Query response using media consumption history
AssigneeGoogle Inc.
InventorMatthew Sharifi
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus for receiving a natural language query of a user, and environmental data, identifying a media item based on the environmental data, determining an entity type based on the natural language query, selecting an entity associated with the media item that matches the entity type, selecting, from a media consumption database that identifies media items that have been indicated as consumed by the user, one or more media items that have been indicated as consumed by the user and that are associated with the selected entity, and providing a response to the query based on selecting the one or more media items that have been indicated as consumed by the user and that are associated with the selected entity.
Claims18
A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
- receiving a query spoken by a user and captured by a user device associated with the user, the query requesting identification of an entity, the query including one or more terms that reference an entity type associated with the entity and does not include any terms that uniquely identify the entity;
- and
- in response to receiving the query spoken by the user:
- processing, using a speech recognition engine, the query spoken by the user to identify the entity type associated with the entity;
- obtaining environmental data comprising at least one of audio data captured by a microphone associated with the user device or image data captured by an image capture device associated with the user device, the audio data representing ambient audio present in an environment of the user device while a media item is playing in the environment, and the image data representing an image of the environment while the media item is playing in the environment;
- processing, using a content recognition engine, the environmental data to identify a particular segment of a plurality of segments of the media item that is playing in the environment of the user device when the at least one of the audio data or the image data is captured;
- identifying, based on one or more timestamps associated with the identified particular segment, supplemental information for the media item that identifies one or more entities that actually appear in or are heard in the identified particular segment;
- identifying as the entity requested by the query, a particular entity of the identified one or more entities based on the entity type;
- and
- providing a response to the query spoken by the user that identifies the identified particular entity.
A system comprising:
- data processing hardware;
- and
- memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
- receiving a query spoken by a user and captured by a user device associated with the user, the query requesting identification of an entity, the query including one or more terms that reference an entity type associated with the entity and does not include any terms that uniquely identify the entity;
- in response to receiving the query spoken by the user:
- processing, using a speech recognition engine, the query spoken by the user to identify the entity type associated with the entity;
- obtaining environmental data comprising at least one of audio data captured by a microphone associated with the user device or image data captured by an image capture device associated with the user device, the audio data representing ambient audio present in an environment of the user device while a media item is playing in the environment and the image data representing an image of the environment while the media item is playing in the environment;
- processing, using a content recognition engine, the environmental data to identify a particular segment of a plurality of segments of the media item that is playing in the environment of the user device when the at least one of the audio data or the image data is captured;
- identifying, based on one or more timestamps associated with the identified particular segment, supplemental information for the media item that identifies one or more entities that actually appear in or are heard in the identified particular segment;
- identifying as the entity requested by the query, a particular entity of the identified one or more entities based on the entity type;
- and
- providing a response to the query spoken by the user that identifies the identified particular entity.
Dependencies are inferred from the claim text.