# US 2017/0285915 A1 · Intelligent automated assistant in a media environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0285915 A1
- Document ID: US20170285915A1
- Status: Published application
- Kind: A1
- Application number: 15/627,869
- Filed: 2017-06-20
- Published: 2017-10-05
- Earliest claimed: 2015-09-08
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Lia T. NAPOLITANO; Grace H. HWANG; Henrique D. PENHA; Jeremiah D. SHAW; Jorge S. FINO
- Web page: https://patentgrep.com/patent/US20170285915A1

## CPC

- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/3329: Natural language query formulation
- G06F 16/438: Presentation of query results
- G06F 16/45: Clustering; Classification
- G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F 16/483: using metadata automatically derived from the content
- G06F 16/487: using geographical or spatial information, e.g. location
- G06F 16/489: {using time information}
- G06F 16/90332: {Natural language query formulation or dialogue systems}
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- H04N 21/41265: {having a remote control device for bidirectional communication between the remote control device and client device (unidirectional remote control device H04N21/42204)}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/42204: {User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor (constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205)}
- H04N 21/47: End-user applications
- H04N 21/4828: {for searching programme descriptors (retrieval of video data G06F16/739)}
- G10L 15/00: Speech recognition (G10L17/00 takes precedence)
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 21/0208: Noise filtering
- G10L 2015/221: {Announcement of recognition results}
- G10L 2015/223: {Execution procedure of a spoken command}
- H04N 21/42206: {characterized by hardware details}
- H04N 21/42224: {Touch pad or touch panel provided on the remote control}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and processes are disclosed for operating a digital assistant in a media environment. In an exemplary embodiment, a user can interact with a digital assistant of a media device while content is displayed by the media device. In one approach, a plurality of exemplary natural language requests can be displayed in response to detecting a user input of a first input type. The plurality of exemplary natural language requests can be contextually-related to the displayed content. In another approach, a user request can be received in response to detecting a user input of a second input type. A task that at least partially satisfies the user request can be performed. The performed task can depend on the nature of the user request and the content being displayed by the media device. In particular, the user request can be satisfied while reducing disruption to user consumption of media content.

## Claims

### Claim 1

An electronic device for operating a digital assistant of a media system comprising:
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying content on a display unit;
while displaying the content, detecting a user input;
in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;
obtaining a determination of a user intent corresponding to the user utterance;
obtaining a determination of whether the user intent comprises a request to adjust a state or a setting of an application on the electronic device; and
in response to obtaining a determination that the user intent comprises a request to adjust a state or a setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent, wherein:
the displayed content includes media content playing on the media device;
the user utterance is a natural language expression indicating that a user did not hear a portion of audio associated with the media content;
the request to adjust a state or a setting of an application on the electronic device comprises a request to replay a portion of the media content corresponding to the portion of the audio that the user did not hear; and
adjusting the state or the setting of the application comprises:
rewinding the media content by a predetermined amount to a previous portion of the media content; and
restarting play of the media content from the previous portion.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the request to adjust a state or a setting of an application on the electronic device comprises a request to play a particular media item, and wherein adjusting the state or the setting of the application to satisfy the user intent comprises playing the particular media item.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the displayed content includes a user interface with a media item, wherein the user utterance does not explicitly define the particular media item to be played, and further comprising:
determining whether a focus of the user interface is on the media item; and
in accordance with a determination that a focus of the user interface is on the media item, identifying the media item as the particular media item to be played.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to launch the application on the electronic device.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the displayed content comprises media content playing on the electronic device, and wherein the state or the setting relates to the media content being played on the electronic device.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to fast-forward or rewind the media content playing on the electronic device.

### Claim 7 (depends on 5)

The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to jump forward or backward in the media content to play a particular portion of the media content.

### Claim 8 (depends on 5)

The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to pause the playing of the media content on the electronic device.

### Claim 9 (depends on 5)

The electronic device of claim 5, wherein the request to adjust a state or a setting of an application on the electronic device includes a request to turn-on or turn-off subtitles of the media content.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein:
the displayed content includes a user interface with a first media item and a second media item;
the request to adjust a state or a setting of an application on the electronic device includes a request to switch a focus of the user interface from the first media item to the second media item; and
adjusting the state or the setting of the application to satisfy the user intent comprises switching a focus of the user interface from the first media item to the second media item.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein adjusting the state or the setting of the application further comprises:
turning on closed captioning prior to restarting play of the media content from the previous portion.

### Claim 12 (depends on 10)

The electronic device of claim 10, wherein:
the request to adjust a state or a setting of an application on the electronic device further comprises a request to increase a volume of the audio associated with the media content; and
adjusting the state or the setting of the application further comprises increasing the volume of the audio associated with the media content prior to restarting play of the media content from the previous portion.

### Claim 13 (depends on 1)

The electronic device of claim 1, wherein obtaining a determination of a user intent corresponding to the user utterance further comprises:
obtaining a determination of a user sentiment associated with the user utterances, wherein the user intent is determined based on the determined user sentiment.

### Claim 14 (depends on 1)

The electronic device of claim 1, further comprising:
in response to obtaining a determination that the user intent does not comprise a request to adjust a state or a setting of an application on the electronic device, obtaining a determination of whether the user intent is one of a plurality of predetermined request types; and
in response to obtaining a determination that the user intent is one of a plurality of predetermined request types:
obtaining results that at least partially satisfy the user intent; and
displaying, on the display unit, the results in text form.

### Claim 15 (depends on 14)

The electronic device of claim 14, wherein the results in text form are overlaid on the displayed content.

### Claim 16 (depends on 14)

The electronic device of claim 14, wherein the displayed content comprises media content playing on the electronic device, and wherein the media content continues to play while the results in text form are displayed.

### Claim 17 (depends on 14)

The electronic device of claim 14, further comprising:
in response to obtaining a determination that the user intent is not one of a plurality of predetermined request types:
obtaining second results that at least partially satisfy the user intent;
determining whether the displayed content comprises media content playing on the electronic device; and
in accordance with a determination that the displayed content comprises media content:
determining whether the media content can be paused; and
in accordance with a determination that the media content cannot be paused, displaying on the display unit a second user interface with a portion of the second results, wherein a display area occupied by the second user interface on the display unit is smaller than a display area occupied by the media content on the display unit.

### Claim 18 (depends on 17)

The electronic device of claim 17, wherein the user intent is not a media search query, and wherein the second results include one or more media items having media content that at least partially satisfies the user intent.

### Claim 19 (depends on 17)

The electronic device of claim 17, wherein the user intent is a media search query and the second results comprise a plurality of media items corresponding to the media search query.

### Claim 20 (depends on 17)

The electronic device of claim 17, further comprising:
in accordance with a determination that the displayed content does not comprise media content playing on the electronic device, displaying on the display unit a third user interface with a portion of the second results, wherein the third user interface occupies a majority of the display area of the display unit.

### Claim 21 (depends on 20)

The electronic device of claim 20, wherein:
the displayed content comprises the third user interface with previous results related to a previous user request received prior to detecting the user input; and
in accordance with a determination that the displayed content does not comprise media content playing on the electronic device, display of the previous results in the third user interface is replaced with the display of the second results.

### Claim 22 (depends on 17)

The electronic device of claim 17, further comprising:
in accordance with the determination that the displayed content comprises media content playing on the electronic device:
determining whether the displayed content includes the second user interface with previous results from a previous user request, wherein in accordance with a determination that the displayed content includes the second user interface with previous results from a previous user request, the previous results are replaced with the second results.

### Claim 23 (depends on 17)

The electronic device of claim 17, further comprising:
in accordance with a determination that the media content can be paused:
pausing the playing of the media content on the electronic device;
displaying on the display unit the third user interface with a portion of the second results, wherein the third user interface occupies a majority of the display area of the display unit.

### Claim 24 (depends on 1)

The electronic device of claim 1, further comprising:
obtaining a determination that the user intent does not correspond to one of a plurality of core competencies associated with the electronic device; and
causing a second electronic device to perform a task in furtherance of satisfying the user intent.

### Claim 25

A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:
displaying content on a display unit;
while displaying the content, detecting a user input;
in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;
obtaining a determination of a user intent corresponding to the user utterance;
obtaining a determination of whether the user intent comprises a request to adjust a state or a setting of an application on the electronic device; and
in response to obtaining a determination that the user intent comprises a request to adjust a state or a setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent, wherein:
the displayed content includes media content playing on the media device;
the user utterance is a natural language expression indicating that a user did not hear a portion of audio associated with the media content;
the request to adjust a state or a setting of an application on the electronic device comprises a request to replay a portion of the media content corresponding to the portion of the audio that the user did not hear; and
adjusting the state or the setting of the application comprises:
rewinding the media content by a predetermined amount to a previous portion of the media content; and
restarting play of the media content from the previous portion.

### Claim 26

A method for operating a digital assistant of a media system, the method comprising:
at an electronic device with one or more processors and memory:
displaying content on a display unit;
while displaying the content, detecting a user input;
in response to detecting the user input, sampling audio data, wherein the audio data includes a user utterance;
obtaining a determination of a user intent corresponding to the user utterance;
obtaining a determination of whether the user intent comprises a request to adjust a state or a setting of an application on the electronic device; and
in response to obtaining a determination that the user intent comprises a request to adjust a state or a setting of an application on the electronic device, adjusting the state or the setting of the application to satisfy the user intent, wherein:
the displayed content includes media content playing on the media device;
the user utterance is a natural language expression indicating that a user did not hear a portion of audio associated with the media content;
the request to adjust a state or a setting of an application on the electronic device comprises a request to replay a portion of the media content corresponding to the portion of the audio that the user did not hear; and
adjusting the state or the setting of the application comprises:
rewinding the media content by a predetermined amount to a previous portion of the media content; and
restarting play of the media content from the previous portion.
