# US 12,014,731 B2 · Suggesting user actions during a video conference

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,014,731 B2
- Document ID: US12014731B2
- Status: Granted
- Kind: B2
- Application number: 17/162,082
- Filed: 2021-01-29
- Published: 2022-08-04
- Granted: 2024-06-18
- Earliest claimed: 2021-01-29
- Assignees: Zoom Video Communications, Inc.
- Applicants: Zoom Video Communications, Inc.
- Inventors: Samuel Lum
- Web page: https://patentgrep.com/patent/US12014731B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04L 51/18: Commands or executable codes
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- H04L 12/1818: {Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties}
- H04L 12/1827: {Network arrangements for conference optimisation or adaptation}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

One example method includes receiving, by a computing device, audio during a video conference having a plurality of participants, the audio comprising spoken words by a user of the computing device; recognizing one or more words from the spoken words; identifying one or more keywords within the one or more recognized words; accessing a set of rules comprising one or more rules, each rule of the one or more rules associated with an application of a set of applications, and at least one rule of the one or more rules associated with a functionality of a respective application; determining a context associated with the one or more keywords; determining an application to execute based on the one or more keywords, the context, and the one or more rules, wherein determining the application comprises determining a functionality of the application to invoke; and in response to receiving user confirmation of the functionality of the application to invoke, executing the application and invoking the functionality.

## Claims

### Claim 1

A method comprising:
receiving and passively monitoring, by a video conference software executed by a client computing device, audio received by a microphone of the client computing device during a video conference having a plurality of participants, the audio comprising spoken words by a user of the client computing device;
recognizing, by the video conference software, one or more words from the spoken words based on the passive monitoring;
identifying one or more keywords based on the one or more recognized words;
accessing a set of rules comprising one or more rules, each rule of the one or more rules associated with an application of a set of applications, and at least one rule of the one or more rules associated with a functionality of a respective application;
determining a context associated with the one or more keywords;
determining an application to execute based on the one or more keywords, the context, and the one or more rules, wherein determining the application comprises determining a functionality of the application to invoke; and
in response to receiving user confirmation of the functionality of the application to invoke, executing the application and invoking the functionality.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
outputting a notification to the user, the notification suggesting executing the application and invoking the functionality; and
wherein receiving the user confirmation comprises receiving a user interaction with the notification.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more keywords comprises a name of the application.

### Claim 4 (depends on 1)

The method of claim 1, wherein the context comprises a keyword associated with an action.

### Claim 5 (depends on 1)

The method of claim 1, wherein the one or more rules comprise associations between applications, contexts, and actions.

### Claim 6 (depends on 1)

The method of claim 1, further comprising determining application data from the one or more keywords and supplying the application data to the application or the invoked functionality.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
recognizing one or more additional words from the spoken words;
identifying one or more additional keywords from the one or more additional words;
accessing the one or more rules;
determining a second context based on the one or more recognized additional words;
determining a second application to execute based on the one or more additional keywords, the second context, and the one or more rules, wherein determining the second application comprises determining a second functionality of the second application to invoke; and
in response to receiving a rejection of the second functionality of the second application to invoke, not executing the application.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
outputting a second notification to the user, the second notification suggesting executing the second application and invoking the second functionality; and
wherein receiving the rejection comprises not receiving a user interaction with the second notification after a predetermined period of time.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
presenting a transcript of the spoken words comprising the one or more recognized words;
emphasizing the one or more keywords in the transcript based on the set of rules;
receiving a first selection of (i) at least one of the one or more keywords, (ii) one or more additional words of the transcript, and (iii) at least one rule of the set of rules; and
updating a context identification component based on the selection.

### Claim 10

A system comprising:
a non-transitory computer-readable medium;
a communications interface;
a microphone; and
a processor communicatively coupled to the non-transitory computer-readable medium and the communications interface, the processor configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to:
execute video conference software to participate in a video conference having a plurality of participants;
receive and passively monitor, by the video conference software, audio received by the microphone during the video conference, the audio comprising spoken words by a user of the video conference software;
recognize, by the video conference software, one or more words from the spoken words based on the passive monitoring;
identify one or more keywords based on the one or more recognized words;
access a set of rules comprising one or more rules, each rule of the one or more rules associated with an application of a set of applications, and at least one rule of the one or more rules associated with a functionality of a respective application;
determine a context associated with the one or more keywords;
determine an application to execute based on the one or more keywords, the context, and the one or more rules, wherein determining the application comprises determining a functionality of the application to invoke; and
in response to receipt of a user confirmation of the functionality of the application to invoke, execute the application and invoke the functionality.

### Claim 11 (depends on 10)

The system of claim 10, wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to:
output a notification to the user, the notification suggesting executing the application and invoking the functionality; and
receive a user interaction with the notification.

### Claim 12 (depends on 10)

The system of claim 10, wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to determine application data from the one or more keywords and supplying the application data to the application or the invoked functionality.

### Claim 13 (depends on 10)

The system of claim 10, wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to:
recognize one or more additional words from the spoken words;
identify one or more additional keywords from the one or more additional words;
access the one or more rules;
determine a second context based on the one or more recognized additional words;
determine a second application to execute based on the one or more additional keywords, the second context, and the one or more rules, wherein determining the second application comprises determining a second functionality of the second application to invoke; and
in response receipt of a rejection of the second functionality of the second application to invoke, not execute the application.

### Claim 14 (depends on 13)

The system of claim 13, wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to:
output a second notification to the user, the second notification suggesting executing the second application and invoking the second functionality; and
wherein receipt of the rejection comprises not receipt of a user interaction with the second notification after a predetermined period of time.

### Claim 15 (depends on 10)

The system of claim 10, wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to:
present a transcript of the spoken words comprising the one or more recognized words;
emphasize the one or more keywords in the transcript based on the set of rules;
receive a first selection of (i) at least one of the one or more keywords, (ii) one or more additional words of the transcript, and (iii) at least one rule of the set of rules; and
update a context identification component based on the selection.

### Claim 16

A non-transitory computer-readable medium comprising processor-executable instructions configured to cause a processor to:
execute video conference software on a client computing device to participate in a video conference having a plurality of participants;
receive and passively monitor, by the video conference software, audio received by a microphone of the client computing device during the video conference, the audio comprising spoken words by a user of the video conference software;
recognize, by the video conference software, one or more words from the spoken words based on the passive monitoring;
identify one or more keywords based on the one or more recognized words;
access a set of rules comprising one or more rules, each rule of the one or more rules associated with an application of a set of applications, and at least one rule of the one or more rules associated with a functionality of a respective application;
determine a context associated with the one or more keywords;
determine an application to execute based on the one or more keywords, the context, and the one or more rules, wherein determining the application comprises determining a functionality of the application to invoke; and
in response to receipt of a user confirmation of the functionality of the application to invoke, execute the application and invoke the functionality.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising non-transitory computer-readable medium comprising processor-executable instructions configured to cause a processor to:
output a notification to the user, the notification suggesting executing the application and invoking the functionality; and
receive a user interaction with the notification.

### Claim 18 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising determining application data from the one or more keywords and supplying the application data to the application or the invoked functionality.

### Claim 19 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the processor is configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to cause the processor to:
recognize one or more additional words from the spoken words;
identify one or more additional keywords from the one or more additional words;
access the one or more rules;
determine a second context based on the one or more recognized additional words;
determine a second application to execute based on the one or more additional keywords, the second context, and the one or more rules, wherein determining the second application comprises determining a second functionality of the second application to invoke; and
in response receipt of a rejection of the second functionality of the second application to invoke, not execute the application.

### Claim 20 (depends on 16)

The non-transitory computer-readable medium of claim 16, further comprising non-transitory computer-readable medium comprising processor-executable instructions configured to cause a processor to:
present a transcript of the spoken words comprising the one or more recognized words;
emphasize the one or more keywords in the transcript based on the set of rules;
receive a first selection of (i) at least one of the one or more keywords, (ii) one or more additional words of the transcript, and (iii) at least one rule of the set of rules; and
update a context identification component based on the selection.