# US 2019/0377546 A1 · Voice Control Of Interactive Whiteboard Appliances

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377546 A1
- Document ID: US20190377546A1
- Status: Published application
- Kind: A1
- Application number: 16/546,963
- Filed: 2019-08-21
- Published: 2019-12-12
- Earliest claimed: 2016-03-04
- Assignees: Ricoh Company, Ltd.
- Inventors: Rathnakara Malatesha; Lana Wong; Hiroshi Kitada
- Web page: https://patentgrep.com/patent/US20190377546A1

## CPC

- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 3/03545: {Pens or stylus}
- G06F 3/041: Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method and apparatus for providing voice command functionality to an interactive whiteboard appliance is provided. An interactive whiteboard appliance comprises: one or more processors; a non-transitory computer-readable medium having instructions embodied thereon, the instructions when executed by the one or more processors cause performance of: detecting, during execution of an annotation window on the interactive whiteboard appliance, a voice input received from a user; storing, in an audio packet, a recording of the voice input; transmitting the audio packet to a speech-to-text service; receiving, from the speech-to-text service, a command string comprising a transcription of the recording of the voice input; using voice mode command processing in a command processor, identifying, from the command string, an executable command that is executable by the interactive whiteboard appliance; causing the application of the interactive whiteboard appliance to execute the executable command.

## Claims

### Claim 1

A mobile device comprising:
one or more processors;
an Interactive Whiteboard (IWB) application configured with user interface controls that allow a user of the mobile device to remotely control an IWB appliance; and
a voice recognition processor configured to:
capture a voice input received from a user of the mobile device and generate an audio packet that includes a recording of the voice input received from the user of the mobile device;
cause the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance;
wherein the speech-to-text service generates and provides, to the IWB appliance, using the data received from the mobile device that identifies the IWB appliance, a command string comprising a transcription of the recording of the voice input;
wherein the IWB appliance:
identifies, from the command string that comprises the transcription of the recording of the voice input, an executable command that is executable by the IWB appliance by comparing the modified transcription of the recording of the voice input to one or more pre-determined executable commands to identify a match between the transcription of the recording of the voice input and the one or more pre-determined executable commands,
causes an application of the IWB appliance to execute the executable command.

### Claim 2 (depends on 1)

The mobile device of claim 1, wherein causing the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance includes transmitting the audio packet and data to an audio packet queue managed by an audio packet queue manager.

### Claim 3 (depends on 1)

The mobile device of claim 1, wherein the data that that identifies the IWB appliance is included in metadata of the audio packet.

### Claim 4 (depends on 1)

The mobile device of claim 1, wherein:
causing the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance includes causing the audio packet, data that identifies the IWB appliance, and second data that identifies a second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service that is physically separate from the IWB appliance; and
wherein the speech-to-text service generates and provides, to both the IWB appliance using the data received from the mobile device that identifies the IWB appliance and to the second IWB appliance using the second data that identifies the second IWB appliance, the command string comprising the transcription of the recording of the voice input.

### Claim 5 (depends on 4)

The mobile device of claim 4, wherein causing the audio packet, the data that identifies the IWB appliance, and the second data that identifies the second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service that is physically separate from the IWB appliance includes transmitting the audio packet, the data that identifies the IWB appliance, and the second data that identifies the second IWB appliance that is different than the IWB appliance, to an audio packet queue managed by an audio packet queue manager that forwards the audio packet to the speech-to-text service.

### Claim 6 (depends on 5)

The mobile device of claim 5, wherein both the data that that identifies the IWB appliance and the second data that identifies the second IWB appliance that is different than the IWB appliance are included in metadata of the audio packet.

### Claim 7 (depends on 4)

The mobile device of claim 4, wherein the voice recognition processor causes the audio packet, the data that identifies the IWB appliance, and the second data that identifies the second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service in response to a user selecting both the IWB appliance and the second IWB appliance via the IWB application on the mobile device.

### Claim 8 (depends on 6)

The mobile device of claim 6, wherein both the data that that identifies the IWB appliance and the second data that identifies the second IWB appliance that is different than the IWB appliance are included in metadata of the audio packet.

### Claim 9

One or more computer-readable media storing instructions which, when processed by one or more processors, cause:
capturing, by a voice recognition processor of a mobile device, a voice input received from a user of the mobile device and generate an audio packet that includes a recording of the voice input received from the user of the mobile device;
causing, by the voice recognition processor of the mobile device, the audio packet and data that identifies an IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance;
wherein the speech-to-text service generates and provides, to the IWB appliance, using the data received from the mobile device that identifies the IWB appliance, a command string comprising a transcription of the recording of the voice input;
wherein the IWB appliance:
identifies, from the command string that comprises the transcription of the recording of the voice input, an executable command that is executable by the IWB appliance by comparing the modified transcription of the recording of the voice input to one or more pre-determined executable commands to identify a match between the transcription of the recording of the voice input and the one or more pre-determined executable commands,
causes an application of the IWB appliance to execute the executable command.

### Claim 10 (depends on 9)

The one or more computer-readable media of claim 9, wherein causing the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance includes transmitting the audio packet and data to an audio packet queue managed by an audio packet queue manager.

### Claim 11 (depends on 9)

The one or more computer-readable media of claim 9, wherein the data that that identifies the IWB appliance is included in metadata of the audio packet.

### Claim 12 (depends on 9)

The one or more computer-readable media of claim 9, wherein:
causing the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance includes causing the audio packet, data that identifies the IWB appliance, and second data that identifies a second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service that is physically separate from the IWB appliance; and
wherein the speech-to-text service generates and provides, to both the IWB appliance using the data received from the mobile device that identifies the IWB appliance and to the second IWB appliance using the second data that identifies the second IWB appliance, the command string comprising the transcription of the recording of the voice input.

### Claim 13 (depends on 12)

The one or more computer-readable media of claim 12, wherein causing the audio packet, the data that identifies the IWB appliance, and the second data that identifies the second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service that is physically separate from the IWB appliance includes transmitting the audio packet, the data that identifies the IWB appliance, and the second data that identifies the second IWB appliance that is different than the IWB appliance, to an audio packet queue managed by an audio packet queue manager that forwards the audio packet to the speech-to-text service.

### Claim 14 (depends on 13)

The one or more computer-readable media of claim 13, wherein both the data that that identifies the IWB appliance and the second data that identifies the second IWB appliance that is different than the IWB appliance are included in metadata of the audio packet.

### Claim 15 (depends on 12)

The one or more computer-readable media of claim 12, wherein the voice recognition processor causes the audio packet, the data that identifies the IWB appliance, and the second data that identifies the second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service in response to a user selecting both the IWB appliance and the second IWB appliance via the IWB application on the mobile device.

### Claim 16 (depends on 14)

The one or more computer-readable media of claim 14, wherein both the data that that identifies the IWB appliance and the second data that identifies the second IWB appliance that is different than the IWB appliance are included in metadata of the audio packet.

### Claim 17

A computer-implemented method comprising:
capturing, by a voice recognition processor of a mobile device, a voice input received from a user of the mobile device and generate an audio packet that includes a recording of the voice input received from the user of the mobile device;
causing, by the voice recognition processor of the mobile device, the audio packet and data that identifies an IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance;
wherein the speech-to-text service generates and provides, to the IWB appliance, using the data received from the mobile device that identifies the IWB appliance, a command string comprising a transcription of the recording of the voice input;
wherein the IWB appliance:
identifies, from the command string that comprises the transcription of the recording of the voice input, an executable command that is executable by the IWB appliance by comparing the modified transcription of the recording of the voice input to one or more pre-determined executable commands to identify a match between the transcription of the recording of the voice input and the one or more pre-determined executable commands,
causes an application of the IWB appliance to execute the executable command.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein causing the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance includes transmitting the audio packet and data to an audio packet queue managed by an audio packet queue manager.

### Claim 19 (depends on 17)

The computer-implemented method of claim 17, wherein the data that that identifies the IWB appliance is included in metadata of the audio packet.

### Claim 20 (depends on 17)

The computer-implemented method of claim 17, wherein:
causing the audio packet and data that identifies the IWB appliance to be transmitted to a speech-to-text service that is physically separate from the IWB appliance includes causing the audio packet, data that identifies the IWB appliance, and second data that identifies a second IWB appliance that is different than the IWB appliance, to be transmitted to the speech-to-text service that is physically separate from the IWB appliance; and
wherein the speech-to-text service generates and provides, to both the IWB appliance using the data received from the mobile device that identifies the IWB appliance and to the second IWB appliance using the second data that identifies the second IWB appliance, the command string comprising the transcription of the recording of the voice input.
