# US 12,003,804 B2 · Information processing device, information processing method, and computer program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,804 B2 - Document ID: US12003804B2 - Status: Granted - Kind: B2 - Application number: 17/275,671 - Filed: 2019-08-16 - Published: 2022-02-10 - Granted: 2024-06-04 - Earliest claimed: 2018-10-15 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Andrew Shin; Naoki Ide - Web page: https://patentgrep.com/patent/US12003804B2 ## CPC - H04N 21/42203: {sound input device, e.g. microphone} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - H04N 21/42222: {Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device} - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G10L 2015/223: {Execution procedure of a spoken command} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract To provide an information processing device, an information processing method, and a computer program that assist a device operation of a user. An information processing device includes an input unit that receives input from a user, and a presentation unit that presents information regarding a command for operating a predetermined device on the basis of the input. The input unit receives input by at least one of voice or gesture from a user. The presentation unit presents a candidate of a command including at least one of voice or gesture and partially matching the input. ## Claims ### Claim 1 An information processing device comprising: an interface configured to receive input from a user by way of at least one of voice input and gesture input of the user; presentation circuitry configured to present at least two candidate commands for at least two different types of operations of a predetermined device for the user's selection of a desired one of the candidate commands for operating the predetermined device, the candidate commands being associated with the input from the user, and each of the candidate commands being presented as a first candidate command of a first type of modal and a second candidate command of a second type of modal which is different from the first type of modal; a memory configured to store preceding voice or gesture input of the user immediately preceding each command; and search circuitry configured to search in the memory the preceding voice or gesture input that is similar to the input from the user, wherein the presentation circuitry is further configured to present at least two candidate commands for at least two different types of operations of the predetermined device corresponding to the preceding voice or gesture input searched by the search circuitry. ### Claim 2 (depends on 1) The information processing device according to claim 1, further comprising: command execution circuitry configured to execute a command based on a combination of the input and the desired one of the candidate commands selected by the user. ### Claim 3 (depends on 1) The information processing device according to claim 1, further comprising: command execution circuitry configured to execute the desired one of the candidate commands selected by the user. ### Claim 4 (depends on 1) The information processing device according to claim 1, wherein the presentation circuitry is further configured to present at least one device operation frequently performed by the user, and to present at least two candidate gesture commands for operation of the predetermined device for selection by the user. ### Claim 5 (depends on 4) The information processing device according to claim 4, wherein a candidate gesture command selected by the user is registered, and the information processing device further comprises starter circuitry configured to start an operation of the predetermined device in response to that the registered gesture command is input from the user. ### Claim 6 An information processing device comprising: an interface configured to receive input from a user; and presentation circuitry configured to present at least two candidate commands for at least two different types of operations of a predetermined device for the user's selection of a desired one of the candidate commands for operating the predetermined device, the candidate commands being associated with the input from the user, and each of the candidate commands being presented as a first candidate command of a first type of modal and a second candidate command of a second type of modal which is different from the first type of modal, wherein the input from the user is in one of a plurality of types of modals, including a voice modal and a gesture modal, and the presentation circuitry is further configured to present the candidate commands for the different types of operations of the predetermined device in a different type of modal from a type of modal of the input from the user. ### Claim 7 (depends on 6) The information processing device according to claim 6, wherein the interface is further configured to receive a remote control command transmitted from an external remote controller operated by the user, and the presentation circuitry is further configured to present a gesture or voice command equivalent to the remote control command. ### Claim 8 (depends on 6) The information processing device according to claim 6, wherein the presentation circuitry is further configured to present a plurality of device operations brought into one group, and to present at least one candidate command corresponding to the group. ### Claim 9 (depends on 8) The information processing device according to claim 8, wherein the presentation circuitry is further configured to present, as one group, a series of device operations frequently performed by the user. ### Claim 10 (depends on 8) The information processing device according to claim 8, wherein the presentation circuitry is further configured to present, as one group, a plurality of device operations selected by the user. ### Claim 11 (depends on 8) The information processing device according to claim 8, wherein the desired one of the candidate commands selected by the user is registered as a command for the device operation, and the information processing device further comprises starter circuitry configured to start an operation of the predetermined device in response to that the registered command is input from the user. ### Claim 12 An information processing method executed by an information processing device, the method comprising: receiving input from a user by way of at least one of voice input and gesture input of the user; presenting at least two candidate commands for at least two different types of operations of a predetermined device for the user's selection of a desired one of the candidate commands for operating the predetermined device, the candidate commands being associated with the input from the user, and each of the candidate commands being presented as a first candidate command of a first type of modal and a second candidate command of a second type of modal which is different from the first type of modal; storing in a memory preceding voice or gesture input of the user immediately preceding each command; and searching in the memory the preceding voice or gesture input that is similar to the input from the user, wherein the presenting further presents at least two candidate commands for at least two different types of operations of the predetermined device corresponding to the preceding voice or gesture input searched in the searching. ### Claim 13 A non-transitory computer-readable storage medium storing a computer program described in a computer-readable format so as to cause a computer to function as an information processing device comprising: an interface configured to receive input from a user; and presentation circuitry configured to present at least two candidate commands for at least two different types of operations of a predetermined device for the user's selection of a desired one of the candidate commands for operating the predetermined device, the candidate commands being associated with the input from the user, and each of the candidate commands being presented as a first candidate command of a first type of modal and a second candidate command of a second type of modal which is different from the first type of modal, wherein the input from the user is in one of a plurality of types of modals, including a voice modal and a gesture modal, and the presentation circuitry is further configured to present the candidate commands for the different types of operations of the predetermined device in a different type of modal from a type of modal of the input from the user.