# US 12,182,475 B2 · Information processing apparatus and command processing method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,475 B2 - Document ID: US12182475B2 - Status: Granted - Kind: B2 - Application number: 17/771,043 - Filed: 2020-10-20 - Published: 2022-11-10 - Granted: 2024-12-31 - Earliest claimed: 2019-10-30 - Assignees: Sony Group Corporation - Applicants: Sony Group Corporation - Inventors: Hiro Iwase; Yuhei Taki; Kunihito Sawai - Web page: https://patentgrep.com/patent/US12182475B2 ## CPC - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 3/012: {Head tracking input arrangements} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06T 7/00: Image analysis - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G06F 2203/04806: Zoom, i.e. interaction techniques or interactors for controlling the zooming operation - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A detection unit (30) detects an input start timing of a command by a gesture on an operation target with a temporal change. A command processing unit (31) performs processing of a command recognized from the gesture based on the state of the operation target at the input start timing detected by the detection unit (30). ## Claims ### Claim 1 An information processing apparatus comprising: circuitry that detects input of a first command; performs processing according to the first command until a second command is detected; detects a first time point at which input of the second command is started, the second command being input by a gesture on an operation target with a temporal change; and performs, beginning at a second time point later than the first time point, processing of the second command recognized from the gesture based on a state of the operation target at the first time point, processing of the second command ending the processing according to the first command. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the first command and the second command are input by voice, and the first time point is a time point when utterance by the voice for the second command begins. ### Claim 3 (depends on 1) The information processing apparatus according to claim 1, wherein circuitry stores the first time point or the state of the operation target at the first time point, and performs processing of the second command on the state of the operation target that has gone back to the state at the stored first time point, or on the stored state of the operation target. ### Claim 4 (depends on 2) The information processing apparatus according to claim 2, wherein, in a case where the second command is input by the voice within a predetermined specified time after a filler utterance, the first time point corresponds to a time point when the filler utterance started. ### Claim 5 (depends on 1) The information processing apparatus according to claim 1, further including: a display that displays the operation target, and the circuitry further captures an image of a user who inputs the second command; and detects at least one of a direction of a face or line of sight of the user from the image captured, and determines whether the user is viewing the display based on at least one of the direction of the face or the line of sight detected when the second command is input, and in a case where the user is viewing the display, the circuitry performs processing of the second command based on the state of the operation target at first time point. ### Claim 6 (depends on 1) The information processing apparatus according to claim 1, wherein the circuitry performs processing of the second command based on a state of the operation target at a time point that differs from the first time point by a length of a perception time corresponding to an attribute of a user who inputs the second command. ### Claim 7 (depends on 5) The information processing apparatus according to claim 5, wherein the display displays a state of the operation target returned to the first time point together with a current state of the operation target with a temporal change. ### Claim 8 (depends on 5) The information processing apparatus according to claim 5, wherein the operation target is an operation in which an item to be selected sequentially transitions from among a plurality of items, and the circuitry, on the display, displays an item whose transition is within a predetermined range beginning with an item to be selected at a timing detected to be selectable by number. ### Claim 9 (depends on 5) The information processing apparatus according to claim 5, wherein the circuitry detects the first time point from a recognition result of the image captured. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the circuitry suspends a change of the state of the operation target, the change being a change accompanying a temporal change, in the state of the first time point, stops the operation target when the second command recognized from the gesture is a stop instruction, and cancels the suspension when the second command is other than the stop instruction. ### Claim 11 (depends on 1) The information processing apparatus according to claim 1, wherein the circuitry performs loop display of a temporal change of the operation target during a certain period from the first time point. ### Claim 12 (depends on 11) The information processing apparatus according to claim 11, wherein the circuitry displays a search tag by which the state of the operation target during the certain period can be identified. ### Claim 13 (depends on 2) The information processing apparatus according to claim 2, wherein, when a time from the first time point of the utterance made before a wake up word of voice recognition to the utterance of the wake up word is a specified time or less, the circuitry performs processing of the second command based on the state of the operation target at the first time point of the utterance made before the wake up word. ### Claim 14 A command processing method to be executed by a computer, the method comprising: detecting input of a first command; performing processing according to the first command until a second command is detected; detecting a first time point at which input of the second command is started, the second command being input by a gesture on an operation target with a temporal change; and executing, beginning at a second time point later than the first time point, processing of the second command recognized from the gesture based on a state of the operation target at the first time point, processing of the second command ending the processing according to the first command. ### Claim 15 A non-transitory computer-readable medium encoded with computer-readable instructions that, when executed by a computer, cause the computer to perform a method comprising: detecting input of a first command; performing processing according to the first command until a second command is detected; detecting a first time point at which input of the second command is started, the second command being input by a gesture on an operation target with a temporal change; and executing, beginning at a second time point later than the first time point, processing of the second command recognized from the gesture based on a state of the operation target at the first time point, processing of the second command ending the processing according to the first command. ### Claim 16 (depends on 1) The information processing apparatus according to claim 1, wherein the processing of the first command and the second command includes volume adjustment. ### Claim 17 (depends on 5) The information processing apparatus according to claim 5, wherein the circuitry periodically captures images of the user. ### Claim 18 (depends on 2) The information processing apparatus according to claim 2, wherein the circuitry detects the first time point when a volume of the voice reaches a predetermined level. ### Claim 19 (depends on 18) The information processing apparatus according to claim 18, wherein the volume of the voice is determined as a root mean square (RMS) volume.