# US 2021/0240984 A1 · Information processing device, information processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240984 A1 - Document ID: US20210240984A1 - Status: Published application - Kind: A1 - Application number: 17/052,035 - Filed: 2019-03-08 - Published: 2021-08-05 - Earliest claimed: 2018-05-09 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: MASATO NISHIO; YUHEI YABE; TOMOO MIZUKAMI - Web page: https://patentgrep.com/patent/US20210240984A1 ## CPC - G06V 10/7784: {based on feedback from supervisors} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/41: {Interactive pattern learning with a human teacher} - G06V 10/7788: {the supervisor being a human, e.g. interactive learning with a human teacher} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An information processing device is provided that includes an operation control unit which controls the operations of an autonomous mobile object that performs an action according to a recognition operation. Based on the detection of the start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile object to obtain information regarding the learning target that is to be learnt in a corresponding manner to a taught label. Moreover, an information processing method is provided that is implemented in a processor and that includes controlling the operations of an autonomous mobile object which performs an action according to a recognition operation. Based on the detection of the start of teaching related to pattern recognition learning, the controlling of the operations includes instructing the autonomous mobile object to obtain information regarding the learning target that is to be learnt in a corresponding manner to a taught label. ## Claims ### Claim 1 An information processing device comprising an operation control unit that controls operation of an autonomous mobile object which performs an action according to a recognition operation, wherein based on detection of start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile object to obtain information regarding a learning target that is to be learnt in a corresponding manner to a taught label. ### Claim 2 (depends on 1) The information processing device according to claim 1, wherein the learning target includes a space region, and based on the label assigned in an environmental map that is generated based on sensor information collected by the autonomous mobile object, the operation control unit instructs the autonomous mobile object to obtain information regarding the space region. ### Claim 3 (depends on 2) The information processing device according to claim 2, wherein the operation control unit instructs the autonomous mobile object to move, in real space, to the space region that corresponds to a specified area to which the label is assigned in the environmental map. ### Claim 4 (depends on 3) The information processing device according to claim 3, wherein the operation control unit instructs the autonomous mobile object to photograph the space region corresponding to the specified region. ### Claim 5 (depends on 2) The information processing device according to claim 2, wherein the environmental map is generated or updated using SLAM technology. ### Claim 6 (depends on 1) The information processing device according to claim 1, wherein the operation control unit controls position or posture of the autonomous mobile object in such a way that learning target is photographed from a plurality of angles. ### Claim 7 (depends on 1) The information processing device according to claim 1, wherein the operation control unit controls position or posture of the autonomous mobile object in such a way that the learning target is photographed in entirety. ### Claim 8 (depends on 7) The information processing device according to claim 7, wherein the operation control unit controls position or posture of the autonomous mobile object based on result of boundary recognition performed regarding the learning target. ### Claim 9 (depends on 1) The information processing device according to claim 1, wherein the teaching is performed by a user, and the operation control unit instructs the autonomous mobile object to photograph the learning target identified based on instruction from the user. ### Claim 10 (depends on 9) The information processing device according to claim 9, wherein instruction from the user is given using a gesture. ### Claim 11 (depends on 1) The information processing device according to claim 1, wherein the teaching is performed based on information obtained from the learning target. ### Claim 12 (depends on 11) The information processing device according to claim 11, wherein the teaching is performed by photographing a marker assigned to the learning target. ### Claim 13 (depends on 11) The information processing device according to claim 11, wherein the teaching is performed based on information sent from the learning target using wireless communication. ### Claim 14 (depends on 11) The information processing device according to claim 11, wherein the teaching is performed based on an inaudible sound emitted from the learning target. ### Claim 15 (depends on 9) The information processing device according to claim 9, wherein the label is extracted from speech of the user. ### Claim 16 (depends on 1) The information processing device according to claim 1, wherein the learning target includes an object, and the operation control unit instructs the autonomous mobile object to photograph the object identified based on the teaching. ### Claim 17 (depends on 1) The information processing device according to claim 1, wherein the learning target includes an action of an object, and the operation control unit instructs the autonomous mobile object to obtain information regarding the action of the object identified based on the teaching. ### Claim 18 (depends on 1) The information processing device according to claim 1, wherein the learning target includes a space region, and the operation control unit instructs the autonomous mobile object to obtain information regarding the space region identified based on the teaching. ### Claim 19 (depends on 1) The information processing device according to claim 1, wherein the operation control unit instructs the autonomous mobile object to perform an action indicating that the pattern recognition learning has started or ended. ### Claim 20 (depends on 1) The information processing device according to claim 1, wherein the operation control unit instructs the autonomous mobile object to perform a guiding operation to guide user to perform the teaching. ### Claim 21 (depends on 1) The information processing device according to claim 1, further comprising an application control unit that controls a user interface to be used in associating the label regarding the pattern recognition learning and the learning target. ### Claim 22 (depends on 21) The information processing device according to claim 21, wherein the application control unit provides a way, in the user interface, for user to specify an arbitrary specified area in an environmental map and to assign the label to the specified area. ### Claim 23 (depends on 22) The information processing device according to claim 22, wherein the application control unit presents, in the user interface, candidates for the label assignable to the specified area by user. ### Claim 24 (depends on 21) The information processing device according to claim 21, wherein the application control unit provides a way, in the user interface, for user to correct association of the label and the learning target. ### Claim 25 (depends on 1) The information processing device according to claim 1, further comprising a learning unit that performs the pattern recognition learning based on the label and based on an image of the learning target. ### Claim 26 An information processing method implemented in a processor, comprising: controlling operation of an autonomous mobile object which performs an action according to a recognition operation, wherein based on detection of start of teaching related to pattern recognition learning, the controlling of the operation includes instructing the autonomous mobile object to obtain information regarding a learning target that is to be learnt in a corresponding manner to a taught label. ### Claim 27 A program that causes a computer to function as an information processing device including an operation control unit that controls operation of an autonomous mobile object which performs an action according to a recognition operation, wherein based on detection of start of teaching related to pattern recognition learning, the operation control unit instructs the autonomous mobile object to obtain information regarding a learning target that is to be learnt in a corresponding manner to a taught label.