# US 10,311,303 B2 · Information processing apparatus, information processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,303 B2 - Document ID: US10311303B2 - Status: Granted - Kind: B2 - Application number: 15/303,391 - Filed: 2015-03-17 - Published: 2017-02-02 - Granted: 2019-06-04 - Earliest claimed: 2014-05-22 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Makoto Murata; Naoki Shibuya; Junko Takabayashi; Yuuji Takimoto; Koji Sato - Web page: https://patentgrep.com/patent/US10311303B2 ## CPC - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 3/002: {Specific input/output arrangements not covered by G06F3/01 - G06F3/16 (other optical apparatus G02B27/00)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/04895: Guidance during keyboard input operation, e.g. prompting - G06F 16/24575: {using context} - G06F 16/435: Filtering based on additional data, e.g. user or group profiles - G06F 18/256: {of results relating to different input data, e.g. multimodal recognition} - G06F 40/00: Handling natural language data (speech analysis or synthesis, speech recognition G10L) - G06Q 10/40: Business processes related to social networking or social networking services - G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns} - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/20: in augmented reality scenes - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract There is provided an information processing apparatus for automatically generating information representing a context surrounding a user, the information processing apparatus including: a recognition processing unit configured to perform, on the basis of user environment information including at least any of location information representing a location where a user is present, image information relating to an environment surrounding a user, and audio information relating to the environment, an analysis process of at least any of the location information, the image information, and the audio information included in the user environment information, at a predetermined time interval, and to recognize a context surrounding the user, using the acquired result of analysis relating to the user environment; and a context candidate information generating unit configured to generate context candidate information representing a candidate of the context surrounding the user, the context candidate information including, at least, information representing the context surrounding the user and information representing the user's emotion in the context, using the result of context recognition performed by the recognition processing unit. ## Claims ### Claim 1 An information processing apparatus comprising: a processor configured to receive an image of a user as an input, perform a recognition process on the image of the user using a machine learning model, output a classification and a score corresponding to the classification as a recognition result from the recognition process performed on the image of the user, and generate context candidate information representing a context surrounding the user based on the recognition result and a recognized emotion of the user, wherein the machine learning model is constructed from a plurality of images collected in advance. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the outputs the recognition result each time the context surrounding the user changes. ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, wherein the recognized emotion of the user is generated using an emotion representation table preliminarily provided for each of the recognized contexts. ### Claim 4 (depends on 2) The information processing apparatus according to claim 2, wherein the context candidate information includes image information of the received image relating to the context surrounding the user. ### Claim 5 (depends on 1) The information processing apparatus according to claim 1, wherein the processor is further configured to initiate display the recognition result in a predetermined area of a predetermined display screen. ### Claim 6 (depends on 1) The information processing apparatus according to claim 1, wherein the processor is further configured to perform the recognition process by recognizing the context by a time-series pattern recognition process based on time course of the recognition result, or a rule-based process based on predetermined conditional processing, using the recognition result. ### Claim 7 (depends on 1) The information processing apparatus according to claim 1, wherein the processor is further configured to update an expression representing the recognized emotion of the user included in an emotion representation table preliminarily provided for each of the recognized contexts, using at least any of a result of an analysis process relating to an environment surrounding the user, a remark or a sentence expression provided by the user, and an output from a biological sensor provided in the information processing apparatus. ### Claim 8 (depends on 1) The information processing apparatus according to claim 1, wherein the processor is further configured to cooperate with another information processing apparatus with which mutual communication is possible, and improve precision of, or share, the generated context candidate information. ### Claim 9 (depends on 1) The information processing apparatus according to claim 1, wherein the processor is further configured to transmit a piece of information selected by the user from the generated context candidate information to an information posting server providing a social network service, as context information representing the context surrounding the user. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the information processing apparatus comprises a personal digital assistant carried by the user, or a wearable terminal worn by the user. ### Claim 11 An information processing method, executed via at least one processor, the method comprising: receiving an image of a user as an input, performing a recognition process on the image of the user by a machine learning model, outputting a classification and a score corresponding to the classification as a recognition result from the recognition process performed on the image of the user, and generating context candidate information representing a context surrounding the user based on the recognition result and a recognized emotion of the user, wherein the machine learning model is constructed from a plurality of images collected in advance. ### Claim 12 A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the compute to execute a method, the method comprising: receiving an image of a user as an input, performing a recognition process on the image of the user by a machine learning model, outputting a classification and a score corresponding to the classification as a recognition result from the recognition process performed on the image of the user, and generating context candidate information representing a context surrounding the user based on the recognition result and a recognized emotion of the user, wherein the machine learning model is constructed from a plurality of images collected in advance. ### Claim 13 (depends on 1) The information processing apparatus according to claim 1, wherein the classification includes at least one of a face, a landscape, a dish, or an object. ### Claim 14 (depends on 13) The information processing apparatus according to claim 13, wherein, when the processor outputs the classification including at least one face, the processor further outputs at least one of a number of faces, coordinates, angles, presence or absence of smile, age, or race. ### Claim 15 (depends on 2) The information processing apparatus according to claim 2, wherein, when the processor outputs the classification including at least one face, a face detection frame is displayed on the image of the at least one face in the image. ### Claim 16 (depends on 1) The information processing apparatus according to claim 1, wherein the image received at the processor is sent from an outside terminal. ### Claim 17 (depends on 1) The information processing apparatus according to claim 1, wherein the processor generates a sentence for the context candidate information.