# US 9,002,707 B2 · Determining the position of the source of an utterance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,002,707 B2 - Document ID: US9002707B2 - Status: Granted - Kind: B2 - Application number: 13/669,843 - Filed: 2012-11-06 - Published: 2013-05-16 - Granted: 2015-04-07 - Earliest claimed: 2011-11-11 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Keiichi Yamada - Web page: https://patentgrep.com/patent/US9002707B2 ## CPC - G10L 17/10: Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems - G06F 18/2134: based on separation criteria, e.g. independent component analysis - G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods} - G06V 20/35: {Categorising the entire scene, e.g. birthday party or wedding scene} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G01S 3/802: Systems for determining direction or deviation from predetermined direction - G06F 2218/22: Source localisation; Inverse modelling - G10L 21/0272: Voice signal separating - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract An information processing apparatus includes: a plurality of information input units; an event detection unit that generates event information including estimated position information and estimated identification information of users present in the real space based on analysis of the information from the information input unit; and an information integration processing unit that inputs the event information, and generates target information including a position of each user and user identification information based on the input event information, and signal information representing a probability value of the event generation source, wherein the information integration processing unit includes an utterance source probability calculation unit, and wherein the utterance source probability calculation unit performs a process of calculating an utterance source score as an index value representing an utterance source probability of each target by multiplying weights based on utterance situations by a plurality of different information items from the event detection unit. ## Claims ### Claim 1 An information processing apparatus comprising: a plurality of information input units that input observation information of a real space; an event detection unit that generates event information including estimated position information and estimated identification information of users present in the real space based on analysis of the information input from the information input unit; and an information integration processing unit that inputs the event information, and generates target information including a position of each user and user identification information on the basis of the input event information, and signal information representing a probability value of the event generation source, wherein the information integration processing unit includes an utterance source probability calculation unit, and wherein the utterance source probability calculation unit performs a process of calculating an utterance source score as an index value representing an utterance source probability of each target by multiplying weights by a plurality of different information items input from the event detection unit, wherein the weights are adjusted based on whether a single utterance occurs from only one target or simultaneous utterance occurs from two targets. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the utterance source probability calculation unit receives inputs of (a) user position information with respect to a sound source direction, and (b) user identification information based on utterer identification which are corresponding to an utterance event as input information from a voice event detection unit constituting the event detection unit, inputs (a) user position information with respect to a face position of a user, (b) user identification information based on face identification, and (c) lip movement information as the target information generated based on input information from an image event detection unit constituting the event detection unit, and performs a process of calculating the utterance source score based on the input information by adopting at least one item of the information. ### Claim 3 An information processing apparatus comprising: a plurality of information input units that input observation information of a real space; an event detection unit that generates event information including estimated position information and estimated identification information of users present in the real space based on analysis of the information input from the information input unit; and an information integration processing unit that inputs the event information, and generates target information including a position of each user and user identification information on the basis of the input event information, and signal information representing a probability value of the event generation source, wherein the information integration processing unit includes an utterance source probability calculation unit, wherein the utterance source probability calculation unit performs a process of calculating an utterance source score as an index value representing an utterance source probability of each target by multiplying weights based on utterance situations by a plurality of different information items input from the event detection unit, and wherein the utterance source probability calculation unit applies sound source direction information D, utterer identification information S, and lip movement information L, and performs a process of calculating the utterance source score according to the following formula for calculating the utterance source score P: P=Dα·Sβ·Lγ where α is a weight coefficient of sound source direction information, β is a weight coefficient of utterer identification information, γ is a weight coefficient of lip movement information, and α+β+γ=1. ### Claim 4 (depends on 3) The information processing apparatus according to claim 3, wherein the utterance source probability calculation unit performs a process of adjusting the weight coefficients α, β, and γ according to an utterance situation. ### Claim 5 (depends on 3) The information processing apparatus according to claim 3, wherein the utterance source probability calculation unit performs a process of adjusting the weight coefficients α, β, and γ according to the following two conditions of (Condition 1) whether single utterance from only one target or simultaneous utterance from two targets and (Condition 2) whether positions of two targets are close to each other or positions of the two targets are far. ### Claim 6 (depends on 3) The information processing apparatus according to claim 3, wherein the utterance source probability calculation unit performs a process of adjusting the weight coefficients α, β, and γ such that the weight coefficient γ of the lip movement information is small in a situation where two targets with an utterance probability are present and the two targets speak simultaneously. ### Claim 7 (depends on 3) The information processing apparatus according to claim 3, wherein the utterance source probability calculation unit performs a process of adjusting the weight coefficients α, β, and γ such that the weight coefficient α of the sound source direction information is small in a situation where two targets with an utterance probability are present and positions of two targets are close to each other and only one target speaks. ### Claim 8 (depends on 3) The information processing apparatus according to claim 3, wherein the utterance source probability calculation unit performs a process of adjusting the weight coefficients α, β, and γ such that the weight coefficient γ of the lip movement information and the weight coefficient α of the sound source direction information are small in a situation where two targets with an utterance probability are present and positions of the two targets are close to each other and the two targets speak simultaneously. ### Claim 9 An information processing method of performing an information analysis process in an information processing apparatus, the method comprising: receiving, by a plurality of information input units, an input of observation information of a real space; generating, by an event detection unit, event information including estimated position information and estimated identification information of users present in the real space based on analysis of the information input from the information input unit; and receiving, by an information integration processing unit, an input of the event, and generating target information including a position of each user and user identification information on the basis of the input event information, and signal information representing a probability value of the event generation source, wherein in the generating of the target information, a process of calculating an utterance source score as an index value representing an utterance source probability of each target by multiplying weights by a plurality of different information items input in the generating of the event information is performed, wherein the weights are adjusted based on whether positions of two targets are close to each other or positions of the two targets are far. ### Claim 10 A non-transitory computer-readable medium, having stored thereon, a computer program having at least one code section executable by a computer, thereby causing the computer to perform the steps comprising: receiving, by a plurality of information input units, an input of observation information of a real space; generating, by an event detection unit, event information including estimated position information and estimated identification information of users present in the real space based on analysis of the information input from the information input unit; and receiving, by an information integration processing unit, an input of the event, and generating target information including a position of each user and user identification information on the basis of the input event information, and signal information representing a probability value of the event generation source, wherein in the generating of the target information, a process of calculating an utterance source score as an index value representing an utterance source probability of each target by multiplying weights by a plurality of different information items input in the generating of the event information is performed, wherein the weights are adjusted based on whether positions of two targets are close to each other or positions of the two targets are far.