# US 12,111,409 B2 · Image processing apparatus, system, image processing method, and image processing program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,111,409 B2 - Document ID: US12111409B2 - Status: Granted - Kind: B2 - Application number: 17/635,304 - Filed: 2019-08-28 - Published: 2022-09-29 - Granted: 2024-10-08 - Earliest claimed: 2019-08-28 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Makoto Koizumi - Web page: https://patentgrep.com/patent/US12111409B2 ## CPC - G01S 5/20: Position of source determined by a plurality of spaced direction-finders - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G01S 5/0294: {Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - H04N 23/60: Control of cameras or camera modules - H04R 1/406: {microphones} - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - G01S 5/22: Position of source determined by co-ordinating a plurality of position lines defined by path-difference measurements (G01S5/28 takes precedence) - H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04S: STEREOPHONIC SYSTEMS ## Abstract An image processing apparatus includes a first reception section, a second reception section, an association processing section, an object detection section, and a process execution section. The first reception section receives image information acquired by an image sensor. The second reception section receives sound information that is acquired by one or plural directional microphones and that is generated for at least a partial region in a field of the image sensor. The association processing section associates the sound information with a pixel address of the image information indicating a position in the field. The object detection section detects, from the image information, at least a part of an object that is present in the field. The process execution section executes a predetermined process on the object on the basis of a result of the association performed by the association processing section. ## Claims ### Claim 1 An image processing apparatus comprising: a first reception section configured to receive image information acquired by an image sensor; a second reception section configured to receive sound information that is acquired by one or plural directional microphones and that is generated for at least a partial region in a field of the image sensor; an association processing section configured to associate the sound information with a pixel address of the image information indicating a position in the field; an object detection section configured to detect, from the image information, at least a part of an object that is present in the field; and a process execution section configured to execute a predetermined process on the object on a basis of a result of the association performed by the association processing section, wherein the predetermined process is a function of whether or not the position in the field at which the sound information is associated overlaps with an area within the field at which the object is located, and wherein the process execution section operates to distinguish between determinations that: (i) the object appears to be moving and is actually moving in the field based on a determination that the sound information indicates an acoustic level above a predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located; and (ii) the object appears to be moving and is not actually moving in the field based on a determination that the sound information indicates an acoustic level not above the predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the second reception section receives the sound information associated with position information indicating the position in the field, and the association processing section associates the sound information with the pixel address by using a result of calibration on the position information and the pixel address. ### Claim 3 (depends on 1) The image processing apparatus according to claim 1, wherein the association processing section associates the sound information with the pixel address corresponding to a region in an image in which the object that has been detected by the object detection section is present, and the process execution section executes the predetermined process on the object corresponding to the pixel address with which the sound information has been associated. ### Claim 4 (depends on 1) The image processing apparatus according to claim 1, wherein the object detection section detects the object in a region in an image that is determined according to the sound information associated with the pixel address, and the process execution section executes the predetermined process on the object that has been detected by the object detection section. ### Claim 5 (depends on 3) The image processing apparatus according to claim 3, further comprising: an object classification section that classifies the object detected by the object detection section into a first object or a second object, according to the sound information associated with the pixel address corresponding to the region in the image in which the object is present, wherein the process execution section executes the predetermined process on the object that is classified as the first object. ### Claim 6 (depends on 1) The image processing apparatus according to claim 1, wherein the sound information includes information indicating whether or not sound has been detected, and the process execution section executes the predetermined process on the object detected with the pixel address associated with the sound information indicating that sound has been detected. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the process execution section executes a tracking process with use of a result of time-series detection of the object. ### Claim 8 (depends on 1) The image processing apparatus according to claim 1, wherein the sound information includes information indicating whether or not sound has been detected, and the process execution section executes the predetermined process on the object detected with the pixel address that has been associated with the sound information indicating that no sound has been detected or that has not been associated with the sound information indicating that sound has been detected. ### Claim 9 (depends on 8) The image processing apparatus according to claim 8, wherein the process execution section executes a self-position estimation process for an apparatus on which the image sensor is mounted, with use of a result of time-series detection of the object. ### Claim 10 (depends on 8) The image processing apparatus according to claim 8, wherein the process execution section executes a motion cancelling process on an apparatus on which the image sensor is mounted, with use of a result of time-series detection of the object. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the process execution section extracts, from the image information, image information including only the object. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein the image sensor is an event-driven vision sensor that generates an event signal upon detecting an intensity change in incident light, for each pixel, and the image information includes the event signal. ### Claim 13 A system comprising: an image sensor configured to acquire image information; one or plural directional microphones configured to acquire sound information that is generated for at least a partial region in a field of the image sensor; and a terminal apparatus including a first reception section configured to receive the image information, a second reception section configured to receive the sound information, an association processing section configured to associate the sound information with a pixel address of the image information indicating a position in the field, an object detection section configured to detect, from the image information, at least a part of an object that is present in the field, and a process execution section configured to execute a predetermined process on the object, on a basis of a result of the association performed by the association processing section, wherein the predetermined process is a function of whether or not the position in the field at which the sound information is associated overlaps with an area within the field at which the object is located, and wherein the process execution section operates to distinguish between determinations that: (i) the object appears to be moving and is actually moving in the field based on a determination that the sound information indicates an acoustic level above a predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located; and (ii) the object appears to be moving and is not actually moving in the field based on a determination that the sound information indicates an acoustic level not above the predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located. ### Claim 14 An image processing method comprising: receiving image information acquired by an image sensor; receiving sound information that is acquired by one or plural directional microphones and that is generated for at least a partial region in a field of the image sensor; associating the sound information with a pixel address of the image information indicating a position in the field; detecting, from the image information, at least a part of an object that is present in the field; and executing a predetermined process on the object on a basis of a result of the association, wherein the predetermined process is a function of whether or not the position in the field at which the sound information is associated overlaps with an area within the field at which the object is located, and wherein the executing includes distinguishing between determinations that: (i) the object appears to be moving and is actually moving in the field based on a determination that the sound information indicates an acoustic level above a predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located; and (ii) the object appears to be moving and is not actually moving in the field based on a determination that the sound information indicates an acoustic level not above the predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located. ### Claim 15 A non-transitory, computer readable storage medium containing a program, which when executed by a computer, causes the computer to perform an image processing method by carrying out actions, comprising: receiving image information acquired by an image sensor; receiving sound information that is acquired by one or plural directional microphones and that is generated for at least a partial region in a field of the image sensor; associating the sound information with a pixel address of the image information indicating a position in the field; detecting, from the image information, at least a part of an object that is present in the field; and executing a predetermined process on the object on a basis of a result of the association, wherein the predetermined process is a function of whether or not the position in the field at which the sound information is associated overlaps with an area within the field at which the object is located, and wherein the executing includes distinguishing between determinations that: (i) the object appears to be moving and is actually moving in the field based on a determination that the sound information indicates an acoustic level above a predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located; and (ii) the object appears to be moving and is not actually moving in the field based on a determination that the sound information indicates an acoustic level not above the predetermined threshold and the position in the field that is associated overlaps with the area within the field at which the object is located.