# US 12,222,421 B2 · Object recognition apparatus, object recognition method, and non-transitory computer readable medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,421 B2 - Document ID: US12222421B2 - Status: Granted - Kind: B2 - Application number: 17/163,662 - Filed: 2021-02-01 - Published: 2021-06-17 - Granted: 2025-02-11 - Earliest claimed: 2018-12-06 - Assignees: Panasonic Intellectual Property Management Co., Ltd. - Applicants: Panasonic Intellectual Property Management Co., Ltd. - Inventors: Yumiko Kato; Satoshi Sato; Yoshihide Sawada - Web page: https://patentgrep.com/patent/US12222421B2 ## CPC - G01S 17/89: for mapping or imaging - G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section} - G01S 7/4816: {of receivers alone} - G01S 17/10: using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32) - G05B 13/0265: {the criterion being a learning criterion} - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2413: based on distances to training or reference patterns - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06V 10/141: Control of illumination - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06F 3/147: using display panels - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10152: Varying illumination - G06T 2207/20081: Training; Learning - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An object recognition apparatus includes a light source, an image sensor, a control circuit, and a signal processing circuit. The control circuit causes the light source to emit first light toward a scene and subsequently emit second light toward the scene, the first light having a first spatial distribution, the second light having a second spatial distribution. The control circuit causes the image sensor to detect first reflected light and second reflected light in the same exposure period, the first reflected light being caused by reflection of the first light from the scene, the second reflected light being caused by reflection of the second light from the scene. The signal processing circuit recognizes an object included in a scene based on photodetection data output from the image sensor, and based on an object recognition model pre-trained by a machine learning algorithm. ## Claims ### Claim 1 An object recognition apparatus comprising: a light source; a photodetector device including a plurality of photodetector elements, the photodetector device outputting photodetection data responsive to an amount of light that is incident on each of the plurality of photodetector elements during a specified exposure period; a control circuit that controls the light source and the photodetector device, the control circuit causing the light source to emit first light toward a scene and subsequently emit second light toward the scene, the first light having a first spatial distribution, the second light having a second spatial distribution, the control circuit causing the photodetector device to detect first reflected light and second reflected light, the first reflected light being caused by reflection of the first light from the scene, the second reflected light being caused by reflection of the second light from the scene, and the control circuit determining emission timings of the first light and the second light and an exposure timing of the photodetector device such that at least a portion of the plurality of photodetector elements of the photodetector device detects only a part of, but not all of the first reflected light and only a part of, but not all of the second reflected light in a same exposure period; and a signal processing circuit that recognizes an object included in the scene based on the photodetection data output by the photodetector device and based on an object recognition model pre-trained by a machine learning algorithm, derives a distance to the object based on the photodetection data, and outputs information representing the object, and range information representing the distance. ### Claim 2 (depends on 1) The object recognition apparatus according to claim 1, wherein the at least portion of the plurality of photodetector elements of the photodetector device includes a plurality of first photodetector elements that receive the first reflected light and do not receive the second reflected light, and a plurality of second photodetector elements that receive the second reflected light and do not receive the first reflected light, and wherein the plurality of photodetector elements further include a plurality of third photodetector elements that receive neither the first reflected light nor the second reflected light. ### Claim 3 (depends on 1) The object recognition apparatus according to claim 1, wherein: the object recognition model is pre-trained with a plurality of training data sets, the plurality of training data sets each includes learning photodetection data and label data, the learning photodetection data are photodetection data used for learning, and the label data identify a target object included in scenes represented by the learning photodetection data. ### Claim 4 (depends on 3) The object recognition apparatus according to claim 3, wherein the learning photodetection data is photodetection data previously output from the photodetector device. ### Claim 5 (depends on 3) The object recognition apparatus according to claim 3, wherein the control circuit determines a direction of emission of the first light and a direction of emission of the second light based on projection pattern information, the projection pattern information being stored on a storage medium and representing the first spatial distribution and the second spatial distribution, and wherein the learning photodetection data is generated based on the projection pattern information, and based on range image data prepared in advance. ### Claim 6 (depends on 3) The object recognition apparatus according to claim 3, wherein the control circuit determines a direction of emission of the first light and a direction of emission of the second light based on projection pattern information, the projection pattern information being stored on a storage medium and representing the first spatial distribution and the second spatial distribution, and wherein the label data is generated based on the projection pattern information, and based on photodetection data previously output from the photodetector device. ### Claim 7 (depends on 3) The object recognition apparatus according to claim 3, wherein the control circuit determines a direction of emission of the first light and a direction of emission of the second light based on projection pattern information, the projection pattern information being stored on a storage medium and representing the first spatial distribution and the second spatial distribution, and wherein the projection pattern information is learned in advance by the machine learning algorithm using the plurality of training data sets. ### Claim 8 (depends on 1) The object recognition apparatus according to claim 1, wherein the control circuit determines a direction of emission of the first light and a direction of emission of the second light based on one of a plurality of kinds of projection pattern information, the plurality of kinds of projection pattern information being stored on a storage medium and representing the first spatial distribution and the second spatial distribution, and wherein the object recognition model includes a plurality of models corresponding one to-one to the plurality of kinds of projection pattern information. ### Claim 9 (depends on 1) The object recognition apparatus according to claim 1, further comprising a display device, the display device outputting a composite image, the composite image combining an image highlighting the recognized object with an image representing the range information. ### Claim 10 (depends on 1) The object recognition apparatus according to claim 1, wherein the distance is derived based on a range-information generation model, the range-information generation model being pre-trained by the machine learning algorithm or by another machine learning algorithm. ### Claim 11 (depends on 1) The object recognition apparatus according to claim 1, wherein the second spatial distribution differs from the first spatial distribution. ### Claim 12 (depends on 1) The object recognition apparatus according to claim 1, wherein the first light includes a plurality of first light beams each emitted in a different direction, and wherein the second light includes a plurality of second light beams each emitted in a different direction. ### Claim 13 (depends on 1) The object recognition apparatus according to claim 1, wherein the second light has a maximum range shorter than a maximum range of the first light. ### Claim 14 (depends on 1) The object recognition apparatus according to claim 1, wherein the second light has an energy per unit area on an irradiated surface lower than an energy of the first light per unit area on the irradiated surface, the irradiated surface being spaced apart from the light source. ### Claim 15 (depends on 1) The object recognition apparatus according to claim 1, wherein the first light has a first wavelength, wherein the second light has a second wavelength, and wherein the second wavelength has an atmospheric absorption coefficient higher than an atmospheric absorption coefficient of the first wavelength. ### Claim 16 (depends on 1) The object recognition apparatus according to claim 1, wherein the control circuit causes the light source to emit third light toward the scene after emission of the second light, the third light having a third spatial distribution, and causes the photodetector device to detect third reflected light in the same exposure period, the third reflected light being caused by reflection of the third light from the scene, and wherein the third light has a maximum range shorter than a maximum range of the second light. ### Claim 17 (depends on 1) The object recognition apparatus according to claim 1, wherein the same exposure period does not include a time at which a portion of the first reflected light arrives at the photodetector device, the portion of the first reflected light being caused by reflection at a position in the scene located at a first distance from the photodetector device, includes a time at which another portion of the first reflected light arrives at the photodetector device, the other portion of the first reflected light being caused by reflection at a position located at a second distance from the photodetector device, the second distance being greater than the first distance, and includes a time at which a portion of the second reflected light arrives at the photodetector device, the portion of the second reflected light being caused by reflection at a position located at the first distance from the photodetector device. ### Claim 18 An object recognition method comprising: causing a light source to emit first light toward a scene and subsequently emit second light toward the scene, the first light having a first spatial distribution, the second light having a second spatial distribution; determining emission timings of the first light and the second light and an exposure timing of a photodetector device such that at least a portion of a plurality of photodetector elements of the photodetector device detects only a part of, but not all of first reflected light and only a part of, but not all of second reflected light in a same exposure period, the first reflected light being caused by reflection of the first light from the scene, the second reflected light being caused by reflection of the second light from the scene; causing the at least the portion of the plurality of photodetector elements of the photodetector device to detect only the part of, but not all of the first reflected light and only the part of, but not all of the second reflected light in the same exposure period; recognizing an object included in the scene based on photodetection data and an object recognition model, the photodetection data being output from the photodetector device, the object recognition model being pre-trained by a machine learning algorithm; deriving a distance to the object based on the photodetection data; and outputting information representing the object, and range information representing the distance. ### Claim 19 A non-transitory computer-readable medium having a program stored thereon, the program causing a computer to execute: causing a light source to emit first light toward a scene and subsequently emit second light toward the scene, the first light having a first spatial distribution, the second light having a second spatial distribution; determining emission timings of the first light and the second light and an exposure timing of a photodetector device such that at least a portion of a plurality of photodetector elements of the photodetector device detects only a part of, but not all of first reflected light and only a part of, but not all of second reflected light in a same exposure period, the first reflected light being caused by reflection of the first light from the scene, the second reflected light being caused by reflection of the second light from the scene; causing the at least the portion of the plurality of photodetector elements of the photodetector device to detect only the part of, but not all of the first reflected light and only the part of, but not all of the second reflected light in the same exposure period; recognizing an object included in the scene based on photodetection data and an object recognition model, the photodetection data being output from the photodetector device, the object recognition model being pre-trained by a machine learning algorithm; deriving a distance to the object based on the photodetection data; and outputting information representing the object, and range information representing the distance.