# US 12,223,714 B2 · Object classification method, vehicle control method, information display method, and object classification device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,714 B2 - Document ID: US12223714B2 - Status: Granted - Kind: B2 - Application number: 17/169,350 - Filed: 2021-02-05 - Published: 2021-05-27 - Granted: 2025-02-11 - Earliest claimed: 2018-10-15 - Assignees: Panasonic Intellectual Property Management Co., Ltd. - Applicants: Panasonic Intellectual Property Management Co., Ltd. - Inventors: Takamasa Ando - Web page: https://patentgrep.com/patent/US12223714B2 ## CPC - G06V 20/20: in augmented reality scenes - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) - G06V 10/17: {using hand-held instruments} - G06V 10/20: Image preprocessing - G06V 10/58: relating to hyperspectral data - G06V 10/764: using classification, e.g. of video objects - G06V 10/7753: {Incorporation of unlabelled data, e.g. multiple instance learning [MIL]} - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 20/194: {using hyperspectral data, i.e. more or other wavelengths than RGB} - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G01J 2003/2826: {Multispectral imaging, e.g. filter imaging} - G06T 2207/10024: Color image - G06T 2207/10036: Multispectral image; Hyperspectral image - G06T 2207/30248: Vehicle exterior or interior - G06V 20/68: Food, e.g. fruit or vegetables - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N) ## Abstract An object classification method includes: acquiring image data of an image including feature information indicating a feature of an object; and classifying the object included in the image, based on the feature information. The image data is acquired by causing a first image capture device to capture the image. The first image capture device includes: an image sensor; and a filter array that is arranged on an optical path of light that is incident on the image sensor and that includes translucent filters two-dimensionally arrayed along a plane that crosses the optical path, the translucent filters including two or more filters in which wavelength dependencies of light transmittances are different from each other, and light transmittance of each of the two or more filters having local maximum values in a plurality of wavelength ranges. ## Claims ### Claim 1 An object classification method comprising: acquiring image data of an image including feature information indicating a feature of an object; and classifying the object included in the image, based on the feature information, by applying a classification model trained with a machine-learning algorithm to the image data that is data of a multiplexed image in which a plurality of wavelength information are multiplexed in one pixel, wherein: the classification model is trained to output a classification result without reconstructing individual images in the plurality of wavelength information from the image data, the image data is acquired by causing a first image capture device to capture the image, the first image capture device including: an image sensor including a pixel; and a filter array that is arranged between the object and the image sensor and includes translucent filters two-dimensionally arrayed, the image sensor detecting light transmitted through the filter array, the translucent filters including two or more filters in which wavelength dependencies of light transmittances are different from each other, light transmittance of each of the two or more filters having local maximum values in a plurality of wavelength ranges, and light detected by the pixel and converted to the image data has the plurality of wavelength ranges corresponding to the local maximum values. ### Claim 2 (depends on 1) The object classification method according to claim 1, wherein the classification model has trained with first training datasets, each of the first training datasets including learning image data for learning and label data for identifying the object included in a learning image for learning, the learning image being indicated by the learning image data for learning. ### Claim 3 (depends on 2) The object classification method according to claim 2, further comprising: causing the classification model to perform further learning with a second training dataset including the image data and second label data for identifying the object. ### Claim 4 (depends on 2) The object classification method according to claim 2, wherein the image data for learning is acquired by capturing an image of the object in a state in which the object occupies a predetermined range or more in the image for learning. ### Claim 5 (depends on 2) The object classification method according to claim 2, wherein pieces of learning image data for learning which are included in the first training datasets include learning image data for learning which is generated by a second image capture device that is different from the first image capture device. ### Claim 6 (depends on 5) The object classification method according to claim 5, wherein the second image capture device includes a filter array having a characteristic that is equivalent to a characteristic of the filter array in the first image capture device. ### Claim 7 (depends on 2) The object classification method according to claim 2, wherein: learning images for learning are included in pieces of learning image data for learning which are included in the first training datasets, and positions of the object in the learning images for learning are different from each other in the pieces of learning image data for learning. ### Claim 8 (depends on 1) The object classification method according to claim 1, wherein the acquiring of the image data is performed using an image capture device including a display, and the object classification method further comprises displaying, on the display, an auxiliary representation for notifying a user of an area where the object is to be located or a range to be occupied by the object in the image, before the image data is acquired. ### Claim 9 (depends on 1) The object classification method according to claim 1, wherein the translucent filters are different from each other in the wavelength dependencies of the light transmittances, and the light transmittance of each of the translucent filters has local maximum values in a plurality of wavelength ranges. ### Claim 10 (depends on 1) A vehicle control method using the object classification method according to claim 1, the vehicle control method comprising: controlling an operation of a vehicle, based on a result of the classifying of the object, wherein the first image capture device is attached to the vehicle. ### Claim 11 (depends on 1) An information display method using the object classification method according to claim 1, the information display method comprising: acquiring, from a database, data indicating at least one selected from the group consisting of a name of the object and a description of the object, based on a result of the classifying of the object; and displaying, on a display, the at least one selected from the group consisting of the name of the object and the description of the object. ### Claim 12 An object classification device comprising: an image sensor that generates image data of an image including feature information indicating a feature of an object and includes a pixel; a filter array that is arranged between the object and the image sensor and that includes translucent filters two-dimensionally arrayed, the image sensor detecting light transmitted through the filter array, the translucent filters including two or more filters in which wavelength dependencies of light transmittances are different from each other, light transmittance of each of the two or more filters having local maximum values in a plurality of wavelength ranges; and a signal processing circuit that classifies the object included in the image, based on the feature information, by applying a classification model trained with a machine-learning algorithm to the image data that is data of a multiplexed image in which a plurality of wavelength information are multiplexed in one pixel, wherein: the classification model is trained to output a classification result without reconstructing individual images in the plurality of wavelength information from the image data, and light detected by the pixel and converted to the image data has the plurality of wavelength ranges corresponding to the local maximum values.