# US 11,113,566 B2 · Image processing system, method, and apparatus specifying an object included in a captured image outside a vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,566 B2 - Document ID: US11113566B2 - Status: Granted - Kind: B2 - Application number: 16/506,122 - Filed: 2019-07-09 - Published: 2020-03-26 - Granted: 2021-09-07 - Earliest claimed: 2018-09-21 - Assignees: Toyota Jidosha Kabushiki Kaisha - Applicants: Toyota Jidosha Kabushiki Kaisha - Inventors: Kazuya Nishimura; Yoshihiro Oe; Hirofumi Kamimaru - Web page: https://patentgrep.com/patent/US11113566B2 ## CPC - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/24: Classification techniques - G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An image processing system includes an image acquisition unit that acquires a captured image obtained by imaging a vehicle outside, a dictionary storage unit that stores dictionary data to be referred to in specifying an object included in the captured image, a specification unit that specifies the object based on the dictionary data, a behavior information acquisition unit that acquires behavior information indicating a behavior state of a vehicle, and a classification unit that classifies, based on the behavior information of the vehicle with respect to an unspecifiable object as the object unspecified by the specification unit, whether or not the vehicle needs to avoid the unspecifiable object. Image data of the unspecifiable object is used for creating the dictionary data along with a classification result of the classification unit. ## Claims ### Claim 1 An image processing system comprising: an image acquisition unit configured to acquire a captured image obtained by imaging a vehicle outside; a dictionary storage memory configured to store dictionary data to be referred to in specifying an object included in the captured image; a specification unit configured to specify the object based on the dictionary data, the specification unit determining the object to be an unspecifiable object when the object cannot be specified based on the dictionary data; a behavior information acquisition unit configured to acquire behavior information indicating a behavior state of the vehicle; and a classification unit configured to, based on the behavior information of the vehicle with respect to the unspecifiable object, classify whether or not the vehicle needs to avoid the unspecifiable object based on whether the vehicle shows a behavior for (i) avoiding the unspecifiable object or (ii) passing through the unspecifiable object without avoiding the unspecifiable object, wherein image data of the unspecifiable object along with a classification result of the classification unit are used for creating the dictionary data for the unspecifiable object. ### Claim 2 (depends on 1) The image processing system according to claim 1, wherein the classification unit is configured to classify the image data of the unspecifiable object into either a first image group as a group of image data including objects that the vehicle needs to avoid, or a second image group as a group of image data including objects that the vehicle does not need to avoid. ### Claim 3 (depends on 2) The image processing system according to claim 2, wherein: the classification unit is configured to classify images of objects included in the image data classified into the first image group into a plurality of first analogous image groups analogous to one another and classify images of objects included in the image data classified into the second image group into a plurality of second analogous image groups analogous to one another; and in the dictionary data, data for specifying a new object generated for each of the first analogous image group and the second analogous image group is added. ### Claim 4 (depends on 1) The image processing system according to claim 1, wherein: the image acquisition unit is configured to acquire the captured images from a plurality of vehicles; the behavior information acquisition unit is configured to acquire the behavior information from the vehicles; and the classification unit is configured to classify whether or not the vehicle needs to avoid the unspecifiable object according to an avoidance proportion of the vehicles derived based on the behavior information of the vehicles with respect to the unspecifiable object. ### Claim 5 (depends on 1) The image processing system according to claim 1, further comprising a learning unit configured to create new dictionary data based on the classification result of the classification unit. ### Claim 6 An image processing method comprising: acquiring a captured image of a vehicle outside; storing dictionary data to be referred to in specifying an object included in the captured image; specifying the object based on the dictionary data, the object being determined to be an unspecifiable object when the object cannot be specified based on the dictionary data; acquiring behavior information indicating a behavior state of the vehicle; based on the behavior information of the vehicle with respect to the unspecifiable object, classifying whether or not the vehicle needs to avoid the unspecifiable object based on whether the vehicle shows a behavior for (i) avoiding the unspecifiable object or (ii) passing through the unspecifiable object without avoiding the unspecifiable object; and creating dictionary data for the unspecifiable object based on image data of the unspecifiable object and a result of the classifying. ### Claim 7 An image processing apparatus comprising: an image acquisition unit configured to acquire a captured image obtained by imaging a vehicle outside; a dictionary storage memory configured to store dictionary data to be referred to in specifying an object included in the captured image; and an information processing apparatus configured to acquire behavior information indicating a behavior state of the vehicle, specify the object based on the captured image and the dictionary data, the object being determined to be an unspecifiable object when the object cannot be specified based on the dictionary data, classify, using the behavior information, whether or not the vehicle needs to avoid the unspecifiable object based on whether the vehicle shows a behavior for (i) avoiding the unspecifiable object or (ii) passing through the unspecifiable object without avoiding the unspecifiable object, and create new dictionary data for the unspecifiable object based on a result of the classifying.