# US 10,373,323 B1 · Method and device for merging object detection information detected by each of object detectors corresponding to each camera nearby for the purpose of collaborative driving by using V2X-enabled applications, sensor fusion via multiple vehicles > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,323 B1 - Document ID: US10373323B1 - Status: Granted - Kind: B1 - Application number: 16/260,244 - Filed: 2019-01-29 - Granted: 2019-08-06 - Earliest claimed: 2019-01-29 - Assignees: StradVision, Inc. - Applicants: Stradvision, Inc. - Inventors: Kye-Hyeon Kim; Yongjoong Kim; Insu Kim; Hak-Kyoung Kim; Woonhyun Nam; SukHoon Boo; Myungchul Sung; Donghun Yeo; Wooju Ryu; Taewoong Jang; Kyungjoong Jeong; Hongmo Je; Hojin Cho - Web page: https://patentgrep.com/patent/US10373323B1 ## CPC - G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)} - G06F 18/254: {of classification results, e.g. of results related to same input data} - G06T 7/32: using correlation-based methods - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - H04W 4/40: for vehicles, e.g. vehicle-to-pedestrians [V2P] - B60R 2300/303: using joined images, e.g. multiple camera images - B60R 2300/50: characterised by the display information being shared, e.g. external display, data transfer to other traffic participants or centralised traffic controller - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30261: Obstacle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR ## Abstract A method for merging object detection information detected by object detectors, each of which corresponds to each of cameras located nearby, by using V2X-based auto labeling and evaluation, wherein the object detectors detect objects in each of images generated from each of the cameras by image analysis based on deep learning is provided. The method includes steps of: if first to n-th object detection information are respectively acquired from a first to an n-th object detectors in a descending order of degrees of detection reliabilities, a merging device generating (k-1)-th object merging information by merging (k-2)-th objects and k-th objects through matching operations, and re-projecting the (k-1)-th object merging information onto an image, by increasing k from 3 to n. The method can be used for a collaborative driving or an HD map update through V2X-enabled applications, sensor fusion via multiple vehicles, and the like. ## Claims ### Claim 1 A method for merging object detection information detected by object detectors, each of which corresponds to each of cameras located nearby, wherein the object detectors detect objects in each of images generated from each of the cameras by image analysis based on deep learning, comprising steps of: (a) a specific merging device, corresponding to a specific object detector, among merging devices corresponding to the object detectors, if object detection information is acquired from the object detectors, each of which corresponds to each of the cameras located nearby, determining an order of degrees of each set of detection reliabilities of each of the object detectors by referring to the object detection information, wherein each piece of the object detection information includes one or more primary objects corresponding to one or more image-taking bodies whose corresponding cameras take each of the images and one or more secondary objects corresponding to one or more image-taken bodies appearing on at least part of the images generated by each of the cameras; and (b) the specific merging device, supposing that at least part of the object detectors are designated as a first object detector to an n-th object detector in a descending order of the degrees of each set of the detection reliabilities and that each piece of the object detection information acquired from each of the first object detector to the n-th object detector is designated as first object detection information to n-th object detection information, (i) confirming correspondence between first total objects in the first object detection information and second total objects in the second object detection information via at least one matching operation, by referring to the first object detection information and the second object detection information and generating first object merging information by merging the first total objects and the second total objects (ii) confirming correspondence between (k-2)-th objects included in the (k-2)-th object merging information and k-th objects in the k-th object detection information through the matching operation by referring to (k-2)-th object merging information and k-th object detection information, generating (k-1)-th object merging information by merging the (k-2)-th objects and the k-th objects, and re-projecting the (k-1)-th object merging information onto at least one specific image acquired from at least one specific camera corresponding to the specific object detector, by increasing k from 3 to n, wherein the first total objects include the primary objects and the secondary objects, and the second total objects include the primary objects and the secondary objects. ### Claim 2 (depends on 1) The method of claim 1, wherein the method further comprises a step of: (c) the specific merging device updating a specific detection reliability of the specific object detector by referring to a degree of similarity between the (k-1)-th object merging information and specific object detection information of the specific object detector. ### Claim 3 (depends on 1) The method of claim 1, wherein the method further comprises a step of: (d) the specific merging device auto-labeling (k-1)-th objects in the (k-1)-th object merging information, and learning at least part of one or more parameters of the specific object detector by referring to labels generated by the auto-labeling. ### Claim 4 (depends on 1) The method of claim 1, wherein each piece of the object detection information includes information on each of the primary objects and information on each of the secondary objects, wherein the information on each of the primary objects includes status information on each of the primary objects and each set of the detection reliabilities of each of the object detectors corresponding to each of the primary objects, wherein the status information on each of the primary objects includes current location information, current direction information, and current speed information of each of the primary objects, and wherein the information on each of the secondary objects includes status information on each of the secondary objects and product information on each of the secondary objects wherein the status information on each of the secondary objects includes current location information, current direction information, and current speed information of each of the secondary object. ### Claim 5 (depends on 4) The method of claim 4, wherein the information on each of the primary objects includes one of product information on each of the primary objects and each piece of product information of each of the object detectors corresponding to each of the primary objects. ### Claim 6 (depends on 5) The method of claim 5, wherein, at the step of (a), the specific merging device performs one of processes of (i) determining the order of degrees of each set of the detection reliabilities of each of the object detectors using the received detection reliabilities, (ii) determining the order of degrees of each set of the detection reliabilities of each of the object detectors by referring to preset first reliabilities corresponding to each piece of the product information of each of the object detectors, and (iii) determining the order of degrees of each set of the detection reliabilities of each of the object detectors by referring to a preset second reliability set as a base reliability if no feature information of the object detectors is acquired. ### Claim 7 (depends on 1) The method of claim 1, wherein, supposing that first total objects corresponding to one of the first object detection information and the (k-2)-th object merging information are designated as A1 to Ai and that second total objects corresponding to one of the second object detection information and the k-th object detection information are designated as B1 to Bj, wherein the first total objects include the primary objects and the secondary objects, and the second total objects include the primary objects and the secondary objects, at the step of (b), the specific merging device calculates cost of correspondence between said A1 to Ai and said B1 to Bj by the matching operation, and merges said A1 to Ai and said B1 to Bj according to the calculated cost. ### Claim 8 (depends on 7) The method of claim 7, wherein the specific merging device calculates the cost of correspondence between said A1 to Ai and said B1 to Bj by using either any one of following (i) to (v) or a weighted sum of at least two of said (i) to (v), wherein said (i) represents one minus a first IoU which is an intersection over union of three dimensional cubes between one of said A1 to Ai and one of said B1 to Bj, wherein said (ii) represents one minus a second IoU which is an intersection over union of two dimensional rectangles resulting from projecting three dimensional cubes including one of said A1 to Ai and one of said B1 to Bj on a two dimensional plane from a bird's eye view, wherein said (iii) represents a Euclidean distance between centers of mass of one of said A1 to Ai and one of said B1 to Bj, wherein said (iv) represents an angle between one of said A1 to Ai and one of said B1 to Bj, and wherein said (v) represents a difference of speeds between one of said A1 to Ai and one of said B1 to Bj. ### Claim 9 (depends on 7) The method of claim 7, wherein the specific merging device adds first undetected objects corresponding to either the first object detection information or the (k-2)-th object merging information by an amount of C1 to Cj such that the first undetected objects correspond to said B1 to Bj, and adds second undetected objects corresponding to either the second object detection information or the k-th object detection information by an amount of D1 to Di such that the second undetected objects correspond to said A1 to Ai, and wherein the specific merging device calculates cost of correspondence between (i) said A1 to Ai and said C1 to Cj and (ii) said B1 to Bj and said D1 to Di, by the matching operation, and merges (i) said A1 to Ai and said C1 to Cj and (ii) said B1 to Bj and said D1 to Di. ### Claim 10 (depends on 9) The method of claim 9, wherein the specific merging device determines (i) cost of correspondence between (i-1) one of said C1 to Cj and (i-2) one of said B1 to Bj and said D1 to Di and (ii) cost of correspondence between (ii-1) one of said D1 to Di and (ii-2) one of said A1 to Ai and said C1 to Cj, as preset initial values. ### Claim 11 (depends on 7) The method of claim 7, wherein, at the step of (b), the specific merging device, during a process of merging said A1 to Ai and said B1 to Bj according to the calculated cost, (i) merges status information on the first and the second total objects into a weighted sum with weights being the detection reliabilities corresponding to each of the first and the second total objects, and (ii) merges product information of a certain object detector whose sum of the detection reliabilities is a highest to product information on the first and the second total objects. ### Claim 12 A specific merging device, corresponding to a specific object detector, among merging devices for merging object detection information detected by object detectors, each of which corresponds to each of cameras located nearby, wherein the object detectors detect objects in each of images generated from each of the cameras by image analysis based on deep learning, comprising: at least one memory that stores instructions; and at least one processor configured to execute the instructions to: perform processes of: (I) if object detection information is acquired from the object detectors, each of which corresponds to each of the cameras located nearby, determining an order of degrees of each set of detection reliabilities of each of the object detectors by referring to the object detection information, wherein each piece of the object detection information includes one or more primary objects corresponding to one or more image-taking bodies whose corresponding cameras take each of the images and one or more secondary objects corresponding to one or more image-taken bodies appearing on at least part of the images generated by each of the cameras, and (II) supposing that at least part of the object detectors are designated as a first object detector to an n-th object detector in a descending order of the degrees of each set of the detection reliabilities and that each piece of the object detection information acquired from each of the first object detector to the n-th object detector is designated as first object detection information to n-th object detection information, (i) confirming correspondence between first total objects in the first object detection information and second total objects in the second object detection information via at least one matching operation, by referring to the first object detection information and the second object detection information and generating first object merging information by merging the first total objects and the second total objects (ii) confirming correspondence between (k-2)-th objects included in the (k-2)-th object merging information and k-th objects in the k-th object detection information through the matching operation by referring to (k-2)-th object merging information and k-th object detection information, generating (k-1)-th object merging information by merging the (k-2)-th objects and the k-th objects, and re-projecting the (k-1)-th object merging information onto at least one specific image acquired from at least one specific camera corresponding to the specific object detector, by increasing k from 3 to n, wherein the first total objects include the primary objects and the secondary objects, and the second total objects include the primary objects and the secondary objects. ### Claim 13 (depends on 12) The specific merging device of claim 12, wherein the processor further performs a process of: (III) updating a specific detection reliability of the specific object detector by referring to a degree of similarity between the (k-1)-th object merging information and specific object detection information of the specific object detector. ### Claim 14 (depends on 12) The specific merging device of claim 12, wherein the processor further performs a process of: (IV) auto-labeling (k-1)-th objects in the (k-1)-th object merging information, and learning at least part of one or more parameters of the specific object detector by referring to labels generated by the auto-labeling. ### Claim 15 (depends on 12) The specific merging device of claim 12, wherein each piece of the object detection information includes information on each of the primary objects and information on each of the secondary objects, wherein the information on each of the primary objects includes status information on each of the primary objects and each set of the detection reliabilities of each of the object detectors corresponding to each of the primary objects, wherein the status information on each of the primary objects includes current location information, current direction information, and current speed information of each of the primary objects, and wherein the information on each of the secondary objects includes status information on each of the secondary objects and product information on each of the secondary objects wherein the status information on each of the secondary objects includes current location information, current direction information, and current speed information of each of the secondary object. ### Claim 16 (depends on 15) The specific merging device of claim 15, wherein the information on each of the primary objects includes one of product information on each of the primary objects and each piece of product information of each of the object detectors corresponding to each of the primary objects. ### Claim 17 (depends on 16) The specific merging device of claim 16, wherein, at the process of (I), the processor performs one of processes of (i) determining the order of degrees of each set of the detection reliabilities of each of the object detectors using the received detection reliabilities, (ii) determining the order of degrees of each set of the detection reliabilities of each of the object detectors by referring to preset first reliabilities corresponding to each piece of the product information of each of the object detectors, and (iii) determining the order of degrees of each set of the detection reliabilities of each of the object detectors by referring to a preset second reliability set as a base reliability if no feature information of the object detectors is acquired. ### Claim 18 (depends on 12) The specific merging device of claim 12, wherein, supposing that first total objects corresponding to one of the first object detection information and the (k-2)-th object merging information are designated as A1 to Ai and that second total objects corresponding to one of the second object detection information and the k-th object detection information are designated as B1 to Bj, wherein the first total objects include the primary objects and the secondary objects, and the second total objects include the primary objects and the secondary objects, at the process of (II), the processor calculates cost of correspondence between said A1 to Ai and said B1 to Bj by the matching operation, and merges said A1 to Ai and said B1 to Bj according to the calculated cost. ### Claim 19 (depends on 18) The specific merging device of claim 18, wherein the processor calculates the cost of correspondence between said A1 to Ai and said B1 to Bj by using either any one of following (i) to (v) or a weighted sum of at least two of said (i) to (v), wherein said (i) represents one minus a first IoU which is an intersection over union of three dimensional cubes between one of said A1 to Ai and one of said B1 to Bj, wherein said (ii) represents one minus a second IoU which is an intersection over union of two dimensional rectangles resulting from projecting three dimensional cubes including one of said A1 to Ai and one of said B1 to Bj on a two dimensional plane from a bird's eye view, wherein said (iii) represents a Euclidean distance between centers of mass of one of said A1 to Ai and one of said B1 to Bj, wherein said (iv) represents an angle between one of said A1 to Ai and one of said B1 to Bj, and wherein said (v) represents a difference of speeds between one of said A1 to Ai and one of said B1 to Bj. ### Claim 20 (depends on 18) The specific merging device of claim 18, wherein the processor adds first undetected objects corresponding to either the first object detection information or the (k-2)-th object merging information by an amount of C1 to Cj such that the first undetected objects correspond to said B1 to Bj, and adds second undetected objects corresponding to either the second object detection information or the k-th object detection information by an amount of D1 to Di such that the second undetected objects correspond to said A1 to Ai, and wherein the processor calculates cost of correspondence between (i) said A1 to Ai and said C1 to Cj and (ii) said B1 to Bj and said D1 to Di, by the matching operation, and merges (i) said A1 to Ai and said C1 to Cj and (ii) said B1 to Bj and said D1 to Di. ### Claim 21 (depends on 20) The specific merging device of claim 20, wherein the processor determines (i) cost of correspondence between (i-1) one of said C1 to Cj and (i-2) one of said B1 to Bj and said D1 to Di and (ii) cost of correspondence between (ii-1) one of said D1 to Di and (ii-2) one of said A1 to Ai and said C1 to Cj, as preset initial values. ### Claim 22 (depends on 18) The specific merging device of claim 18, wherein, at the process of (II), the processor, during a process of merging said A1 to Ai and said B1 to Bj according to the calculated cost, (i) merges status information on the first and the second total objects into a weighted sum with weights being the detection reliabilities corresponding to each of the first and the second total objects, and (ii) merges product information of a certain object detector whose sum of the detection reliabilities is a highest to product information on the first and the second total objects.