# US 9,752,880 B2 · Object linking method, object linking apparatus, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,752,880 B2 - Document ID: US9752880B2 - Status: Granted - Kind: B2 - Application number: 14/989,422 - Filed: 2016-01-06 - Published: 2016-07-14 - Granted: 2017-09-05 - Earliest claimed: 2015-01-09 - Assignees: Fujitsu Limited - Applicants: Fujitsu Limited - Inventors: Bin Chen - Web page: https://patentgrep.com/patent/US9752880B2 ## CPC - G01C 21/1656: {with passive imaging devices, e.g. cameras} - G01P 15/02: by making use of inertia forces {using solid seismic masses}(G01P15/14 takes precedence) - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G06T 2207/30241: Trajectory - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G01P: MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT (measuring or recording blood flow A61B5/02, A61B8/06; monitoring speed or deceleration of electrically-propelled vehicles B60L3/00; vehicle lighting systems adapted to indicate speed B60Q1/54; determining position or course in navigation, measuring ground distance in geodesy or surveying G01C; combined measuring devices for measuring two or more variables of movement G01C23/00; measuring velocity of sound G01H; measuring velocity of light G01J7/00; measuring direction or velocity of solid objects by reception or emission of radiowaves or other waves and based on propagation effects, e.g. Doppler effect, propagation time, direction of propagation, G01S; measuring speed of nuclear radiation G01T; measuring acceleration of gravity G01V; {measuring or recording the speed of trains B61L23/00; speed indicators incorporated in motor vehicles B60K35/00; measuring frequency or phase G01R; traffic control G08G}) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An object linking method including: detecting each first object based on a video image, detecting each second object based on each data measured by each inertial sensor, each first object or each second object having each first state or each second state that is one of states including a moving state and a stopping state, and linking, by a computer, each first object and each second object in one-to-one, based on each first change of each first state and each second change of each second state, wherein when a specified first object of at least one first object is not linked to any of the at least one second object, the specified first object is linked to a virtual second object that is added to the at least one second object. ## Claims ### Claim 1 An object linking method comprising: detecting each of at least one first object based on a video image, each of the at least one first object having each first state that is one of states including a moving state and a stopping state; detecting each of at least one second object based on each data measured by each inertial sensor in each of the at least one second object, each of the at least one second object having each second state that is one of states including a moving state and a stopping state; and linking, by a computer, each of the at least one first object and each of the at least one second object in one-to-one, based on each first change of each first state and each second change of each second state, wherein when a specified first object of the at least one first object is not linked to any of the at least one second object, the specified first object is linked to a virtual second object that is added to the at least one second object, and when a specified second object of the at least one second object is not linked to any of the at least one first object, the specified second object is linked to a virtual first object that is added to the at least one first object. ### Claim 2 (depends on 1) The object linking method according to claim 1, wherein the linking of each of the at least one first object and each of the at least one second object in one-to-one is based on a square matrix, each row of the square matrix corresponding to each of the at least one first object to which the virtual first object is added when each column of the square matrix corresponding to each of the at least one second object to which the virtual second object is added, each column of the square matrix corresponding to each of the at least one first object to which the virtual first object is added when each row of the square matrix corresponding to each of the at least one second object to which the virtual second object is added. ### Claim 3 (depends on 2) The object linking method according to claim 2, further comprising: deleting, when a first object become undetected, the undetected first object and a second object or a virtual second object corresponding to the undetected first object from the square matrix, and deleting, when a second object become undetected, the undetected second object and a first object or a virtual first object corresponding to the undetected second object from the square matrix. ### Claim 4 (depends on 1) The object linking method according to claim 1, wherein the linking of each of the at least one first object and each of the at least one second object in one-to-one is based on each certainty factor that is given to each of combinations of each of the at least one first object and each of the at least one second object based on each first change of each first state and each second change of each second state. ### Claim 5 (depends on 4) The object linking method according to claim 4, wherein each certainty factor of each of event pairs of each first event and each second event is calculated based on a specified probability model, each first event relating to each change of each first state of each first object of each of the event pairs, each second event relating to each change of each second state of each second object of each of the event pairs, each first event and each second event being detected within a specified time window. ### Claim 6 (depends on 5) The object linking method according to claim 5, wherein each certainty factor is calculated based on a first undetected event when a first event is undetected, and each certainty factor is calculated based on a second undetected event when a second event is undetected. ### Claim 7 (depends on 5) The object linking method according to claim 5, wherein the specified probability model is based on each difference between each first time in which each first event is detected and each second time in which each second event is detected. ### Claim 8 (depends on 5) The object linking method according to claim 5, wherein the specified probability model is based on each difference between each first time in which each first event is detected and each second time in which each second event is detected, and when one of a first event and a second event is undetected, a difference of the first event and the second event is obtained based on uniform distribution within the specified time window. ### Claim 9 (depends on 8) The object linking method according to claim 8, wherein the specified probability model includes each probability of each transition of each first event and each second event. ### Claim 10 (depends on 5) The object linking method according to claim 5, wherein the specified probability model estimates each of subsequent first events and each of subsequent second events based on each probability of each transition of each first event and each second event, the subsequent first events include a first undetected event, and the subsequent second events include a second undetected event. ### Claim 11 (depends on 5) The object linking method according to claim 5, further comprising: selecting a first combination of each of object pairs of each first object and each second object from combination candidates of each of the object pairs so that a first sum of certainty factors of the first combination of each of the object pairs is the greatest in the combination candidates, wherein the linking of each of the at least one first object and each of the at least one second object in one-to-one is based on the first combination of each of object pairs. ### Claim 12 (depends on 11) The object linking method according to claim 11, further comprising: selecting a second combination of each of object pairs of each first object and each second object from combination candidates of each of the object pairs so that a second sum of certainty factors of the second combination of each of the object pairs is the second greatest in the combination candidates, wherein the linking of each of the at least one first object and each of the at least one second object in one-to-one is based on the first combination of each of object pairs when a difference between the first sum and the second sum is greater than a specified value. ### Claim 13 (depends on 1) The object linking method according to claim 1, wherein each of the at least one first object is each person, and each of the at least one second object is each device carried by each person. ### Claim 14 An object linking apparatus comprising: a memory; and a processor coupled to the memory and configured to: detect each of at least one first object based on a video image, each of the at least one first object having each first state that is one of states including a moving state and a stopping state, detect each of at least one second object based on each data measured by each inertial sensor in each of the at least one second object, each of the at least one second object having each second state that is one of states including a moving state and a stopping state, and link each of the at least one first object and each of the at least one second object in one-to-one, based on each first change of each first state and each second change of each second state, wherein when a specified first object of the at least one first object is not linked to any of the at least one second object, the specified first object is linked to a virtual second object that is added to the at least one second object, and when a specified second object of the at least one second object is not linked to any of the at least one first object, the specified second object is linked to a virtual first object that is added to the at least one first object. ### Claim 15 A non-transitory computer-readable storage medium storing a program that causes a computer to execute a process, the computer including a memory, the process comprising: detecting each of at least one first object based on a video image, each of the at least one first object having each first state that is one of states including a moving state and a stopping state; detecting each of at least one second object based on each data measured by each inertial sensor in each of the at least one second object, each of the at least one second object having each second state that is one of states including a moving state and a stopping state; and linking each of the at least one first object and each of the at least one second object in one-to-one, based on each first change of each first state and each second change of each second state, wherein when a specified first object of the at least one first object is not linked to any of the at least one second object, the specified first object is linked to a virtual second object that is added to the at least one second object, and when a specified second object of the at least one second object is not linked to any of the at least one first object, the specified second object is linked to a virtual first object that is added to the at least one first object.