# US 2018/0025500 A1 · Method of tracking one or more mobile objects in a site and a system employing same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025500 A1 - Document ID: US20180025500A1 - Status: Published application - Kind: A1 - Application number: 15/656,428 - Filed: 2017-07-21 - Published: 2018-01-25 - Earliest claimed: 2016-07-22 - Assignees: Appropolis Inc. - Applicants: Appropolis Inc. - Inventors: Jorgen S. NIELSEN; Christopher S. NIELSEN; Zhe HE - Web page: https://patentgrep.com/patent/US20180025500A1 ## CPC - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G01S 5/02521: {using a radio-map} - G01S 5/02526: {using non-dedicated equipment, e.g. user equipment or crowd-sourcing} - G01S 5/02528: {Simulating radio frequency fingerprints} - G01S 5/0264: {at least one of the systems being a non-radio wave positioning system} - G01S 5/0278: {involving statistical or probabilistic considerations (G01S5/0252, G01S5/0294 take precedence)} - G01S 5/0294: {Trajectory determination or predictive filtering, e.g. target tracking or Kalman filtering} - G06F 18/24155: {Bayesian classification} - G06F 18/256: {of results relating to different input data, e.g. multimodal recognition} - G06T 7/254: involving subtraction of images - G06T 7/74: {involving reference images or patches} - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - H04N 7/181: {for receiving images from a plurality of remote sources} - G06T 2207/10016: Video; Image sequence - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06T 2207/30241: Trajectory - G06V 2201/10: Recognition assisted with metadata - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A mobile object tracking system has one or more imaging devices for capturing images of a site, one or more reference wireless devices in wireless communication with one or more mobile wireless devices (MWDs) via one or more wireless signals, and one or more received signal strength models (RSSMs) of the site for the wireless signals. Each MWD is associated with a mobile object and movable therewith. The system tracks the mobile objects by combining the captured images, the received signal strength (RSS) observables of the wireless signals, and the RSSMs. The system may calibrate the RSSMs at an initial stage and during mobile object tracking. ## Claims ### Claim 1 A system for tracking at least one mobile object in a site, the system comprising: one or more imaging devices each capturing images of at least a portion of the site; one or more reference wireless devices; one or more mobile wireless devices (MWDs) in wireless communication with the one or more reference wireless devices via one or more wireless signals, each of the one or more MWDs being associated with one of the at least one mobile object, and movable therewith; and at least one processing structure functionally coupled to the one or more imaging devices, the one or more reference wireless devices, and the one or more MWDs, the at least one processing structure acting for: maintaining one or more received signal strength models (RSSMs) of the site for the one or more wireless signals; obtaining received signal strength (RSS) observables of the one or more wireless signals; and tracking the at least one mobile object by combining the captured images, the obtained RSS observables, and the one or more RSSMs. ### Claim 2 (depends on 1) The system of claim 1 wherein each of the one or more RSSMs is any one or a combination of a parametric RSSM and a nonparametric RSSM, and wherein each nonparametric RSSM comprises a radio map of the site for one of the one or more wireless signals. ### Claim 3 (depends on 1) The system of claim 1 wherein said tracking the at least one mobile object comprises: detecting the at least one mobile object based on the captured images; determining a trajectory for each of the detected at least one mobile object based on the captured images; determining a trajectory for each of the one or more MWDs based on the obtained RSSM observables and the one or more RSSMs; and associating each of the one or more MWDs with one of the detected at least one mobile object if a probability of association (PoA) between the trajectory of the MWD and the trajectory of the mobile object is greater than a threshold. ### Claim 4 (depends on 3) The system of claim 3 wherein said determining the trajectory for each of the detected at least one mobile object based on the captured images comprises: determining the trajectory for each of the detected at least one mobile object using a Bayesian model based on the captured images. ### Claim 5 (depends on 3) The system of claim 3 wherein the system comprises a plurality of imaging devices, at least a first and a second ones of the plurality of imaging devices are respectively associated with two main field of view (FOV) areas, and the two main FOV areas are non-overlapped with an estimation area therebetween; and wherein said determining the trajectory for each of the detected at least one mobile object based on the captured images comprises, for each of the detected at least one mobile object: calculating the position of the mobile object by projecting the mobile object from a plane of the captured images into a coordinate system of the site; if the calculated position of the mobile object is within the estimation area, recalculating the position of the mobile object by using a spatial interpolation method; and determining the trajectory for the mobile object. ### Claim 6 (depends on 1) The system of claim 1 wherein the at least one processing structure further acts for: updating the one or more RSSMs using the location of the mobile object and the RSS observables of the MWD, after associating one of the one or more MWDs with one of the at least one mobile object. ### Claim 7 (depends on 6) The system of claim 6 wherein the at least one processing structure further acts for: identifying, from the at least one mobile object and the one or more MWDs, at least one pair of associated mobile object and MWD movable in the site, said association between each of the at least one pair of associated mobile object and MWD being maintained through a calibration period; and building the one or more RSSMs based on the locations of the identified mobile objects and the RSS observables of the identified MWDs. ### Claim 8 (depends on 7) The system of claim 7 wherein said identifying at least one pair of associated mobile object and MWD comprises: allowing only one of the at least one mobile object movable in the site, one of the one or more MWDs being associated with said only one mobile object and movable therewith. ### Claim 9 (depends on 7) The system of claim 7 wherein the at least one processing structure further acts for: moving said identified at least one pair of associated mobile object and MWD in the site along a survey path. ### Claim 10 (depends on 9) The system of claim 9 wherein the site is partitioned into a two-dimensional (2D) matrix of grid points; and wherein the at least one processing structure further acts for: calculating a set of location weights using a Gaussian kernel and the location of the mobile object; calculating a radio-map weight matrix using the calculated set of location weights, each radio-map weight being associated with a location of the site; and calculating a radio-map value matrix using the obtained RSSM observables obtained at the location of the mobile object. ### Claim 11 (depends on 10) The system of claim 10 wherein said updating the one or more RSSMs using the location of the mobile object and the RSS observables of the MWD comprises: updating the one or more RSSMs using the location of the mobile object and the RSS observables of the MWD if the radio-map weight associated with the location of the mobile object is greater than a weight-acceptance threshold. ### Claim 12 A method of tracking at least one mobile object in a site, the method comprising: maintaining one or more received signal strength models (RSSMs) of the site for one or more wireless signals transmitted between one or more mobile wireless devices (MWDs) and one or more reference wireless devices, each of the one or more MWDs being associated with one of the at least one mobile object and movable therewith; obtaining received signal strength (RSS) observables of the one or more wireless signals; capturing images of at least a portion of the site; and tracking the at least one mobile object by combining the captured images, the obtained RSS observables, and the one or more RSSMs. ### Claim 13 (depends on 12) The method of claim 12 wherein each of the one or more RSSMs is any one or a combination of a parametric RSSM and a nonparametric RSSM, and wherein each nonparametric RSSM comprises a radio map of the site for one of the one or more wireless signals. ### Claim 14 (depends on 12) The method of claim 12 wherein said tracking the at least one mobile object comprises: detecting the at least one mobile object based on the captured images; determining a trajectory for each of the detected at least one mobile object based on the captured images; determining a trajectory for each of the one or more MWDs based on the obtained RSSM observables and the one or more RSSMs; and associating each of the one or more MWDs with one of the detected at least one mobile object if a probability of association (PoA) between the trajectory of the MWD and the trajectory of the mobile object is greater than a threshold. ### Claim 15 (depends on 14) The method of claim 14 wherein said determining the trajectory for each of the detected at least one mobile object based on the captured images comprises: determining the trajectory for each of the detected at least one mobile object using a Bayesian model based on the captured images. ### Claim 16 (depends on 14) The method of claim 14 wherein said determining the trajectory for each of the detected at least one mobile object based on the captured images comprises, for each of the detected at least one mobile object: calculating the position of the mobile object by projecting the mobile object from a plane of the captured images into a coordinate system of the site; determining if the calculated position of the mobile object is within an estimation area between two non-overlapped main field of view (FOV) areas; if the calculated position of the mobile object is within the estimation area, recalculating the position of the mobile object by using a spatial interpolation method; and determining a trajectory for the mobile object. ### Claim 17 (depends on 12) The method of claim 12 further comprising: updating the one or more RSSMs using the location of the mobile object and the RSS observables of the MWD, after associating one of the one or more MWDs with one of the at least one mobile object. ### Claim 18 (depends on 17) The method of claim 17 further comprising: identifying, from the at least one mobile object and the one or more MWDs, at least one pair of associated mobile object and MWD movable in the site, said association between each of the at least one pair of associated mobile object and MWD being maintained through a calibration period; and building the one or more RSSMs based on the locations of the identified mobile objects and the RSS observables of the identified MWDs. ### Claim 19 (depends on 18) The method of claim 18 wherein said identifying at least one pair of associated mobile object and MWD comprises: allowing only one of the at least one mobile object movable in the site, one of the one or more MWDs being associated with said only one mobile object and movable therewith. ### Claim 20 (depends on 18) The method of claim 18 further comprising: moving said identified at least one pair of associated mobile object and MWD in the site along a survey path.