# US 12,000,945 B2 · Visual and RF sensor fusion for multi-agent tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,000,945 B2 - Document ID: US12000945B2 - Status: Granted - Kind: B2 - Application number: 17/825,687 - Filed: 2022-05-26 - Published: 2022-12-01 - Granted: 2024-06-04 - Earliest claimed: 2021-05-28 - Assignees: NEC Corporation - Applicants: NEC Laboratories America, Inc. - Inventors: Karthikeyan Sundaresan; Ramanujan Sheshadri; Mallesham Dasari - Web page: https://patentgrep.com/patent/US12000945B2 ## CPC - G01S 5/0264: {at least one of the systems being a non-radio wave positioning system} - G01S 5/011: {Identifying the radio environment} - G01S 5/0218: {Multipath in signal reception} - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Methods and systems for determining a device position include determining a first position estimate using radio-based range information. A second position estimate is determined using visual odometry information. The first position estimate and the second position estimate are fused based on radio environmental conditions and visual environmental conditions to determine a final position estimate. Resources are deployed based on the final position estimate. ## Claims ### Claim 1 A computer-implemented method for determining a device position, comprising: determining a first position estimate using radio-based range information; determining a second position estimate using visual odometry information; determining an impact of non-line-of-sight paths on the first position estimate; fusing the first position estimate and the second position estimate based on radio environmental conditions and visual environmental conditions to determine a final position estimate; and deploying resources based on the final position estimate. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the impact of non-line-of-sight paths includes determining a radio signal strength of a plurality of anchor devices. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the impact of non-line-of-sight paths includes determining that fewer than a threshold number of anchor devices have line-of-sight paths. ### Claim 4 (depends on 2) The method of claim 2, wherein fusing the first position and the second position estimate includes processing a vector that is a concatenation of the first position estimate and the radio signal strength of the plurality of anchor devices. ### Claim 5 (depends on 1) The method of claim 1, further comprising determining an impact of environmental conditions on the second position estimate. ### Claim 6 (depends on 5) The method of claim 5, wherein determining the impact of environmental conditions includes determining keypoint matches between image frames. ### Claim 7 (depends on 6) The method of claim 6, wherein determining the impact of environmental conditions includes determining that the number of keypoint matches is below a threshold value. ### Claim 8 (depends on 6) The method of claim 6, wherein fusing the first position and the second position estimate includes processing a vector that is a concatenation of the second position estimate and keypoint matches. ### Claim 9 (depends on 1) The method of claim 1, further comprising determining the radio-based range information using an ultra-wide band transceiver and determining the visual odometry information using a stereo camera. ### Claim 10 A system for determining a device position, comprising: a hardware processor; and a memory that stores a computer program, which, when executed by the hardware processor, causes the hardware processor to: determine a first position estimate using radio-based range information; determine a second position estimate using visual odometry information; determine an impact of non-line-of-sight paths on the first position estimate; fuse the first position estimate and the second position estimate based on radio environmental conditions and visual environmental conditions to determine a final position estimate; and deploy resources based on the final position estimate. ### Claim 11 (depends on 10) The system of claim 10, wherein the computer program further causes the hardware processor to determine a radio signal strength of a plurality of anchor devices. ### Claim 12 (depends on 11) The system of claim 11, wherein the computer program further causes the hardware processor to determine that fewer than a threshold number of anchor devices have line-of-sight paths. ### Claim 13 (depends on 11) The system of claim 11, wherein the computer program further causes the hardware processor to determine process a vector that is a concatenation of the first position estimate and the radio signal strength of the plurality of anchor devices for the fusing of the first position estimate and the second position estimate. ### Claim 14 (depends on 10) The system of claim 10, wherein the computer program further causes the hardware processor to determine an impact of environmental conditions on the second position estimate. ### Claim 15 (depends on 14) The system of claim 14, wherein the computer program further causes the hardware processor to determine keypoint matches between image frames. ### Claim 16 (depends on 15) The system of claim 15, wherein the computer program further causes the hardware processor to determine that the number of keypoint matches is below a threshold value. ### Claim 17 (depends on 15) The system of claim 15, wherein the computer program further causes the hardware processor to process a vector that is a concatenation of the second position estimate and keypoint matches for the fusing of the first position estimate and the second position estimate. ### Claim 18 (depends on 10) The system of claim 10, further comprising an ultra-wide band transceiver configured to capture the radio-based range information and a stereo camera configured to capture the visual odometry information. ### Claim 19 A computer-implemented method for determining a device position, comprising: determining a first position estimate using radio-based range information; determining a second position estimate using visual odometry information; determining an impact of environmental conditions on the second position estimate by determining keypoint matches between image frames; fusing the first position estimate and the second position estimate based on radio environmental conditions and visual environmental conditions to determine a final position estimate, including processing a vector that is a concatenation of the second position estimate and keypoint matches; and deploying resources based on the final position estimate.