# US 12,217,454 B2 · Method and device with sensing data processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,454 B2 - Document ID: US12217454B2 - Status: Granted - Kind: B2 - Application number: 17/698,266 - Filed: 2022-03-18 - Published: 2023-04-27 - Granted: 2025-02-04 - Earliest claimed: 2021-10-21 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Donghoon Sagong; Hojin Ju - Web page: https://patentgrep.com/patent/US12217454B2 ## CPC - G06T 7/73: using feature-based methods - G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 10/764: using classification, e.g. of video objects - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps - G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence - G06T 2207/30244: Camera pose - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A sensing data processing device is provided. The sensing data processing device may track keypoints within a full observable range of the sensing data processing device based on sensing data collected from a previous frame and a sensing data collected from a current frame through a reference sensor, the full observable range determined based on a combined field of view (FOV) of sensors, manage a new keypoint together with the keypoints in response to the new keypoint being detected, and perform localization of the device based on the managed keypoints. ## Claims ### Claim 1 A sensing data processing device, comprising: a reference sensor configured to generate sensing data by capturing a scene with a field of view (FOV); and a processor configured to: track keypoints within a full observable range of the sensing data processing device based on sensing data collected from a previous frame and sensing data collected from a current frame by the reference sensor, and manage a new keypoint together with the tracked keypoints in response to a detection of the new keypoint, wherein the full observable range is determined based on a combined FOV of the reference sensor and a second sensor, and wherein the processor is further configured to update which region the tracked keypoints belong to within the full observable range based on changed positions of the tracked keypoints. ### Claim 2 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to determine, for at least one partial region in the full observable range, a changed position of a keypoint in the at least one partial region among the tracked keypoints. ### Claim 3 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to: determine a changed position of a keypoint of the tracked keypoints based on sensing by the reference sensor, for a region of the FOV of the reference sensor that does not overlap a FOV of the second sensor, and determine a changed position of a keypoint of the tracked keypoints based on sensing by the second sensor, for a region of the FOV of the second sensor that does not overlap the FOV of the reference sensor. ### Claim 4 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to: determine a changed position of a keypoint of the tracked keypoints based on sensing by the reference sensor, for a region of the FOV of the reference sensor that overlaps a FOV of the second sensor, and skip tracking a keypoint of the tracked keypoints based on sensing by the second sensor, for a region of the FOV of the second sensor that overlaps the FOV of the reference sensor. ### Claim 5 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to exclude a keypoint whose changed position is out of the full observable range, among the tracked keypoints, from tracking. ### Claim 6 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to classify the tracked keypoints as belonging to one of a non-overlapping region and an overlapping region based on a non-overlapping mask that indicates the non-overlapping region, and an overlapping mask that indicates the overlapping region. ### Claim 7 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to remove an outlier from among the tracked keypoints based on the sensing data collected from the previous frame and the sensing data collected from the current frame by the reference sensor. ### Claim 8 (depends on 1) The sensing data processing device of claim 1, wherein the processor is further configured to detect the new keypoint in an overlapping region that overlaps an FOV of the second sensor based on sensing data collected by the reference sensor. ### Claim 9 (depends on 1) The sensing data processing device of claim 1, further comprising: an output device configured to output a localization result estimated for the sensing data processing device, wherein the processor is further configured to estimate at least one of a position and a pose of the sensing data processing device as the localization result based on a result of refining the tracked keypoints. ### Claim 10 A processor-implemented sensing data processing method, the method, comprising: tracking keypoints within a full observable range of a sensing data processing device based on sensing data collected from a previous frame and sensing data collected from a current frame by a reference sensor by capturing a scene with a field of view (FOV); refining the tracked keypoints; and managing a new keypoint together with the tracked keypoints in response to a detection of the new keypoint, wherein the full observable range is determined based on a combined FOV of the reference sensor and a second sensor, and wherein the tracking comprises updating which region the tracked keypoints belong to within the full observable range based on changed positions of the tracked keypoints. ### Claim 11 (depends on 10) The sensing data processing method of claim 10, wherein the tracking comprises determining, for at least one partial region in the full observable range, a changed position of a keypoint in the at least one partial region among the tracked keypoints. ### Claim 12 (depends on 10) The sensing data processing method of claim 10, wherein the tracking comprises: determining a changed position of a keypoint of the tracked keypoints based on sensing by the reference sensor, for a region of the FOV of the reference sensor that does not overlap a FOV of the second sensor; and determining a changed position of a keypoint of the tracked keypoints based on sensing by the second sensor, for a region of the FOV of the second sensor that does not overlap the FOV of the reference sensor. ### Claim 13 (depends on 10) The sensing data processing method of claim 10, wherein the tracking comprises: determining a changed position of a keypoint of the tracked keypoints based on sensing by the reference sensor, for a region of the FOV of the reference sensor that overlaps a FOV of the second sensor; and skipping tracking a keypoint of the tracked keypoints based on sensing by the second sensor, for a region of the FOV of the second sensor that overlaps the FOV of the reference sensor. ### Claim 14 (depends on 10) The sensing data processing method of claim 10, wherein the tracking comprises excluding a keypoint whose changed position is out of the full observable range, among the tracked keypoints, from tracking. ### Claim 15 (depends on 10) The sensing data processing method of claim 10, wherein the tracking comprises classifying the tracked keypoints as belonging to one of a non-overlapping region and an overlapping region based on a non-overlapping mask that indicates the non-overlapping region, and an overlapping mask that indicates the overlapping region. ### Claim 16 (depends on 10) The sensing data processing method of claim 10, wherein the refining comprises removing an outlier from among the tracked keypoints based on the sensing data collected from the previous frame and the sensing data collected from the current frame by the reference sensor. ### Claim 17 (depends on 10) The sensing data processing method of claim 10, wherein the managing comprises detecting a new keypoint in an overlapping region that overlaps an FOV of the second sensor based on sensing data collected by the reference sensor. ### Claim 18 (depends on 10) A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the sensing data processing method of claim 10. ### Claim 19 A sensing data processing device, comprising: a first sensor, configured to capture a first image with a first field of view (FOV); a second sensor, divergent from the first sensor, and configured to capture a second image with a second FOV; one or more processors, configured to: determine an overlapping area and non-overlapping areas between the first image and the second image; detect keypoints in the overlapping area and the non-overlapping areas; track changes in positions of the keypoints in the overlapping area and the non-overlapping areas for each consecutive frame of the first image and the second image; refine the tracked keypoints by removing an outlier from the tracked keypoints, and manage a new keypoint together with the tracked keypoints in response to a detection of the new keypoint in a current frame. ### Claim 20 (depends on 19) The sensing data processing device of claim 19, wherein the new keypoint is detected in an overlapping region that overlaps an FOV of the second sensor based on sensing data collected by the first sensor. ### Claim 21 (depends on 19) The sensing data processing device of claim 19, further comprising tracking the keypoints in the overlapping region and keypoints in the non-overlapping regions, and applying the tracked keypoints to localization.