# US 2025/0378564 A1 · Semantic segmentation-based exclusion for localization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0378564 A1 - Document ID: US20250378564A1 - Status: Published application - Kind: A1 - Application number: 19/220,771 - Filed: 2025-05-28 - Published: 2025-12-11 - Earliest claimed: 2024-06-07 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Huiwen Guo; Jose Maria Facil Ledesma; Lina M. Paz-Perez; Oleg Naroditsky; Abdelhamid Dine; Joseph A. Menke - Web page: https://patentgrep.com/patent/US20250378564A1 ## CPC - G06T 7/248: {involving reference images or patches} - G06T 7/74: {involving reference images or patches} - G06V 10/764: using classification, e.g. of video objects - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06V 20/20: in augmented reality scenes - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Various implementations disclosed herein include devices, systems, and methods that localize (e.g., determine a pose) of a device in a 3D environment based on sensor data and semantic segmentation information. Some implementations provide device localization on moving platforms (e.g., trains, buses, cars, etc.) based on camera images (i.e., vision). Since motion (i.e., IMU) data may not be reliable in such moving environments, image and/or other sensor data may be more heavily relied upon than in other circumstances. Some implementations improve the usability of vision-based tracking features points. This may involve identifying and removing outlier tracking feature points based on semantics. For example, tracking features points corresponding to the outside environment, which is not moving with the moving platform, may be excluded based on semantic information identifying that they are not part of the moving platform (e.g., that they are instead seen through a window, not trackable, etc.). ## Claims ### Claim 1 A method comprising: at a device having a processor and one or more sensors: obtaining semantic keyframes corresponding to a physical environment while the device is on a moving platform, the semantic keyframes each providing a two-dimensional (2D) image identifying semantic labels for portions of the physical environment visible from a respective viewpoint within the physical environment, wherein the semantic keyframes are generated based on images captured by a first set of one or more of the sensors; determining a plurality of tracking features based on data from a second set of the one or more sensors; determining semantic labels for the tracking features based on the semantic keyframes; selecting a subset of the tracking features based on the semantic labels determined for the tracking features, the subset excluding tracking features associated with an external environment separate from the moving platform; and tracking a pose of the device in the physical environment using the subset of tracking features. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the semantic labels for the tracking features comprises: determining a set of three-dimensional (3D) positions of the tracking features; and determining the semantic labels based on the 3D positions and the semantic keyframes. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the 3D positions comprises triangulating tracking features based on simultaneous images captured by multiple sensors of the second set of one or more sensors. ### Claim 4 (depends on 2) The method of claim 2, wherein determining the 3D positions comprises approximating the 3D positions based on one or more images captured by a single sensor of the second set of one or more sensors. ### Claim 5 (depends on 2) The method of claim 2, wherein the semantic labels are determined based on projecting the 3D positions of the tracking features into one or more of the semantic keyframes. ### Claim 6 (depends on 1) The method of claim 1, wherein the second set of sensors is different from the first set of sensors. ### Claim 7 (depends on 1) The method of claim 1, wherein the semantic keyframes are generated at a different frame rate than tracking features. ### Claim 8 (depends on 1) The method of claim 1, wherein the semantic labels for the portions of the physical environment identify whether the portions of the environment correspond to transparent window portions. ### Claim 9 (depends on 1) The method of claim 1, wherein the semantic labels for the portions of the physical environment identify which portions of the environment correspond to portions that move with the moving platform and which portions of the environment do not move with the moving platform. ### Claim 10 (depends on 1) The method of claim 1, wherein the moving platform is a bus, train, or automobile. ### Claim 11 (depends on 1) The method of claim 1, wherein the device is a head mounted device (HMD). ### Claim 12 A system comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining semantic keyframes corresponding to a physical environment while the device is on a moving platform, the semantic keyframes each providing a two-dimensional (2D) image identifying semantic labels for portions of the physical environment visible from a respective viewpoint within the physical environment, wherein the semantic keyframes are generated based on images captured by a first set of one or more of the sensors; determining a plurality of tracking features based on data from a second set of the one or more sensors; determining semantic labels for the tracking features based on the semantic keyframes; selecting a subset of the tracking features based on the semantic labels determined for the tracking features, the subset excluding tracking features associated with an external environment separate from the moving platform; and tracking a pose of the device in the physical environment using the subset of tracking features. ### Claim 13 (depends on 12) The system of claim 12, wherein determining the semantic labels for the tracking features comprises: determining a set of three-dimensional (3D) positions of the tracking features; and determining the semantic labels based on the 3D positions and the semantic keyframes. ### Claim 14 (depends on 13) The system of claim 13, wherein determining the 3D positions comprises triangulating tracking features based on simultaneous images captured by multiple sensors of the second set of one or more sensors. ### Claim 15 (depends on 13) The system of claim 13, wherein determining the 3D positions comprises approximating the 3D positions based on one or more images captured by a single sensor of the second set of one or more sensors. ### Claim 16 (depends on 13) The system of claim 13, wherein the semantic labels are determined based on projecting the 3D positions of the tracking features into one or more of the semantic keyframes. ### Claim 17 (depends on 12) The system of claim 12, wherein the second set of sensors is different from the first set of sensors. ### Claim 18 (depends on 12) The system of claim 12, wherein the semantic keyframes are generated at a different frame rate than tracking features. ### Claim 19 (depends on 12) The system of claim 12, wherein the semantic labels for the portions of the physical environment identify whether the portions of the environment correspond to transparent window portions. ### Claim 20 A non-transitory computer-readable storage medium, storing program instructions executable via a processor to perform operations comprising: obtaining semantic keyframes corresponding to a physical environment while the device is on a moving platform, the semantic keyframes each providing a two-dimensional (2D) image identifying semantic labels for portions of the physical environment visible from a respective viewpoint within the physical environment, wherein the semantic keyframes are generated based on images captured by a first set of one or more sensors; determining a plurality of tracking features based on data from a second set of one or more sensors; determining semantic labels for the tracking features based on the semantic keyframes; selecting a subset of the tracking features based on the semantic labels determined for the tracking features, the subset excluding tracking features associated with an external environment separate from the moving platform; and tracking a pose of the device in the physical environment using the subset of tracking features.