# US 10,121,247 B2 · Method of tracking a mobile device and method of generating a geometrical model of a real environment using a camera of a mobile device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,247 B2 - Document ID: US10121247B2 - Status: Granted - Kind: B2 - Application number: 15/167,391 - Filed: 2016-05-27 - Published: 2016-12-01 - Granted: 2018-11-06 - Earliest claimed: 2013-12-19 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Peter Meier; Sebastian Lieberknecht - Web page: https://patentgrep.com/patent/US10121247B2 ## CPC - H04N 7/185: {from a mobile camera, e.g. for remote control} - G06T 7/251: {involving models} - G06T 7/579: from motion - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G06T 2207/10016: Video; Image sequence - G06T 2207/30244: Camera pose - G06T 2207/30264: Parking - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method of tracking a mobile device comprising at least one camera in a real environment comprises the steps of receiving image information associated with at least one image captured by the at least one camera, generating a first geometrical model of at least part of the real environment based on environmental data or mobile system state data acquired in an acquisition process by at least one sensor of a mobile system, which is different from the mobile device, and performing a tracking process based on the image information associated with the at least one image and at least partially according to the first geometrical model, wherein the tracking process determines at least one parameter of a pose of the mobile device relative to the real environment. The invention is also related to a method of generating a geometrical model of at least part of a real environment using image information from at least one camera of a mobile device. ## Claims ### Claim 1 A method of tracking a mobile device, comprising: receiving image information associated with at least one image captured by at least one camera of a mobile device in a real environment; generating a first geometrical model of at least part of the real environment based on data acquired by at least one sensor of a vehicle, which is different from the mobile device; tracking the mobile device based on the image information associated with the at least one image, and at least partially according to the first geometrical model, by determining at least one parameter of a pose of the mobile device relative to the real environment, wherein the tracking is vision-based tracking based on Simultaneous Localization and Mapping (SLAM); and generating a second geometrical model based on received image information associated with at least one further image captured by the at least one camera or further based on received depth information associated with the at least one image, wherein the first geometrical model is generated according to one or more provided routes to a destination of the vehicle. ### Claim 2 (depends on 1) The method according to claim 1, wherein tracking the mobile device is performed within a set time period after the data is at least partially acquired. ### Claim 3 (depends on 1) The method according to claim 1, wherein the second geometrical model is generated by extending the first geometrical model. ### Claim 4 (depends on 1) The method according to claim 1, wherein at least part of the first geometrical model is generated according to one or more images captured by the at least one camera. ### Claim 5 (depends on 1) The method according to one of claim 1, wherein the data is at least partially acquired while the mobile device is being conveyed by the vehicle. ### Claim 6 (depends on 1) The method according to one of claim 1, wherein the data is at least partially acquired while the vehicle is moving and the data is acquired from the at least one sensor of the vehicle at different positions. ### Claim 7 (depends on 1) The method according to claim 1, wherein the data is acquired according to a position of the vehicle and at least one set destination for the vehicle. ### Claim 8 (depends on 7) The method according to claim 7, wherein the data is acquired after the vehicle is within a distance to the at least one set destination, or according to the position of the vehicle, a speed of the vehicle, and the at least one set destination. ### Claim 9 (depends on 1) The method according to claim 1, wherein the first geometrical model is generated based on image information associated with at least one image captured by a further camera placed in the real environment. ### Claim 10 (depends on 1) The method according to claim 1, wherein the at least one sensor of the vehicle comprises at least two vehicle cameras, with a known spatial relationship between the at least two vehicle cameras, and wherein a metric scale of the first geometrical model is determined according to the spatial relationship. ### Claim 11 (depends on 1) The method according to claim 1, wherein generating the first geometrical model is performed by a processing device of the vehicle, and the first geometrical model is transferred from the vehicle to the mobile device. ### Claim 12 (depends on 1) The method according to claim 1, wherein the data is transferred from the vehicle to the mobile device, and wherein generating the first geometrical model or a part of the first geometrical model is performed on the mobile device. ### Claim 13 (depends on 1) The method according to claim 1, wherein the data is transferred from the vehicle to a server computer, and wherein generating the first geometrical model is performed on the server computer. ### Claim 14 (depends on 1) The method according to claim 1, wherein the first geometrical model has a correct metric scale determined from vehicle-mounted sensors. ### Claim 15 (depends on 1) The method according to claim 1, wherein: one or more routes to a destination of the vehicle are provided, and wherein the data is acquired according to one or more of the provided routes. ### Claim 16 (depends on 1) The method according to claim 1, wherein at least one of the first and second geometrical models at least describes depth information of the real environment. ### Claim 17 (depends on 1) The method according to claim 1, wherein the mobile device is a device that is one selected from a group consisting of: a hand-held device, a mobile phone, a head worn glasses or helmet, or a wearable device. ### Claim 18 (depends on 1) The method according to claim 1, wherein the tracking is used for augmented reality. ### Claim 19 (depends on 1) The method according to claim 1, wherein the vision-based tracking comprises feature extraction, feature description, feature matching, and pose determination. ### Claim 20 (depends on 19) The method according to claim 19, wherein features are at least one of or combinations of: intensities, gradients, edges, lines, segments, corners, descriptive features, primitives, histograms, polarities, and orientations. ### Claim 21 A non-transitory computer readable medium comprising software code executable by one or more processors to: receive image information associated with at least one image captured by at least one camera of a mobile device in a real environment; generate a first geometrical model of at least part of the real environment based on data acquired by at least one sensor of a vehicle, which is different from the mobile device; track the mobile device based on the image information associated with the at least one image, and at least partially according to the first geometrical model, by determining at least one parameter of a pose of the mobile device relative to the real environment, wherein the tracking is vision-based tracking based on Simultaneous Localization and Mapping (SLAM); and generate a second geometrical model based on received image information associated with at least one further image captured by the at least one camera or further based on received depth information associated with the at least one image, wherein the first geometrical model is generated according to one or more provided routes to a destination of the vehicle. ### Claim 22 (depends on 21) The non-transitory computer readable medium of claim 21, wherein the data is acquired according to a position of the vehicle and at least one set destination for the vehicle. ### Claim 23 A system for tracking a mobile device, comprising: at least one camera; one or more processors; and a memory coupled to the one or more processors and comprising computer readable code executable by the one or more processors to: receive image information associated with at least one image captured by the at least one camera in a real environment, generate a first geometrical model of at least part of the real environment based on data acquired by at least one sensor of a vehicle that is separate from the mobile device, track the mobile device based on the image information associated with the at least one image, and at least partially according to the first geometrical model, by determining at least one parameter of a pose of the mobile device relative to the real environment, wherein the tracking is vision-based tracking based on Simultaneous Localization and Mapping (SLAM), and generate a second geometrical model based on received image information associated with at least one further image captured by the at least one camera or further based on received depth information associated with the at least one image, wherein the first geometrical model is generated according to one or more provided routes to a destination of the vehicle. ### Claim 24 (depends on 23) The system of claim 23, wherein the data is acquired according to a position of the vehicle and at least one set destination for the vehicle.