# US 11,994,405 B2 · Methods and systems for determining geographic orientation based on imagery > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,994,405 B2 - Document ID: US11994405B2 - Status: Granted - Kind: B2 - Application number: 16/978,374 - Filed: 2018-03-12 - Published: 2021-02-11 - Granted: 2024-05-28 - Earliest claimed: 2018-03-07 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Daniel Joseph Filip; Zhen Yang - Web page: https://patentgrep.com/patent/US11994405B2 ## CPC - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 21/28: with correlation of data from several navigational instruments - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/176: {Urban or other man-made structures} - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 2207/20081: Training; Learning - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure is directed to determining geographic orientation based at least in part on imagery. In particular, the methods and systems of the present disclosure can: receive data generated by a camera (118) and representing imagery that includes at least a portion of a physical real-world environment comprising the camera (118) and a travelway (312); and determine, based at least in part on the data and a machine-learning model, a geographic orientation of the camera (118) with respect to the travelway (312). ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by a computing system, data generated by a camera and representing imagery that includes at least a portion of a physical real-world environment comprising the camera and a travelway; estimating a position of a light source based on a time of day associated with the imagery; and determining, by the computing system, a geographic orientation of the camera with respect to the travelway, wherein the geographic orientation of the camera with respect to the travelway further comprises: determining a geographic location of the camera; and selecting, based at least in part on the geographic location of the camera, a machine-learning model from amongst a plurality of different machine-learning models for determining geographic orientations of cameras with respect to travelways, the plurality of different machine-learning models being based at least in part on training data comprising imagery from different corresponding geographic regions, the machine-learning model being based at least in part on training data comprising imagery from a geographic region comprising the geographic location of the camera, the imagery from the geographic region not including imagery that comprises the travelway. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, comprising: determining, by the computing system, a geographic orientation of the travelway with respect to the physical real-world environment; and determining, by the computing system and based at least in part on the geographic orientation of the camera with respect to the travelway and the geographic orientation of the travelway with respect to the physical real-world environment, a geographic orientation of the camera with respect to the physical real-world environment. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein determining the geographic orientation of the camera with respect to the travelway comprises: determining, based at least in part on a machine-learning model, two possible geographic orientations of the camera with respect to the travelway, the two possible geographic orientations differing by one hundred and eighty degrees; and selecting, from amongst the two possible geographic orientations, the geographic orientation of the camera with respect to the travelway. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein selecting the geographic orientation of the camera with respect to the travelway comprises: identifying, in the imagery, one or more of at least a portion of a building or at least a portion of a different travelway; and selecting the geographic orientation of the camera with respect to the travelway based at least in part on the one or more of the at least a portion of the building or the at least a portion of the different travelway. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein identifying the one or more of the at least a portion of the building or the at least a portion of the different travelway comprises: recognizing, in the imagery, text associated with the one or more of the at least a portion of the building or the at least a portion of the different travelway; and identifying, based at least in part on the text, the one or more of the at least a portion of the building or the at least a portion of the different travelway. ### Claim 6 (depends on 3) The computer-implemented method of claim 3, wherein: a user device comprises the camera; the user device comprises one or more of a global positioning system (GPS) receiver, a magnetometer, or a wireless-network interface; and selecting the geographic orientation of the camera with respect to the travelway comprises: determining, based at least in part on data generated by the one or more of the GPS receiver, the magnetometer, or the wireless-network interface, a geographic orientation of the user device with respect to the physical real-world environment; and selecting the geographic orientation of the camera with respect to the travelway based at least in part on the geographic orientation of the user device with respect to the physical real-world environment. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein a machine-learning model is based at least in part on training data comprising: a plurality of images cropped from panoramic imagery generated by a camera mounted on a vehicle that includes one or more sensors for determining a geographic orientation of the camera mounted on the vehicle with respect to one or more of: a travelway upon a portion of which the vehicle is traveling while the camera mounted on the vehicle captures the panoramic imagery, or a physical real-world environment comprising the vehicle and the travelway upon the portion of which the vehicle is traveling; and for each image of the plurality of images, a geographic orientation of the image with respect to a travelway upon a portion of which the vehicle was traveling when the camera mounted on the vehicle captured panoramic imagery from which the image was cropped, the geographic orientation of the image being determined based at least in part on data generated by the one or more sensors when the camera mounted on the vehicle captured the panoramic imagery from which the image was cropped. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, comprising communicating, by the computing system, data based at least in part on the geographic orientation to one or more of a geographic-mapping application or a geographic-navigation application. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, comprising communicating, by the computing system, data based at least in part on the geographic orientation to an augmented reality (AR) application. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein: a user device comprises the camera; receiving the data comprises receiving, locally, by the user device and from the camera, the data; and determining the geographic orientation comprises determining, locally, by the user device, the geographic orientation. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein: a user device comprises the camera; receiving the data comprises, receiving by a computing system remotely located from the user device and via one or more networks that interface the user device and the computing system remotely located from the user device, the data; and determining the geographic orientation comprises determining, by the computing system remotely located from the user device, the geographic orientation. ### Claim 12 A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors cause the system to perform operations comprising: receiving data generated by a camera and representing imagery that includes at least a portion of a physical real-world environment comprising the camera and a travelway; estimating a position of a light source based on a time of day associated with the imagery; determining a geographic orientation of the camera with respect to the travelway, wherein the geographic orientation of the camera with respect to the travelway further comprises: determining a geographic location of the camera; and selecting, based at least in part on the geographic location of the camera, a machine-learning model from amongst a plurality of different machine-learning models for determining geographic orientations of cameras with respect to travelways, the plurality of different machine-learning models being based at least in part on training data comprising imagery from different corresponding geographic regions, the machine-learning model being based at least in part on training data comprising imagery from a geographic region comprising the geographic location of the camera, the imagery from the geographic region not including imagery that comprises the travelway. ### Claim 13 (depends on 12) The system of claim 12, wherein the operations comprise: determining a geographic orientation of the travelway with respect to the physical real-world environment; and determining, based at least in part on the geographic orientation of the camera with respect to the travelway and the geographic orientation of the travelway with respect to the physical real-world environment, a geographic orientation of the camera with respect to the physical real-world environment. ### Claim 14 One or more non-transitory computer-readable media comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving data generated by a camera and representing imagery that includes at least a portion of a physical real-world environment comprising the camera and a travelway having a known orientation with respect to the physical real-world environment; estimating a position of a light source based on a time of day associated with the imagery; determining a geographic orientation of the camera with respect to the travelway, wherein the geographic orientation of the camera with respect to the travelway further comprises: determining a geographic location of the camera; and selecting, based at least in part on the geographic location of the camera, a machine-learning model from amongst a plurality of different machine-learning models for determining geographic orientations of cameras with respect to travelways, the plurality of different machine-learning models being based at least in part on training data comprising imagery from different corresponding geographic regions, the machine-learning model being based at least in part on training data comprising imagery from a geographic region comprising the geographic location of the camera, the imagery from the geographic region not including imagery that comprises the travelway. ### Claim 15 (depends on 14) The one or more non-transitory computer-readable media of claim 14, wherein determining the geographic orientation of the camera with respect to the travelway comprises: determining, based at least in part on a machine-learning model, two possible geographic orientations of the camera with respect to the travelway, the two possible geographic orientations differing by one hundred and eighty degrees; and selecting, from amongst the two possible geographic orientations, the geographic orientation of the camera with respect to the travelway.