# US 12,181,301 B2 · Hands-free augmented reality system for picking and/or sorting assets > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,181,301 B2 - Document ID: US12181301B2 - Status: Granted - Kind: B2 - Application number: 18/200,165 - Filed: 2023-05-22 - Published: 2023-09-14 - Granted: 2024-12-31 - Earliest claimed: 2017-08-15 - Assignees: United Parcel Service of America, Inc. - Applicants: United Parcel Service of America, Inc. - Inventors: Julio Gil - Web page: https://patentgrep.com/patent/US12181301B2 ## CPC - G01C 21/365: {Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself} - G01C 21/3632: {Guidance using simplified or iconic instructions, e.g. using arrows (G01C21/365 takes precedence)} - G01C 21/3641: {Personalized guidance, e.g. limited guidance on previously travelled routes} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 3/147: using display panels - G06Q 10/047: Optimisation of routes or paths, e.g. travelling salesman problem - G06Q 10/0833: Tracking - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)} - G09G 2340/12: Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels - G09G 2354/00: Aspects of interface with display user - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract A system is provided for hands-free handling of at least one asset by a user. The system includes a user device configured to be worn by a user and a control system remotely located relative to the user device and configured to exchange asset-related data with the user device, the asset-related data including at least one or more notifications related to the handling of the at least one asset by the user at the asset location. The user device contains a processor configured to determine location data associated with the at least one asset and including a location for the at least one asset, the determination being based, in part, upon the obtained asset identifier data; dynamically generate and display, at the user device, one or more navigational projections configured to guide the user to the asset location; and detect handling of the at least one asset by the user. ## Claims ### Claim 1 A method comprising: detecting, by computing hardware, handling of an asset located within a physical environment; responsive to detecting handling of the asset, obtaining, by the computing hardware, sort location data associated with the asset, wherein the sort location data comprises a sort location within the physical environment; determining, by the computing hardware, a present location of the asset; obtaining, by the computing hardware and based on the sort location data and the present location, sort instructions comprising a route to travel from the present location to the sort location through the physical environment; and causing, by the computing hardware and based at least in part on the sort instructions, display of one or more navigational projections over a surface within the physical environment, wherein the one or more navigational projections are configured to guide a user over the route from the present location to the sort location. ### Claim 2 (depends on 1) The method of claim 1, wherein the computing hardware comprises a user device being worn by the user that comprises a pivoting laser projector configured for displaying of the one or more navigational projections and a depth sensor configured for collecting data used in obtaining the present location. ### Claim 3 (depends on 2) The method of claim 2, wherein the user device further comprises a camera and detecting handling of the asset comprises: obtaining, via an image taken of the asset by the camera, asset identifier data associated with the asset; and identifying, based at least in part on the asset identifier data, the asset. ### Claim 4 (depends on 1) The method of claim 1, wherein the computing hardware comprises a control system, and causing display of the one or more navigational projections over the surface within the physical environment comprises sending the sort instructions to a user device being worn by the user to cause the user device to generate the one or more navigational projections based on the sort instructions and display the one or more navigational projections over the surface within the physical environment. ### Claim 5 (depends on 1) The method of claim 1 further comprising generating, by the computing hardware and based at least in part on mapping the physical environment, a multidimensional graphical representation of the physical environment, wherein causing display of the one or more navigational projections over the surface within the physical environment comprises: generating, based at least in part on the sort instructions and the multidimensional graphical representation, the one or more navigational projections; and causing display, via a user device being worn by the user, of the one or more navigational projections over the surface within the physical environment. ### Claim 6 (depends on 1) The method of claim 1 further comprising causing a transmission of audible instructions to the user concurrent with displaying of the one or more navigational projections. ### Claim 7 (depends on 6) The method of claim 6, wherein the audible instructions are configured to provide feedback in response to detection of a deviation of the user from the one or more navigational projections. ### Claim 8 An apparatus comprising: a non-transitory computer-readable medium storing instructions; and processing hardware communicatively coupled to the non-transitory computer-readable medium, wherein the processing hardware is configured to execute the instructions and thereby perform operations comprising: detecting handling of an asset located within a physical environment; responsive to detecting handling of the asset, obtaining a sort location within the physical environment for the asset; obtaining, based on the sort location and a present location of the asset, sort instructions comprising a route to travel from the present location to the sort location through the physical environment; and causing, based at least in part on the sort instructions, display of one or more navigational projections over a surface within the physical environment, wherein the one or more navigational projections are configured to guide a user over the route from the present location to the sort location. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the apparatus comprises a user device being worn by the user, and the apparatus further comprises: a pivoting laser projector configured for displaying of the one or more navigational projections; and a depth sensor configured for collecting data used in obtaining the present location of the asset. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the apparatus further comprises a camera and detecting handling of the asset comprises: obtaining, via the camera, an image of the asset; identifying asset identifier data associated with the asset from the image; and identifying, based at least in part on the asset identifier data, the asset. ### Claim 11 (depends on 8) The apparatus of claim 8, wherein the apparatus comprises a control system, and causing display of the one or more navigational projections over the surface within the physical environment comprises sending the sort instructions to a user device being worn by the user to cause the user device to generate the one or more navigational projections based on the sort instructions and display the one or more navigational projections over the surface within the physical environment. ### Claim 12 (depends on 8) The apparatus of claim 8, wherein the operations further comprising generating a multidimensional graphical representation of the physical environment, and causing display of the one or more navigational projections over the surface within the physical environment comprises: generating, based at least in part on the sort instructions and the multidimensional graphical representation, the one or more navigational projections; and causing display, via a user device being worn by the user, of the one or more navigational projections over the surface within the physical environment. ### Claim 13 (depends on 8) The apparatus of claim 8, wherein the operations further comprising causing a transmission of audible instructions to the user concurrent with displaying of the one or more navigational projections. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the audible instructions are configured to provide feedback in response to detection of a deviation of the user from the one or more navigational projections. ### Claim 15 A non-transitory computer-readable medium storing computer-executable instructions that, when executed by processing hardware, configure the processing hardware to perform operations comprising: obtaining sort location data associated with an asset being handled within a physical environment, wherein the sort location data comprises a sort location within the physical environment; determining a present location of the asset; obtaining, based on the sort location data and the present location, sort instructions comprising a route to travel from the present location to the sort location through the physical environment; and causing, based at least in part on the sort instructions, display of one or more navigational projections over a surface within the physical environment, wherein the one or more navigational projections are configured to guide a user over the route from the present location to the sort location. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein a user device being worn by the user comprises the processing hardware. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the user device further comprises: a pivoting laser projector configured for displaying of the one or more navigational projections; and a depth sensor configured for collecting data used in obtaining the present location of the asset. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein a control system comprises the processing hardware, and causing display of the one or more navigational projections over the surface within the physical environment comprises sending the sort instructions to a user device being worn by the user to cause the user device to generate the one or more navigational projections based on the sort instructions and display the one or more navigational projections over the surface within the physical environment. ### Claim 19 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the operations further comprise generating, based at least in part on mapping the physical environment, a multidimensional graphical representation of the physical environment, and causing display of the one or more navigational projections over the surface within the physical environment comprises: generating, based at least in part on the sort instructions and the multidimensional graphical representation, the one or more navigational projections; and causing display, via a user device being worn by the user, of the one or more navigational projections over the surface within the physical environment. ### Claim 20 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the operations further comprise causing a transmission of audible instructions to the user concurrent with displaying of the one or more navigational projections.