# US 11,907,889 B2 · System, method, and computer-readable medium for displaying layout-based instructions using augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,889 B2 - Document ID: US11907889B2 - Status: Granted - Kind: B2 - Application number: 18/144,660 - Filed: 2023-05-08 - Published: 2023-08-31 - Granted: 2024-02-20 - Earliest claimed: 2017-03-01 - Assignees: Blue Yonder Group, Inc. - Applicants: Blue Yonder Group, Inc. - Inventors: Justin Soles; Benoît Rochon; Christopher Tyler - Web page: https://patentgrep.com/patent/US11907889B2 ## CPC - G06Q 10/083: Shipping - G06Q 10/0841: {Compliance with shipping requirements} - G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders - G06Q 30/0635: {replenishment orders; recurring orders} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and method are disclosed for a supply chain entity having an inventory of one or more items and a warehouse management system receiving orders for one or more items, determining a container for the one or more items, and generating a packing plan for the items. Aspects of the embodiments may include one or more instruction rendering devices having a sensor and a displaying augmented reality instructions and feedback. ## Claims ### Claim 1 A system, comprising: a supply chain entity comprising an inventory having one or more items; an instruction rendering device comprising one or more sensors and a display and configured to: display augmented reality instructions and feedback according to instructions generated according to a packing plan; overlay the augmented reality instructions and feedback in connection with a physical location of a container, wherein the instruction rendering device repeatedly updates the physical location of the augmented reality instructions and feedback according to a progress of a user in following the augmented reality instructions according to the packing plan; display the augmented reality instructions and feedback in connection with a detected movement of the container or a detected movement of the instruction rendering device; and identify on the instruction rendering device when one or more ordered items of the one or more items have been correctly placed and oriented in the container according to the packing plan. ### Claim 2 (depends on 1) The system of claim 1, wherein the instruction rendering device further comprises a head-mounted display, a smartphone, a tablet computer, a mobile device or a projector. ### Claim 3 (depends on 1) The system of claim 1, wherein the instruction rendering device is further configured to: display the augmented reality instructions and feedback by overlaying one or more visual elements over a camera feed and altering an appearance and placement of the visual elements based on one or more frames of the camera feed. ### Claim 4 (depends on 1) The system of claim 1, wherein the augmented reality instructions and feedback further comprise displayed visual elements superimposed on a real-world packing area and the one or more ordered items. ### Claim 5 (depends on 1) The system of claim 1, wherein the augmented reality instructions and feedback further comprise: a particular side of an item needs to be in an up direction, how many items may be packed on top of a particular item, or where in the container an item is to be placed. ### Claim 6 (depends on 1) The system of claim 1, wherein the feedback further comprises: one or more visual elements comprising one or more of colors, symbols or text; and one or more auditory elements comprising speech and sounds, wherein the feedback further comprises identification of completion or incompletion of an instruction, the identification of incompletion further comprising a degree to which the instruction has not been completed. ### Claim 7 (depends on 1) The system of claim 1, wherein the instruction rendering device is further configured to: map an object visualization and a packing area boundary to a physical environment, such that a three dimensional representation of the container and a contour delineating an extent of a packing area are in a same spot regardless of where the instruction rendering device is located or positioned. ### Claim 8 A computer-implemented method, comprising: receiving, by a computer, a current state of one or more items in a supply chain network comprising one or more supply chain entities, wherein an inventory of the one or more supply chain entities is used to store the one or more items; displaying, by an instruction rendering device, augmented reality instructions and feedback according to instructions generated according to a packing plan; overlaying, by the instruction rendering device, the augmented reality instructions and feedback in connection with a physical location of a container, wherein the instruction rendering device repeatedly updates the physical location of the augmented reality instructions and feedback according to a progress of a user in following the augmented reality instructions according to the packing plan; displaying, by the instruction rendering device, the augmented reality instructions and feedback in connection with a detected movement of the container or a detected movement of the instruction rendering device; and identifying on the instruction rendering device when one or more ordered items of the one or more items have been correctly placed and oriented in the container according to the packing plan. ### Claim 9 (depends on 8) The method of claim 8, wherein the instruction rendering device further comprises a head-mounted display, a smartphone, a tablet computer, a mobile device or a projector. ### Claim 10 (depends on 8) The method of claim 8, further comprising: displaying, by the instruction rendering device, the augmented reality instructions and feedback by overlaying one or more visual elements over a camera feed and altering an appearance and placement of the visual elements based on one or more frames of the camera feed. ### Claim 11 (depends on 8) The method of claim 8, wherein the augmented reality instructions and feedback further comprise displayed visual elements superimposed on a real-world packing area and the one or more ordered items. ### Claim 12 (depends on 8) The method of claim 8, wherein the augmented reality instructions and feedback further comprise: a particular side of an item needs to be in an up direction, how many items may be packed on top of a particular item, or where in the container an item is to be placed. ### Claim 13 (depends on 8) The method of claim 8, wherein the feedback further comprises: one or more visual elements comprising one or more of colors, symbols or text; and one or more auditory elements comprising speech and sounds, wherein the feedback further comprises identification of completion or incompletion of an instruction, the identification of incompletion further comprising a degree to which the instruction has not been completed. ### Claim 14 (depends on 8) The method of claim 8, further comprising: mapping, by the instruction rendering device, an object visualization and a packing area boundary to a physical environment, such that a three dimensional representation of the container and a contour delineating an extent of a packing area are in a same spot regardless of where the instruction rendering device is located or positioned. ### Claim 15 A non-transitory computer-readable medium embodied with software, the software when executed configured to: receive a current state of one or more items in a supply chain network comprising one or more supply chain entities, wherein an inventory of the one or more supply chain entities is used to store the one or more items; display, by an instruction rendering device, augmented reality instructions and feedback according to instructions generated according to a packing plan; overlay, by the instruction rendering device, the augmented reality instructions and feedback in connection with a physical location of a container, wherein the instruction rendering device repeatedly updates the physical location of the augmented reality instructions and feedback according to a progress of a user in following the augmented reality instructions according to the packing plan; display, by the instruction rendering device, the augmented reality instructions and feedback in connection with a detected movement of one or more of the container and the instruction rendering device; and identify, by the instruction rendering device, when the one or more ordered items have been correctly placed and oriented in the container according to the packing plan. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the instruction rendering device further comprises a head-mounted display, a smartphone, a tablet computer, a mobile device or a projector. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the software is further configured to: display, by the instruction rendering device, the augmented reality instructions and feedback by overlaying one or more visual elements over a camera feed and altering an appearance and placement of the visual elements based on one or more frames of the camera feed. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the augmented reality instructions and feedback further comprise displayed visual elements superimposed on a real-world packing area and the one or more ordered items. ### Claim 19 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the augmented reality instructions and feedback further comprise: a particular side of an item needs to be in an up direction, how many items may be packed on top of a particular item, or where in the container an item is to be placed. ### Claim 20 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the feedback further comprises: one or more visual elements comprising one or more of colors, symbols or text; and one or more auditory elements comprising speech and sounds, wherein the feedback further comprises identification of completion or incompletion of an instruction, the identification of incompletion further comprising a degree to which the instruction has not been completed.