# US 10,872,264 B2 · Shelf space allocation management device and shelf space allocation management method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,264 B2 - Document ID: US10872264B2 - Status: Granted - Kind: B2 - Application number: 16/125,383 - Filed: 2018-09-07 - Published: 2019-01-03 - Granted: 2020-12-22 - Earliest claimed: 2015-01-22 - Assignees: NEC Corporation - Applicants: NEC Corporation - Inventors: Nobuyuki Yamashita - Web page: https://patentgrep.com/patent/US10872264B2 ## CPC - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - B65G 1/137: with arrangements or automatic control means for selecting which articles are to be removed - G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target} - G06F 18/22: Matching criteria, e.g. proximity measures - G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders - G06Q 10/08743: {using optimisation of routes} - G06Q 30/0639: {Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility} - G06V 10/17: {using hand-held instruments} - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 20/20: in augmented reality scenes - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - B64U 10/17: Helicopters (flying platforms B64U10/13) - B64U 2101/26: for manufacturing, inspections or repairs - B64U 2101/31: for surveillance - B64U 2201/20: Remote controls - G06V 40/172: {Classification, e.g. identification} - G06V 40/23: {Recognition of whole body movements, e.g. for sport training} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR ## Abstract A shelf space allocation management device manages products allocated on shelves aligned in a store by use of an image captured by an imaging device. The shelf space allocation management device acquires an image including a position assumed to be changed in allocation status of each product on each shelf, it determines whether the type and the allocation status of each product reflected in the image match the predetermined type and the predetermined allocation status; then, it determines whether to execute a product allocation inspection based on the determination result. Herein, the shelf space allocation management device specifies a position at which a person conducts a behavior to cause any change in the allocation status of each product on each shelf, and therefore it may control the imaging device to capture an image including the position. It is possible to carry out a product allocation inspection for each period determined in advance depending on the type of each product, or it is possible to carry out a product allocation inspection being triggered by a customer purchasing each product. ## Claims ### Claim 1 A shelf space allocation management device for managing a plurality of products allocated on a shelf among a plurality of shelves using a mobile imaging device movable in a store, the shelf space allocation management device comprising: a memory storing instructions; and a processor coupled to the memory and configured to execute the instructions to: acquire a first image taken by the mobile imaging device configured to move in a store under an automatic tracking mode configured to track a clerk; determine whether the clerk located in the store and captured in the first image has performed a predetermined action on the shelf; specify position information of the shelf for a product allocation inspection, wherein the position information is assumed to be changed in product allocation due to the predetermined action on the shelf; change the automatic tracking mode to a position control mode directing movement according to position information of the mobile imaging device and a target position determined in advance after specifying the position information of the shelf; move a drone having the mobile imaging device and configured to automatically track a person located in the store to an image-capture position in a direction calculated according to the position information of the shelf in the position control mode; adjust the direction of the mobile imaging device upon determining that the mobile imaging device arrives at the image-capture position in the position control mode; control the mobile imaging device to capture a second image on the shelf at the image-capture position; determine whether a type and an allocation status of a plurality of products reflected in the second image match a predetermined type and a predetermined allocation status of the plurality of products; and execute the product allocation inspection based on a determination result of the type and the allocation status of the product. ### Claim 2 (depends on 1) The shelf space allocation management device according to claim 1, wherein the predetermined action is determined using a third image of the person captured by a secondary imaging device fixed at a predetermined position in the store as to whether the person has displayed the product on the shelf. ### Claim 3 (depends on 1) The shelf space allocation management device according to claim 1, wherein the predetermined action is determined using a third image of the person captured by a secondary imaging device fixed at a predetermined position in the store as to whether the person has repeatedly extended an arm toward at least one of the plurality of products on the shelf a predetermined number of times in a predetermined period of time. ### Claim 4 (depends on 1) The shelf space allocation management device according to claim 1, wherein the processor is further configured to execute the product allocation inspection in a first time interval which is determined for each product among the plurality of products in advance or in a second time interval which is determined for each shelf among the plurality of shelves in advance. ### Claim 5 (depends on 1) The shelf space allocation management device according to claim 1, wherein the processor is further configured to estimate an allocated orientation of each product, wherein the allocated orientation comprises at least one of a vertical or horizontal orientation of each product or a backward or forward orientation of each product. ### Claim 6 (depends on 1) The shelf space allocation management device according to claim 1, wherein the processor is further configured to determine that the allocation status of the plurality of products matches the predetermined allocation status of the plurality of products when a number of products reflected in the second image is greater than or equal to a predetermined number of products. ### Claim 7 A shelf space allocation management method for managing a plurality of products allocated on a shelf among a plurality of shelves using a mobile imaging device movable in a store, the shelf space allocation management method comprising: acquiring a first image taken by the mobile imaging device configured to move in a store under an automatic tracking mode configured to track a clerk; determining whether the clerk located in the store and captured in the first image has performed a predetermined action on the shelf; specifying position information of the shelf for a product allocation inspection, wherein the position information is assumed to be changed in product allocation due to the predetermined action on the shelf; changing the automatic tracking mode to a position control mode directing movement according to position information of the mobile imaging device and a target position determined in advance after specifying the position information of the shelf; moving a drone having the mobile imaging device and configured to automatically track a person located in the store to an image-capture position in a direction calculated according to the position information of the shelf in the position control mode; adjusting the direction of the mobile imaging device upon determining that the mobile imaging device arrives at the image-capture position in the position control mode; controlling the mobile imaging device to capture a second image on the shelf at the image-capture position; determining whether a type and an allocation status of a plurality of products reflected in the second image match a predetermined type and a predetermined allocation status of the plurality of products; and executing the product allocation inspection based on a determination result of the type and the allocation status of the product. ### Claim 8 (depends on 7) The shelf space allocation management method according to claim 7, wherein the predetermined action is determined using a third image of the person captured by a secondary imaging device fixed at a predetermined position in the store as to whether the person has displayed the product on the shelf. ### Claim 9 (depends on 7) The shelf space allocation management method according to claim 7, wherein the predetermined action is determined using a third image of the person captured by a secondary imaging device fixed at a predetermined position in the store as to whether the person has repeatedly extended an arm toward at least one of the plurality of products on the shelf a predetermined number of times in a predetermined period of time. ### Claim 10 (depends on 7) The shelf space allocation management method according to claim 7, further comprising executing the product allocation inspection in a first time interval which is determined for each product among the plurality of products in advance or in a second time interval which is determined for each shelf among the plurality of shelves in advance. ### Claim 11 (depends on 7) The shelf space allocation management method according to claim 7, further comprising estimating an allocated orientation of each product, wherein the allocated orientation comprises at least one of a vertical or horizontal orientation of each product or a backward or forward orientation of each product. ### Claim 12 (depends on 7) The shelf space allocation management method according to claim 7, further comprising determining that the allocation status of the plurality of products matches the predetermined allocation status of the plurality of products when a number of products reflected in the second image is greater than or equal to a predetermined number of products. ### Claim 13 A non-transitory computer readable medium having stored therein a shelf space allocation management program causing a computer to perform a shelf space allocation management process for managing a plurality of products allocated on a shelf among a plurality of shelves using a mobile imaging device movable in a store, the shelf space allocation management process comprising: acquiring a first image taken by the mobile imaging device configured to move in a store under an automatic tracking mode configured to track a clerk; determining whether the clerk located in the store and captured in the first image has performed a predetermined action on the shelf; specifying position information of the shelf for a product allocation inspection, wherein the position information is assumed to be changed in product allocation due to the predetermined action on the shelf; changing the automatic tracking mode to a position control mode directing movement according to position information of the mobile imaging device and a target position determined in advance after specifying the position information of the shelf; moving a drone having the mobile imaging device and configured to automatically track a person located in the store to an image-capture position in a direction according to the position information of the shelf in the position control mode; adjusting the direction of the mobile imaging device upon determining that the mobile imaging device arrives at the image-capture position in the position control mode; controlling the mobile imaging device to capture a second image on the shelf at the image-capture position; determining whether a type and an allocation status of a plurality of products reflected in the second image match a predetermined type and a predetermined allocation status of the plurality of products; and executing the product allocation inspection based on a determination result of the type and the allocation status of the product. ### Claim 14 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the predetermined action is determined using a third image of the person captured by a secondary imaging device fixed at a predetermined position in the store as to whether the person has displayed the product on the shelf. ### Claim 15 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the predetermined action is determined using a third image of the person captured by a secondary imaging device fixed at a predetermined position in the store as to whether the person has repeatedly extended an arm toward at least one of the plurality of products on the shelf a predetermined number of times in a predetermined period of time. ### Claim 16 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the shelf space allocation management process further comprises executing the product allocation inspection in a first time interval which is determined for each product among the plurality of products in advance or in a second time interval which is determined for each shelf among the plurality of shelves in advance. ### Claim 17 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the shelf space allocation management process further comprises estimating an allocated orientation of each product, wherein the allocated orientation comprises at least one of a vertical or horizontal orientation of each product or a backward or forward orientation of each product. ### Claim 18 (depends on 13) The non-transitory computer readable medium according to claim 13, wherein the shelf space allocation management process further comprises determining that the allocation status of the plurality of products matches the predetermined allocation status of the plurality of products when a number of products reflected in the second image is greater than or equal to a predetermined number of products.