# US 11,113,825 B2 · Multi-surface image projection item tracking system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,825 B2
- Document ID: US11113825B2
- Status: Granted
- Kind: B2
- Application number: 16/562,926
- Filed: 2019-09-06
- Published: 2020-01-23
- Granted: 2021-09-07
- Earliest claimed: 2018-07-16
- Assignees: Accel Robotics Corporation
- Applicants: Accel Robotics Corporation
- Inventors: Marius Buibas; John Quinn; Kaylee Feigum; Csaba Petre; Michael Brandon Maseda; Martin Alan Cseh
- Web page: https://patentgrep.com/patent/US11113825B2

## CPC

- G01G 19/4144: {for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems}
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06Q 20/203: {Inventory monitoring}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/06: Buying, selling or leasing transactions
- G06T 7/248: {involving reference images or patches}
- G06T 7/292: Multi-camera tracking
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/003: {Navigation within 3D models or images}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G07F 9/026: {for alarm, monitoring and auditing in vending machines or means for indication, e.g. when empty}
- G07G 1/0036: {Checkout procedures}
- G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48)
- G06T 2210/21: Collision detection, intersection
- G01G: WEIGHING (sorting by weighing B07C5/16)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07F: COIN-FREED OR LIKE APPARATUS (coin sorting G07D3/00; coin testing G07D5/00; {handling coins or paper currencies apart from payment activated apparatus G07D; payment architectures, schemes or protocols G06Q20/00})
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A projected image item tracking system that analyzes projected camera images to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed on them may then be attributed to a shopper near the area. Projected images may be combined to generate a 3D volume difference between the state of the area before and after shopper interaction. The volume difference may be calculated using plane-sweep stereo, or using convolutional neural networks. Because these methods may be computationally intensive, the system may first localize a change volume where items appear to have been displaced, and then generate a volume difference only within that change volume. This optimization results in significant savings in power consumption and in more rapid identification of items. The 3D volume difference may also indicate the quantity of items displaced, for example from a vertical stack.

## Claims

### Claim 1

A projected image item tracking system comprising:
a processor coupled to
a sensor configured to generate
an enter signal when a shopper reaches into or towards an item storage area in a store, wherein said item storage area comprises a volume in said store that contains items; and,
an exit signal when said shopper retracts from said item storage area; and,
a plurality of cameras oriented to view said item storage area;
wherein said processor is configured to
obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said enter signal;
obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said exit signal;
project said plurality of before images onto two or more surfaces contained in said volume associated with said item storage area to generate a plurality of projected before images corresponding to each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
project said plurality of after images onto said two or more surfaces to generate a plurality of projected after images corresponding to said each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
analyze
said plurality of projected before images, and
said plurality of projected after images, to
identify an item of said items taken from or put into said item storage area between said enter signal and said exit signal; and,
associate said item with said shopper;
wherein said two or more surfaces are at different positions along a sweep axis in said volume associated with said item storage area.

### Claim 2 (depends on 1)

The system of claim 1, wherein said sweep axis is vertical.

### Claim 3 (depends on 2)

The system of claim 2, wherein said two or more surfaces are at different heights along said sweep axis.

### Claim 4 (depends on 1)

The system of claim 1, wherein said two or more surfaces comprise two or more planes intersected with said volume associated with said item storage area.

### Claim 5 (depends on 4)

The system of claim 4, wherein said two or more planes are horizontal.

### Claim 6 (depends on 1)

The system of claim 1, wherein said volume associated with said item storage area comprises a 3D volume between two horizontally oriented shelves.

### Claim 7 (depends on 6)

The system of claim 6, wherein said two or more surfaces are between said two horizontally oriented shelves.

### Claim 8 (depends on 7)

The system of claim 7, wherein said two or more surfaces are parallel to said two horizontally oriented shelves.

### Claim 9 (depends on 1)

The system of claim 1, wherein
said plurality of cameras oriented to view said item storage area comprises C cameras, wherein C is a number greater than or equal to two;
said two or more surfaces contained in said volume associated with said item storage area comprises S surfaces, wherein S is a number greater than or equal to two;
said plurality of projected before images comprises C×S projected before images; and,
said plurality of projected after images comprises C×S projected after images.

### Claim 10 (depends on 9)

The system of claim 9, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 11 (depends on 10)

The system of claim 10, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, each image difference of said C×S image differences comprising pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 12 (depends on 11)

The system of claim 11, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in said each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 13 (depends on 1)

The system of claim 1, further comprising
a modular shelf comprising
said plurality of cameras oriented to view said item storage area;
a right-facing camera mounted on or proximal to a left edge of said modular shelf;
a left-facing camera mounted on or proximal to a right edge of said modular shelf;
a shelf processor; and,
a network switch;
wherein said processor comprises a network of computing devices, said computing devices comprising
a store processor; and,
said shelf processor.

### Claim 14 (depends on 13)

The system of claim 13, wherein
said sensor comprises said right-facing camera and said left-facing camera; and,
said processor is further configured to
analyze images from said right-facing camera and said left-facing camera to detect when said shopper reaches into or towards said item storage area, and to generate said enter signal; and,
analyze images from said right-facing camera and said left-facing camera to detect when said shopper retracts from said item storage area, and to generate said exit signal.

### Claim 15 (depends on 13)

The system of claim 13, wherein
said shelf processor comprises or is coupled to a memory; and,
said shelf processor is configured to
receive images from said plurality of cameras and store said images in said memory;
when said shelf processor receives or generates said enter signal, retrieve said plurality of before images from said memory.

### Claim 16 (depends on 15)

The system of claim 15, wherein said shelf processor is further configured to
when said shelf processor receives or generates said enter signal, transmit said plurality of before images from said memory to said store processor; and,
when said shelf processor receives or generates said exit signal, receive said plurality of after images from said plurality of cameras or obtain said plurality of after images from said memory and transmit said plurality of after images to said store processor.

### Claim 17 (depends on 1)

The system of claim 1, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
input at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images into a neural network trained to output said item of said items taken from or put into said item storage area between said enter signal and said exit signal.

### Claim 18 (depends on 17)

The system of claim 17, wherein said neural network is further trained to output an action that indicates whether said item of said items is taken from or is put into said item storage area between said enter signal and said exit signal.

### Claim 19 (depends on 18)

The system of claim 18, wherein said neural network comprises
a feature extraction layer, wherein
said feature extraction layer applied to each of said at least a portion of said plurality of projected before images outputs before image features; and
said feature extraction layer applied to each of said at least a portion of said plurality of projected after images outputs after image features;
a differencing layer applied to said before image features and said after image features associated with each camera of said plurality of cameras, wherein said differencing layer outputs feature differences associated with said each camera;
one or more convolutional layers applied to said feature differences associated with said each camera of said plurality of cameras;
an item classifier layer applied to an output of said one or more convolutional layers; and,
an action classifier layer applied to said output of said one or more convolutional layers.

### Claim 20 (depends on 1)

The system of claim 1, further comprising
a second plurality of cameras in said store oriented to view shoppers in said store, wherein said shoppers comprise said shopper;
wherein said processor is further configured to
receive a time sequence of images from each camera of said second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;
analyze said time sequence of images to
determine a sequence of locations of said shopper in said store during said time period; and
calculate a field of influence volume around each location of said sequence of locations; and,
when said field of influence volume intersects said item storage area at a time between said enter signal and said exit signal,
associate said item with said shopper.

### Claim 21 (depends on 20)

The system of claim 20, wherein
said second plurality of cameras in said store comprise a plurality of ceiling cameras mounted on a ceiling of said store;
said analyze said time sequence of images comprises
project said time sequence of images from each ceiling camera onto a plane parallel to a floor of said store, to form a time sequence of projected images corresponding to each ceiling camera;
analyze said time sequence of projected images corresponding to each ceiling camera to
determine said sequence of locations of said shopper in said store during said time period; and
calculate said field of influence volume around each location of said sequence of locations.

### Claim 22 (depends on 21)

The system of claim 21, wherein said each ceiling camera of said plurality of ceiling cameras is a fisheye camera.

### Claim 23 (depends on 21)

The system of claim 21, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
subtract a store background image from each projected image of said projected images captured at said each time to form a corresponding plurality of masks at said each time;
combine said plurality of masks at said each time to form a combined mask; and,
identify a location of said shopper at said each time as a high intensity location in said combined mask.

### Claim 24 (depends on 21)

The system of claim 21, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
input said projected images captured at said each time into a machine learning system that outputs an intensity map, wherein
said intensity map comprises a likelihood at each location that said shopper is at said location.

### Claim 25 (depends on 24)

The system of claim 24, wherein said determine said sequence of locations of said shopper in said store during said time period further comprises
input into said machine learning system a position map corresponding to each ceiling camera of said plurality of ceiling cameras, wherein a value of said position map at a location is a function of a distance between said location on said plane and said each ceiling camera.

### Claim 26 (depends on 25)

The system of claim 25, wherein
said plurality of cameras oriented to view said item storage area comprises C cameras, wherein C is a number greater than or equal to two;
said two or more surfaces contained in said volume associated with said item storage area comprises S surfaces, wherein S is a number greater than or equal to two;
said plurality of projected before images comprises C×S projected before images; and,
said plurality of projected after images comprises C×S projected after images.

### Claim 27 (depends on 26)

The system of claim 26, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 28 (depends on 27)

The system of claim 27, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, where said C×S image differences comprise pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 29 (depends on 28)

The system of claim 28, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 30

A projected image item tracking system comprising:
a processor coupled to
a sensor configured to generate
an enter signal when a shopper reaches into or towards an item storage area in a store, wherein said item storage area comprises a volume in said store that contains items; and,
an exit signal when said shopper retracts from said item storage area; and,
a plurality of cameras oriented to view said item storage area;
wherein said processor is configured to
obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said enter signal;
obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said exit signal;
project said plurality of before images onto two or more surfaces contained in said volume associated with said item storage area to generate a plurality of projected before images corresponding to each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
project said plurality of after images onto said two or more surfaces to generate a plurality of projected after images corresponding to said each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
analyze
said plurality of projected before images, and
said plurality of projected after images, to
identify an item of said items taken from or put into said item storage area between said enter signal and said exit signal; and,
associate said item with said shopper;
wherein
said plurality of cameras oriented to view said item storage area comprises C cameras, wherein C is a number greater than or equal to two;
said two or more surfaces contained in said volume associated with said item storage area comprises S surfaces, wherein S is a number greater than or equal to two;
said plurality of projected before images comprises C×S projected before images; and,
said plurality of projected after images comprises C×S projected after images.

### Claim 31 (depends on 30)

The system of claim 30, wherein said two or more surfaces are at different positions along a sweep axis in said volume associated with said item storage area.

### Claim 32 (depends on 31)

The system of claim 31, wherein said sweep axis is vertical.

### Claim 33 (depends on 32)

The system of claim 32, wherein said two or more surfaces are at different heights along said sweep axis.

### Claim 34 (depends on 30)

The system of claim 30, wherein said two or more surfaces comprise two or more planes intersected with said volume associated with said item storage area.

### Claim 35 (depends on 34)

The system of claim 34, wherein said two or more planes are horizontal.

### Claim 36 (depends on 30)

The system of claim 30, wherein said volume associated with said item storage area comprises a 3D volume between two horizontally oriented shelves.

### Claim 37 (depends on 36)

The system of claim 36, wherein said two or more surfaces are between said two horizontally oriented shelves.

### Claim 38 (depends on 37)

The system of claim 37, wherein said two or more surfaces are parallel to said two horizontally oriented shelves.

### Claim 39 (depends on 30)

The system of claim 30, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 40 (depends on 39)

The system of claim 39, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, each image difference of said C×S image differences comprising pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 41 (depends on 40)

The system of claim 40, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in said each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 42 (depends on 30)

The system of claim 30, further comprising
a modular shelf comprising
said plurality of cameras oriented to view said item storage area;
a right-facing camera mounted on or proximal to a left edge of said modular shelf;
a left-facing camera mounted on or proximal to a right edge of said modular shelf;
a shelf processor; and,
a network switch;
wherein said processor comprises a network of computing devices, said computing devices comprising
a store processor; and,
said shelf processor.

### Claim 43 (depends on 42)

The system of claim 42, wherein
said sensor comprises said right-facing camera and said left-facing camera; and,
said processor is further configured to
analyze images from said right-facing camera and said left-facing camera to detect when said shopper reaches into or towards said item storage area, and to generate said enter signal; and,
analyze images from said right-facing camera and said left-facing camera to detect when said shopper retracts from said item storage area, and to generate said exit signal.

### Claim 44 (depends on 42)

The system of claim 42, wherein
said shelf processor comprises or is coupled to a memory; and,
said shelf processor is configured to
receive images from said plurality of cameras and store said images in said memory;
when said shelf processor receives or generates said enter signal, retrieve said plurality of before images from said memory.

### Claim 45 (depends on 44)

The system of claim 44, wherein said shelf processor is further configured to
when said shelf processor receives or generates said enter signal, transmit said plurality of before images from said memory to said store processor; and,
when said shelf processor receives or generates said exit signal, receive said plurality of after images from said plurality of cameras or obtain said plurality of after images from said memory and transmit said plurality of after images to said store processor.

### Claim 46 (depends on 30)

The system of claim 30, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
input at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images into a neural network trained to output said item of said items taken from or put into said item storage area between said enter signal and said exit signal.

### Claim 47 (depends on 46)

The system of claim 46, wherein said neural network is further trained to output an action that indicates whether said item of said items is taken from or is put into said item storage area between said enter signal and said exit signal.

### Claim 48 (depends on 47)

The system of claim 47, wherein said neural network comprises
a feature extraction layer, wherein
said feature extraction layer applied to each of said at least a portion of said plurality of projected before images outputs before image features; and
said feature extraction layer applied to each of said at least a portion of said plurality of projected after images outputs after image features;
a differencing layer applied to said before image features and said after image features associated with each camera of said plurality of cameras, wherein said differencing layer outputs feature differences associated with said each camera;
one or more convolutional layers applied to said feature differences associated with said each camera of said plurality of cameras;
an item classifier layer applied to an output of said one or more convolutional layers; and,
an action classifier layer applied to said output of said one or more convolutional layers.

### Claim 49 (depends on 30)

The system of claim 30, further comprising
a second plurality of cameras in said store oriented to view shoppers in said store, wherein said shoppers comprise said shopper;
wherein said processor is further configured to
receive a time sequence of images from each camera of said second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;
analyze said time sequence of images to
determine a sequence of locations of said shopper in said store during said time period; and
calculate a field of influence volume around each location of said sequence of locations; and,
when said field of influence volume intersects said item storage area at a time between said enter signal and said exit signal,
associate said item with said shopper.

### Claim 50 (depends on 49)

The system of claim 49, wherein
said second plurality of cameras in said store comprise a plurality of ceiling cameras mounted on a ceiling of said store;
said analyze said time sequence of images comprises
project said time sequence of images from each ceiling camera onto a plane parallel to a floor of said store, to form a time sequence of projected images corresponding to each ceiling camera;
analyze said time sequence of projected images corresponding to each ceiling camera to
determine said sequence of locations of said shopper in said store during said time period; and
calculate said field of influence volume around each location of said sequence of locations.

### Claim 51 (depends on 50)

The system of claim 50, wherein said each ceiling camera of said plurality of ceiling cameras is a fisheye camera.

### Claim 52 (depends on 50)

The system of claim 50, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
subtract a store background image from each projected image of said projected images captured at said each time to form a corresponding plurality of masks at said each time;
combine said plurality of masks at said each time to form a combined mask; and,
identify a location of said shopper at said each time as a high intensity location in said combined mask.

### Claim 53 (depends on 50)

The system of claim 50, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
input said projected images captured at said each time into a machine learning system that outputs an intensity map, wherein
said intensity map comprises a likelihood at each location that said shopper is at said location.

### Claim 54 (depends on 53)

The system of claim 53, wherein said determine said sequence of locations of said shopper in said store during said time period further comprises
input into said machine learning system a position map corresponding to each ceiling camera of said plurality of ceiling cameras, wherein a value of said position map at a location is a function of a distance between said location on said plane and said each ceiling camera.

### Claim 55 (depends on 30)

The system of claim 30, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 56 (depends on 55)

The system of claim 55, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, where said C×S image differences comprise pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 57 (depends on 56)

The system of claim 56, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 58

A projected image item tracking system comprising:
a processor coupled to
a sensor configured to generate
an enter signal when a shopper reaches into or towards an item storage area in a store, wherein said item storage area comprises a volume in said store that contains items; and,
an exit signal when said shopper retracts from said item storage area; and,
a plurality of cameras oriented to view said item storage area;
wherein said processor is configured to
obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said enter signal;
obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said exit signal;
project said plurality of before images onto two or more surfaces contained in said volume associated with said item storage area to generate a plurality of projected before images corresponding to each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
project said plurality of after images onto said two or more surfaces to generate a plurality of projected after images corresponding to said each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
analyze
said plurality of projected before images, and
said plurality of projected after images, to
identify an item of said items taken from or put into said item storage area between said enter signal and said exit signal; and,
associate said item with said shopper;
wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
input at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images into a neural network trained to output said item of said items taken from or put into said item storage area between said enter signal and said exit signal;
wherein said neural network is further trained to output an action that indicates whether said item of said items is taken from or is put into said item storage area between said enter signal and said exit signal;
wherein said neural network comprises
a feature extraction layer, wherein
said feature extraction layer applied to each of said at least a portion of said plurality of projected before images outputs before image features; and
said feature extraction layer applied to each of said at least a portion of said plurality of projected after images outputs after image features;
a differencing layer applied to said before image features and said after image features associated with each camera of said plurality of cameras, wherein said differencing layer outputs feature differences associated with said each camera;
one or more convolutional layers applied to said feature differences associated with said each camera of said plurality of cameras;
an item classifier layer applied to an output of said one or more convolutional layers; and,
an action classifier layer applied to said output of said one or more convolutional layers.

### Claim 59 (depends on 58)

The system of claim 58, wherein said two or more surfaces are at different positions along a sweep axis in said volume associated with said item storage area.

### Claim 60 (depends on 59)

The system of claim 59, wherein said sweep axis is vertical.

### Claim 61 (depends on 60)

The system of claim 60, wherein said two or more surfaces are at different heights along said sweep axis.

### Claim 62 (depends on 58)

The system of claim 58, wherein said two or more surfaces comprise two or more planes intersected with said volume associated with said item storage area.

### Claim 63 (depends on 62)

The system of claim 62, wherein said two or more planes are horizontal.

### Claim 64 (depends on 58)

The system of claim 58, wherein said volume associated with said item storage area comprises a 3D volume between two horizontally oriented shelves.

### Claim 65 (depends on 64)

The system of claim 64, wherein said two or more surfaces are between said two horizontally oriented shelves.

### Claim 66 (depends on 65)

The system of claim 65, wherein said two or more surfaces are parallel to said two horizontally oriented shelves.

### Claim 67 (depends on 58)

The system of claim 58, wherein
said plurality of cameras oriented to view said item storage area comprises C cameras, wherein C is a number greater than or equal to two;
said two or more surfaces contained in said volume associated with said item storage area comprises S surfaces, wherein S is a number greater than or equal to two;
said plurality of projected before images comprises C×S projected before images; and,
said plurality of projected after images comprises C×S projected after images.

### Claim 68 (depends on 67)

The system of claim 67, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 69 (depends on 68)

The system of claim 68, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, each image difference of said C×S image differences comprising pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 70 (depends on 69)

The system of claim 69, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in said each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 71 (depends on 58)

The system of claim 58, further comprising
a modular shelf comprising
said plurality of cameras oriented to view said item storage area;
a right-facing camera mounted on or proximal to a left edge of said modular shelf;
a left-facing camera mounted on or proximal to a right edge of said modular shelf;
a shelf processor; and,
a network switch;
wherein said processor comprises a network of computing devices, said computing devices comprising
a store processor; and,
said shelf processor.

### Claim 72 (depends on 71)

The system of claim 71, wherein
said sensor comprises said right-facing camera and said left-facing camera; and,
said processor is further configured to
analyze images from said right-facing camera and said left-facing camera to detect when said shopper reaches into or towards said item storage area, and to generate said enter signal; and,
analyze images from said right-facing camera and said left-facing camera to detect when said shopper retracts from said item storage area, and to generate said exit signal.

### Claim 73 (depends on 71)

The system of claim 71, wherein
said shelf processor comprises or is coupled to a memory; and,
said shelf processor is configured to
receive images from said plurality of cameras and store said images in said memory;
when said shelf processor receives or generates said enter signal, retrieve said plurality of before images from said memory.

### Claim 74 (depends on 73)

The system of claim 73, wherein said shelf processor is further configured to
when said shelf processor receives or generates said enter signal, transmit said plurality of before images from said memory to said store processor; and,
when said shelf processor receives or generates said exit signal, receive said plurality of after images from said plurality of cameras or obtain said plurality of after images from said memory and transmit said plurality of after images to said store processor.

### Claim 75 (depends on 58)

The system of claim 58, further comprising
a second plurality of cameras in said store oriented to view shoppers in said store, wherein said shoppers comprise said shopper;
wherein said processor is further configured to
receive a time sequence of images from each camera of said second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;
analyze said time sequence of images to
determine a sequence of locations of said shopper in said store during said time period; and
calculate a field of influence volume around each location of said sequence of locations; and,
when said field of influence volume intersects said item storage area at a time between said enter signal and said exit signal,
associate said item with said shopper.

### Claim 76 (depends on 75)

The system of claim 75, wherein
said second plurality of cameras in said store comprise a plurality of ceiling cameras mounted on a ceiling of said store;
said analyze said time sequence of images comprises
project said time sequence of images from each ceiling camera onto a plane parallel to a floor of said store, to form a time sequence of projected images corresponding to each ceiling camera;
analyze said time sequence of projected images corresponding to each ceiling camera to
determine said sequence of locations of said shopper in said store during said time period; and
calculate said field of influence volume around each location of said sequence of locations.

### Claim 77 (depends on 76)

The system of claim 76, wherein said each ceiling camera of said plurality of ceiling cameras is a fisheye camera.

### Claim 78 (depends on 76)

The system of claim 76, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera, subtract a store background image from each projected image of said projected images
captured at said each time to form a corresponding plurality of masks at said each time;
combine said plurality of masks at said each time to form a combined mask; and,
identify a location of said shopper at said each time as a high intensity location in said combined mask.

### Claim 79 (depends on 76)

The system of claim 76, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
input said projected images captured at said each time into a machine learning system that outputs an intensity map, wherein
said intensity map comprises a likelihood at each location that said shopper is at said location.

### Claim 80 (depends on 79)

The system of claim 79, wherein said determine said sequence of locations of said shopper in said store during said time period further comprises
input into said machine learning system a position map corresponding to each ceiling camera of said plurality of ceiling cameras, wherein a value of said position map at a location is a function of a distance between said location on said plane and said each ceiling camera.

### Claim 81 (depends on 80)

The system of claim 80, wherein
said plurality of cameras oriented to view said item storage area comprises C cameras, wherein C is a number greater than or equal to two;
said two or more surfaces contained in said volume associated with said item storage area comprises S surfaces, wherein S is a number greater than or equal to two;
said plurality of projected before images comprises C×S projected before images; and,
said plurality of projected after images comprises C×S projected after images.

### Claim 82 (depends on 81)

The system of claim 81, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 83 (depends on 82)

The system of claim 82, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, where said C×S image differences comprise pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 84 (depends on 83)

The system of claim 83, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 85

A projected image item tracking system comprising:
a processor coupled to
a sensor configured to generate
an enter signal when a shopper reaches into or towards an item storage area in a store, wherein said item storage area comprises a volume in said store that contains items; and,
an exit signal when said shopper retracts from said item storage area; and,
a plurality of cameras oriented to view said item storage area;
wherein said processor is configured to
obtain a plurality of before images captured by said plurality of cameras, each before image of said plurality of before images corresponding to a camera of said plurality of cameras, wherein said each before image is captured at a time before said enter signal;
obtain a plurality of after images captured by said plurality of cameras, each after image of said plurality of after images corresponding to a camera of said plurality of cameras, wherein said each after image is captured at a time after said exit signal;
project said plurality of before images onto two or more surfaces contained in said volume associated with said item storage area to generate a plurality of projected before images corresponding to each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
project said plurality of after images onto said two or more surfaces to generate a plurality of projected after images corresponding to said each combination of a surface of said two or more surfaces and a camera of said plurality of cameras;
analyze
said plurality of projected before images, and
said plurality of projected after images, to
identify an item of said items taken from or put into said item storage area between said enter signal and said exit signal; and,
associate said item with said shopper;
a second plurality of cameras in said store oriented to view shoppers in said store, wherein said shoppers comprise said shopper;
wherein said processor is further configured to
receive a time sequence of images from each camera of said second plurality of cameras in said store, wherein said time sequence of images from each camera is captured over a time period;
analyze said time sequence of images to
determine a sequence of locations of said shopper in said store during said time period; and
calculate a field of influence volume around each location of said sequence of locations; and,
when said field of influence volume intersects said item storage area at a time between said enter signal and said exit signal,
associate said item with said shopper;
wherein
said second plurality of cameras in said store comprise a plurality of ceiling cameras mounted on a ceiling of said store;
said analyze said time sequence of images comprises
project said time sequence of images from each ceiling camera onto a plane parallel to a floor of said store, to form a time sequence of projected images corresponding to each ceiling camera;
analyze said time sequence of projected images corresponding to each ceiling camera to
determine said sequence of locations of said shopper in said store during said time period; and
calculate said field of influence volume around each location of said sequence of locations;
wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
input said projected images captured at said each time into a machine learning system that outputs an intensity map, wherein
said intensity map comprises a likelihood at each location that said shopper is at said location;
wherein said determine said sequence of locations of said shopper in said store during said time period further comprises
input into said machine learning system a position map corresponding to each ceiling camera of said plurality of ceiling cameras, wherein a value of said position map at a location is a function of a distance between said location on said plane and said each ceiling camera;
wherein
said plurality of cameras oriented to view said item storage area comprises C cameras, wherein C is a number greater than or equal to two;
said two or more surfaces contained in said volume associated with said item storage area comprises S surfaces, wherein S is a number greater than or equal to two;
said plurality of projected before images comprises C×S projected before images; and,
said plurality of projected after images comprises C×S projected after images.

### Claim 86 (depends on 85)

The system of claim 85, wherein said two or more surfaces are at different positions along a sweep axis in said volume associated with said item storage area.

### Claim 87 (depends on 86)

The system of claim 86, wherein said sweep axis is vertical.

### Claim 88 (depends on 87)

The system of claim 87, wherein said two or more surfaces are at different heights along said sweep axis.

### Claim 89 (depends on 85)

The system of claim 85, wherein said two or more surfaces comprise two or more planes intersected with said volume associated with said item storage area.

### Claim 90 (depends on 89)

The system of claim 89, wherein said two or more planes are horizontal.

### Claim 91 (depends on 85)

The system of claim 85, wherein said volume associated with said item storage area comprises a 3D volume between two horizontally oriented shelves.

### Claim 92 (depends on 91)

The system of claim 91, wherein said two or more surfaces are between said two horizontally oriented shelves.

### Claim 93 (depends on 92)

The system of claim 92, wherein said two or more surfaces are parallel to said two horizontally oriented shelves.

### Claim 94 (depends on 85)

The system of claim 85, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 95 (depends on 94)

The system of claim 94, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, each image difference of said C×S image differences comprising pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 96 (depends on 95)

The system of claim 95, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in said each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.

### Claim 97 (depends on 85)

The system of claim 85, further comprising
a modular shelf comprising
said plurality of cameras oriented to view said item storage area;
a right-facing camera mounted on or proximal to a left edge of said modular shelf;
a left-facing camera mounted on or proximal to a right edge of said modular shelf;
a shelf processor; and,
a network switch;
wherein said processor comprises a network of computing devices, said computing devices comprising
a store processor; and,
said shelf processor.

### Claim 98 (depends on 97)

The system of claim 97, wherein
said sensor comprises said right-facing camera and said left-facing camera; and,
said processor is further configured to
analyze images from said right-facing camera and said left-facing camera to detect when said shopper reaches into or towards said item storage area, and to generate said enter signal; and,
analyze images from said right-facing camera and said left-facing camera to detect when said shopper retracts from said item storage area, and to generate said exit signal.

### Claim 99 (depends on 97)

The system of claim 97, wherein
said shelf processor comprises or is coupled to a memory; and,
said shelf processor is configured to
receive images from said plurality of cameras and store said images in said memory;
when said shelf processor receives or generates said enter signal, retrieve said plurality of before images from said memory.

### Claim 100 (depends on 99)

The system of claim 99, wherein said shelf processor is further configured to
when said shelf processor receives or generates said enter signal, transmit said plurality of before images from said memory to said store processor; and,
when said shelf processor receives or generates said exit signal, receive said plurality of after images from said plurality of cameras or obtain said plurality of after images from said memory and transmit said plurality of after images to said store processor.

### Claim 101 (depends on 85)

The system of claim 85, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
input at least a portion of said plurality of projected before images and at least a portion of said plurality of projected after images into a neural network trained to output said item of said items taken from or put into said item storage area between said enter signal and said exit signal.

### Claim 102 (depends on 101)

The system of claim 101, wherein said neural network is further trained to output an action that indicates whether said item of said items is taken from or is put into said item storage area between said enter signal and said exit signal.

### Claim 103 (depends on 102)

The system of claim 102, wherein said neural network comprises
a feature extraction layer, wherein
said feature extraction layer applied to each of said at least a portion of said plurality of projected before images outputs before image features; and
said feature extraction layer applied to each of said at least a portion of said plurality of projected after images outputs after image features;
a differencing layer applied to said before image features and said after image features associated with each camera of said plurality of cameras, wherein said differencing layer outputs feature differences associated with said each camera;
one or more convolutional layers applied to said feature differences associated with said each camera of said plurality of cameras;
an item classifier layer applied to an output of said one or more convolutional layers; and,
an action classifier layer applied to said output of said one or more convolutional layers.

### Claim 104 (depends on 85)

The system of claim 85, wherein said each ceiling camera of said plurality of ceiling cameras is a fisheye camera.

### Claim 105 (depends on 85)

The system of claim 85, wherein said determine said sequence of locations of said shopper in said store during said time period comprises
for each time in said time sequence of projected images corresponding to each ceiling camera,
subtract a store background image from each projected image of said projected images captured at said each time to form a corresponding plurality of masks at said each time;
combine said plurality of masks at said each time to form a combined mask; and,
identify a location of said shopper at said each time as a high intensity location in said combined mask.

### Claim 106 (depends on 85)

The system of claim 85, wherein said analyze said plurality of projected before images and said plurality of projected after images comprises
perform C×S image comparisons between said C×S projected before images and said C×S projected after images; and,
combine said C×S image comparisons across said C cameras to form S change regions, wherein each change region of said S change regions is associated with a surface of said S surfaces.

### Claim 107 (depends on 106)

The system of claim 106, wherein
said C×S image comparisons comprise C×S image differences between said C×S projected before images and said C×S projected after images, where said C×S image differences comprise pixels; and,
each image difference of said C×S image differences corresponds to a corresponding camera of said C cameras and a corresponding surface of said S surfaces.

### Claim 108 (depends on 107)

The system of claim 107, wherein said combine said C×S image comparisons across said C cameras to form S change regions comprises
weight each pixel of said pixels in each image difference of said C×S image differences based on a distance between
a point in said corresponding surface corresponding to said each pixel, and
a location of said corresponding camera,
to form C×S weighted image differences; and,
average C weighted image differences corresponding to each surface of said S surfaces to form said S change regions.