# US 2021/0279970 A1 · System and method for presenting a virtual store shelf that emulates a physical store shelf

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279970 A1
- Document ID: US20210279970A1
- Status: Published application
- Kind: A1
- Application number: 17/330,661
- Filed: 2021-05-26
- Published: 2021-09-09
- Earliest claimed: 2019-10-25
- Assignees: 7-Eleven, Inc.
- Applicants: 7-Eleven, Inc.
- Inventors: Shahmeer Ali Mirza; Matthew Raymond Magee; Sarath Vakachrla; Trong Nghia Nguyen; Franklin Philips
- Web page: https://patentgrep.com/patent/US20210279970A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06F 3/002: {Specific input/output arrangements not covered by G06F3/01 - G06F3/16 (other optical apparatus G02B27/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 20/20: in augmented reality scenes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus includes a display, interface, and processor. The interface receives a camera feed from a camera directed at a first physical rack located in a physical store. The processor displays virtual racks assigned to physical racks. The processor receives an indication of an event associated with the first physical rack and accordingly displays a first virtual rack, which includes first and second virtual shelves. The first and second virtual shelves include virtual items that emulate physical items located on physical shelves of the first physical rack. The processor additionally displays a recording of the camera feed, which depicts the event associated with the first physical rack.

## Claims

### Claim 1

An apparatus comprising:
a display;
an interface configured to receive a live rack camera feed from a rack camera located in a physical store, the rack camera directed at a first physical rack of a set of physical racks located in the physical store; and
a processor communicatively coupled to the display, the processor configured to:
receive an indication of an event associated with the first physical rack, the event comprising a physical person located in the physical store interacting with the first physical rack;
in response to receiving the indication of the event associated with the first physical rack:
determine that a first virtual rack is assigned to the first physical rack with which the event is associated;
display, in a first region of the display, the first virtual rack assigned to the first physical rack, the first virtual rack comprising a first virtual shelf and a second virtual shelf, the first virtual shelf comprising a first virtual item, the second virtual shelf comprising a second virtual item, wherein the first virtual item comprises a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item comprises a graphical representation of a second physical item located on a second physical shelf of the first physical rack; and
display, in a second region of the display, a rack video segment corresponding to a recording of the live rack camera feed from a starting timestamp to an ending timestamp, wherein the rack video segment depicts the event associated with the first physical rack.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the processor is further configured to display, in a third region of the display, a virtual layout of a virtual store, the virtual layout configured to emulate a physical layout of the physical store, the virtual layout comprising the first virtual rack assigned to the first physical rack and a second virtual rack assigned to a second physical rack, wherein an arrangement of the first virtual rack and the second virtual rack in the virtual layout is configured to emulate an arrangement of the first physical rack and the second physical rack in the physical layout.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein:
the interface is further configured to receive a second rack camera feed from a second rack camera located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks located in the physical store; and
the processor is further configured to:
receive an indication of a second event associated with the second physical rack, the second event comprising the person interacting with the second physical rack;
in response to receiving the indication of the second event associated with the second physical rack:
display, in the first region of the display, the second virtual rack, the second virtual rack comprising a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack; and
display, in the second region of the display, a second rack video segment corresponding to a recording of the second rack camera feed from a second starting timestamp to a second ending timestamp, wherein the second rack video segment depicts the second event associated with the second physical rack.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein:
the event associated with the first physical rack comprises the first physical item being selected from a first zone of the first physical shelf of the first physical rack, wherein a weight sensor coupled to the first zone of the first physical shelf is configured to detect the first physical item being selected from the first physical shelf; and
the indication of the event comprises information received from the weight sensor.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein, in response to displaying the first virtual rack, the processor is further configured to highlight the first virtual item on the first virtual shelf.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein:
the interface is further configured to receive a video feed corresponding to a first camera located in a physical store; and
the processor is further configured to:
store a first video segment in a memory, the first video segment assigned to a first person and capturing a portion of a shopping session of the first person in the physical store, the first video segment corresponding to a recording of the video feed from the starting timestamp to the ending timestamp;
assign a slider bar to the first video segment, wherein:
the slider bar controls a playback progress of the first video segment;
the event occurs at a first time;
the slider bar comprises a marker located at the first time on the slider bar, the marker associated with the event occurring at the first time, the marker comprising metadata comprising an item identification number assigned to the first virtual item, a rack identification number assigned to the first virtual rack, and a zone identification number assigned to a zone of the first virtual rack; and
the indication of the event comprises the playback progress of the first video segment reaching the marker; and
in response to displaying the first virtual rack, highlight the first virtual item on the first virtual rack, based on the metadata.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein:
the slider bar comprises a second marker located at a second time on the slider bar, the second marker associated with a second event occurring at the second time, the second event associated with the second physical rack, the second marker comprising second metadata comprising a second item identification number assigned to a third virtual item, a second rack identification number assigned to the second virtual rack, and a second zone identification number assigned to a second zone of the second virtual rack; and
the processor is further configured to:
determine that the playback progress of the first video segment reached the second marker; and
in response to determining that the playback progress of the first video segment reached the second marker:
display, in the first region of the display, the second virtual rack, the second virtual rack comprising a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack; and
highlight the third virtual item on the second virtual rack, based on the second metadata.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the indication of the event associated with the first physical rack comprises information generated by an algorithm in response to a detection of the event based on a set of inputs received from sensors located within the physical store.

### Claim 9

A method comprises:
receiving a live rack camera feed from a rack camera located in a physical store, the rack camera directed at a first physical rack of a set of physical racks located in the physical store;
receiving an indication of an event associated with the first physical rack, the event comprising a physical person located in the physical store interacting with the first physical rack;
in response to receiving the indication of the event associated with the first physical rack:
determining that a first virtual rack is assigned to the first physical rack with which the event is associated;
displaying, in a first region of the display, the first virtual rack assigned to the first physical rack, the first virtual rack comprising a first virtual shelf and a second virtual shelf, the first virtual shelf comprising a first virtual item, the second virtual shelf comprising a second virtual item, wherein the first virtual item comprises a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item comprises a graphical representation of a second physical item located on a second physical shelf of the first physical rack; and
displaying, in a second region of the display, a rack video segment corresponding to a recording of the live rack camera feed from a starting timestamp to an ending timestamp, wherein the rack video segment depicts the event associated with the first physical rack.

### Claim 10 (depends on 9)

The method of claim 9, further comprising displaying, in a third region of the display, a virtual layout of a virtual store, the virtual layout configured to emulate a physical layout of the physical store, the virtual layout comprising a first virtual rack assigned to a first physical rack and a second virtual rack assigned to a second physical rack, wherein an arrangement of the first virtual rack and the second virtual rack in the virtual layout is configured to emulate an arrangement of the first physical rack and the second physical rack in the physical layout.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
receiving a second rack camera feed from a second rack camera located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks located in the physical store;
receiving an indication of a second event associated with the second physical rack, the second event comprising the person interacting with the second physical rack;
in response to receiving the indication of the second event associated with the second physical rack:
displaying, in the first region of the display, the second virtual rack, the second virtual rack comprising a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack; and
displaying, in the second region of the display, a second rack video segment corresponding to a recording of the second rack camera feed from a second starting timestamp to a second ending timestamp, wherein the second rack video segment depicts the second event associated with the second physical rack.

### Claim 12 (depends on 9)

The method of claim 9, wherein:
the event associated with the first physical rack comprises the first physical item being selected from a first zone of the first physical shelf of the first physical rack, wherein a weight sensor coupled to the first zone of the first physical shelf is configured to detect the first physical item being selected from the first physical shelf; and
the indication of the event comprises information received from the weight sensor.

### Claim 13 (depends on 12)

The method of claim 12, further comprising highlighting the first virtual item on the first virtual shelf, in response to displaying the first virtual rack.

### Claim 14 (depends on 9)

The method of claim 9, further comprising:
receiving a video feed corresponding to a first camera located in a physical store;
storing a first video segment assigned to a first person and capturing a portion of a shopping session of the first person in the physical store, the first video segment corresponding to a recording of the video feed from the starting timestamp to the ending timestamp;
assigning a slider bar to the first video segment, wherein:
the slider bar controls a playback progress of the first video segment;
the event occurs at a first time;
the slider bar comprises a marker located at the first time on the slider bar, the marker associated with the event occurring at the first time, the marker comprising metadata comprising an item identification number assigned to the first virtual item, a rack identification number assigned to the first virtual rack, and a zone identification number assigned to a zone of the first virtual rack; and
the indication of the event comprises the playback progress of the first video segment reaching the marker; and
in response to displaying the first virtual rack, highlighting the first virtual item on the first virtual rack, based on the metadata.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
determining that the playback progress of the first video segment reached a second marker, wherein the slider bar comprises the second marker located at a second time on the slider bar, the second marker associated with a second event occurring at the second time, the second event associated with the second physical rack, the second marker comprising second metadata comprising a second item identification number assigned to a third virtual item, a second rack identification number assigned to the second virtual rack, and a second zone identification number assigned to a second zone of the second virtual rack; and
in response to determining that the playback progress of the first video segment reached the second marker:
displaying, in the first region of the display, the second virtual rack, the second virtual rack comprising a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack; and
highlighting the third virtual item on the second virtual rack, based on the second metadata.

### Claim 16 (depends on 9)

The method of claim 9, wherein the indication of the event associated with the first physical rack comprises information generated by an algorithm in response to a detection of the event based on a set of inputs received from sensors located within the physical store.

### Claim 17

A system comprising:
a display;
an interface operable to:
receive a first live rack camera feed from a first rack camera located in a physical store, the first rack camera directed at a first physical rack of a set of physical racks located in the physical store; and
receive a second live rack camera feed from a second rack camera located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks located in the physical store; and
a processor communicatively coupled to the display, the processor operable to:
receive an indication of an event associated with the first physical rack, the event comprising a physical person located in the physical store interacting with the first physical rack;
in response to receiving the indication of the event associated with the first physical rack:
determine that a first virtual rack is assigned to the first physical rack with which the event is associated;
display, in a first region of the display, the first virtual rack assigned to the first physical rack, the first virtual rack comprising a first virtual shelf and a second virtual shelf, the first virtual shelf comprising a first virtual item, the second virtual shelf comprising a second virtual item, wherein the first virtual item comprises a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item comprises a graphical representation of a second physical item located on a second physical shelf of the first physical rack; and
display, in a second region of the display, a rack video segment corresponding to a recording of the live rack camera feed from a starting timestamp to an ending timestamp, wherein the rack video segment depicts the event associated with the first physical rack;
receive an indication of a second event associated with the second physical rack, the second event comprising the person interacting with the second physical rack;
in response to receiving the indication of the second event associated with the second physical rack:
display, in the first region of the display, the second virtual rack, the second virtual rack comprising a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack; and
display, in the second region of the display, a second rack video segment corresponding to a recording of the second rack camera feed from a second starting timestamp to a second ending timestamp, wherein the second rack video segment depicts the second event associated with the second physical rack.

### Claim 18

The system of 17, wherein the processor is further configured to display, in a third region of the display, a virtual layout of a virtual store, the virtual layout configured to emulate a physical layout of the physical store, the virtual layout comprising a first virtual rack assigned to a first physical rack and a second virtual rack assigned to a second physical rack, wherein an arrangement of the first virtual rack and the second virtual rack in the virtual layout is configured to emulate an arrangement of the first physical rack and the second physical rack in the physical layout;

### Claim 19 (depends on 17)

The system of claim 17, wherein:
the event associated with the first physical rack comprises the first physical item being selected from a first zone of the first physical shelf of the first physical rack, wherein a weight sensor coupled to the first zone of the first physical shelf is configured to detect the first physical item being selected from the first physical shelf; and
the indication of the event comprises information received from the weight sensor.

### Claim 20 (depends on 19)

The system of claim 19, wherein, in response to displaying the first virtual rack, the processor is further operable to highlight the first virtual item on the first virtual shelf.

### Claim 21 (depends on 17)

The system of claim 17, further comprising a memory, wherein:
the interface is further operable to receive a video feed corresponding to a first camera located in a physical store; and
the processor is further operable to:
store a first video segment in the memory, the first video segment assigned to a first person and capturing a portion of a shopping session of the first person in the physical store, the first video segment corresponding to a recording of the video feed from the starting timestamp to the ending timestamp;
assign a slider bar to the first video segment, wherein:
the slider bar controls a playback progress of the first video segment;
the event occurs at a first time;
the slider bar comprises a marker located at the first time on the slider bar, the marker associated with the event occurring at the first time, the marker comprising metadata comprising an item identification number assigned to the first virtual item, a rack identification number assigned to the first virtual rack, and a zone identification number assigned to a zone of the first virtual rack; and
the indication of the event comprises the playback progress of the first video segment reaching the marker; and
in response to displaying the first virtual rack, highlight the first virtual item on the first virtual rack, based on the metadata.

### Claim 22 (depends on 21)

The system of claim 21, wherein:
the slider bar comprises a second marker located at a second time on the slider bar, the second marker associated with a second event occurring at the second time, the second event associated with the second physical rack, the second marker comprising second metadata comprising a second item identification number assigned to a third virtual item, a second rack identification number assigned to the second virtual rack, and a second zone identification number assigned to a second zone of the second virtual rack; and
the processor is further operable to:
determine that the playback progress of the first video segment reached the second marker; and
in response to determining that the playback progress of the first video segment reached the second marker:
display, in the first region of the display, the second virtual rack, the second virtual rack comprising a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack; and
highlight the third virtual item on the second virtual rack, based on the second metadata.

### Claim 23 (depends on 17)

The system of claim 17, wherein the indication of the event associated with the first physical rack comprises information generated by an algorithm in response to a detection of the event based on a set of inputs received from sensors located within the physical store.