# US 2024/0185691 A1 · Monitoring device and monitoring method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0185691 A1
- Document ID: US20240185691A1
- Status: Published application
- Kind: A1
- Application number: 18/443,290
- Filed: 2024-02-15
- Published: 2024-06-06
- Earliest claimed: 2021-11-25
- Assignees: Toshiba Tec Kabushiki Kaisha
- Applicants: Toshiba Tec Kabushiki Kaisha
- Inventors: Yuishi TAKENO; Keita YAMAZAKI
- Web page: https://patentgrep.com/patent/US20240185691A1

## CPC

- G07G 3/003: {Anti-theft control (theft alarms G08B13/00)}
- G08B 13/19613: {Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion}
- G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 20/382: {insuring higher security of transaction}
- G06Q 20/4016: {involving fraud or risk level assessment in transaction processing}
- G06V 10/40: Extraction of image or video features
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/60: Type of objects
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G07G 3/00: Alarm indicators, e.g. bells
- G07G 3/006: {False operation}
- G08B 13/196: using television cameras
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)}
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A monitoring device for monitoring a customer operating a checkout terminal in a store, includes a network interface configured to communicate with the checkout terminal configured to perform checkout processing and an attendant terminal, a camera interface through which an image captured by a camera is received, the camera being installed so as to image the customer operating the checkout terminal, and a processor configured to, upon receipt of a first command indicating that the checkout processing has been started from the checkout terminal, begin monitoring of the image captured by the camera, upon detection of presence of the customer in the image, acquire feature information representing features of the customer from the image, and upon detection of absence of the customer in the image after the feature information is acquired, control the network interface to output to the attendant terminal an alert signal with the acquired feature information.

## Claims

### Claim 1

A monitoring device for monitoring a customer operating a checkout terminal to perform checkout processing in a store, the monitoring device comprising:
a memory including first and second memory regions;
an interface circuit connectable to the checkout terminal;
a camera interface through which an image captured by a camera is received, the camera being positioned so as to image the customer operating the checkout terminal; and
a processor configured to:
upon receipt of a first command indicating that the checkout processing has been started from the checkout terminal, initialize the first and second memory regions and begin monitoring the image captured by the camera,
repeatedly attempt to acquire skeleton information representing features of a customer shown in the image, and when the skeleton information is acquired, store the acquired skeleton information in the first memory region,
execute same-customer determination processing for determining whether the customer indicated by latest skeleton information that has been stored most recently in the first memory region is same as the customer indicated by previous skeleton information that was stored in the first memory region before the most recent skeleton information,
upon determining that the customer indicated by the most recent skeleton information is same as the customer indicated by the previous skeleton information:
calculate a posture score from the most recent skeleton information, and
store, in the second memory region, the calculated posture score in association with the image corresponding to the most recent skeleton information when no posture score is stored in the second memory region or when the calculated posture store is higher than another posture score that has been stored in the second memory region, and
upon detecting absence of the customer in the image after receiving the first command and before receiving a second command indicating that the checkout processing has been completed, control the interface circuit to output an alert signal and the image stored most recently in the second memory region.

### Claim 2 (depends on 1)

The monitoring device according to claim 1, wherein the processor is configured to perform skeleton estimation processing on the image to calculate the posture score.

### Claim 3 (depends on 2)

The monitoring device according to claim 2, wherein the posture score is higher when more body parts of the customer are recognized in the image through the skeleton estimation processing.

### Claim 4 (depends on 1)

The monitoring device according to claim 1, wherein the processor is configured to override said another posture score and another image associated therewith stored in the second memory region with the calculated posture and the image associated therewith.

### Claim 5 (depends on 1)

The monitoring device according to claim 1, wherein the processor is configured to perform object recognition on the image to recognize one or more commodities to be purchased by the customer during the monitoring.

### Claim 6 (depends on 1)

The monitoring device according to claim 1, wherein the alert signal causes an external terminal to display an alert message including identification information of the checkout terminal and indicating that a customer left the checkout terminal before the checkout processing has been completed.

### Claim 7 (depends on 1)

The monitoring device according to claim 1, wherein the camera is positioned above the checkout terminal.

### Claim 8 (depends on 1)

The monitoring device according to claim 1, wherein the checkout terminal is a self-service type checkout terminal.

### Claim 9

A monitoring method performed by a monitoring device for monitoring a customer operating a checkout terminal to perform checkout processing in a store, the monitoring method comprising:
receiving from the checkout terminal a first command indicating that checkout processing has been started from the checkout terminal;
initializing first and second memory regions and beginning monitoring an image captured by a camera positioned so as to image the customer operating the checkout terminal;
repeatedly attempting to acquire skeleton information representing features of a customer shown in the image, and when the skeleton information is acquired, storing the acquired skeleton information in the first memory region;
executing same-customer determination processing for determining whether the customer indicated by latest skeleton information that has been stored most recently in the first memory region is same as the customer indicated by previous skeleton information that was stored in the first memory region before the most recent skeleton information;
upon determining that the customer indicated by the most recent skeleton information is same as the customer indicated by the previous skeleton information:
calculating a posture score from the most recent skeleton information, and
storing, in the second memory region, the calculated posture score in association with the image corresponding to the most recent skeleton information when no posture score is stored in the second memory region or when the calculated posture store is higher than another posture score that has been stored in the second memory region; and
detecting absence of the customer in the image after receiving the first command and before receiving a second command indicating that the checkout processing has been completed, and outputting an alert signal and the image stored most recently in the second memory region.

### Claim 10 (depends on 9)

The monitoring method according to claim 9, wherein calculating the posture score includes performing skeleton estimation processing on the image.

### Claim 11 (depends on 10)

The monitoring method according to claim 10, wherein the posture score is higher when more body parts of the customer are recognized in the image through the skeleton estimation processing.

### Claim 12 (depends on 9)

The monitoring method according to claim 9, wherein storing the calculated posture score includes overriding said another posture score and another image associated therewith with the calculated posture score and the image associated therewith.

### Claim 13 (depends on 9)

The monitoring method according to claim 9, further comprising:
performing object recognition on the image to recognize one or more commodities to be purchased by the customer during the monitoring.

### Claim 14 (depends on 9)

The monitoring method according to claim 9, wherein the alert signal causes an external terminal to display an alert message including identification information of the checkout terminal and indicating that a customer left the checkout terminal before the checkout processing has been completed.

### Claim 15 (depends on 9)

The monitoring method according to claim 9, wherein the camera is positioned above the checkout terminal.

### Claim 16 (depends on 9)

The monitoring method according to claim 9, wherein the checkout terminal is a self-service type checkout terminal.

### Claim 17

A non-transitory computer readable medium storing a program causing a computer to execute a monitoring method for monitoring a customer operating a checkout terminal to perform checkout processing in a store, the monitoring method comprising:
receiving from the checkout terminal a first command indicating that checkout processing has been started from the checkout terminal;
initializing first and second memory regions and beginning monitoring an image captured by a camera positioned so as to image the customer operating the checkout terminal;
repeatedly attempting to acquire skeleton information representing features of a customer shown in the image, and when the skeleton information is acquired, storing the acquired skeleton information in the first memory region;
executing same-customer determination processing for determining whether the customer indicated by latest skeleton information that has been stored most recently in the first memory region is same as the customer indicated by previous skeleton information that was stored in the first memory region before the most recent skeleton information;
upon determining that the customer indicated by the most recent skeleton information is same as the customer indicated by the previous skeleton information:
calculating a posture score from the most recent skeleton information, and
storing, in the second memory region, the calculated posture score in association with the image corresponding to the most recent skeleton information when no posture score is stored in the second memory region or when the calculated posture store is higher than another posture score that has been stored in the second memory region; and
detecting absence of the customer in the image after receiving the first command and before receiving a second command indicating that the checkout processing has been completed, and outputting an alert signal and the image stored most recently in the second memory region.

### Claim 18 (depends on 17)

The computer readable medium according to claim 17, wherein calculating the posture score includes performing skeleton estimation processing on the image.

### Claim 19 (depends on 18)

The computer readable medium according to claim 18, wherein the posture score is higher when more body parts of the customer are recognized in the image through the skeleton estimation processing.

### Claim 20 (depends on 17)

The computer readable medium according to claim 17, wherein storing the calculated posture score includes overriding said another posture score and another image associated therewith with the calculated posture score and the image associated therewith.
