# US 12,223,726 B2 · Electronic device and related methods for monitoring objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,726 B2
- Document ID: US12223726B2
- Status: Granted
- Kind: B2
- Application number: 17/693,642
- Filed: 2022-03-14
- Published: 2022-10-06
- Granted: 2025-02-11
- Earliest claimed: 2021-03-30
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Peter Exner; Hannes Bergkvist; Anders Isberg; Peter Blomqvist
- Web page: https://patentgrep.com/patent/US12223726B2

## CPC

- G06Q 10/0635: Risk analysis of enterprise or organisation activities
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06Q 10/20: Administration of product repair or maintenance
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G08B 21/18: Status alarms (G08B21/02 takes precedence)
- G06T 2207/30196: Human being; Person
- G06T 2207/30232: Surveillance
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

An electronic device configured to monitor an object. The electronic device is configured to generate a first area indicative of a person, generate a second area indicative of an object, determine, based on the first area and the second area, an interaction parameter indicative of an interaction between the person and the object, determine, based on the interaction parameter, a status indicator of the object, generate a maintenance indicator based on the status indicator, and output the maintenance indicator.

## Claims

### Claim 1

An electronic device configured to monitor an object, the electronic device comprising:
memory circuitry comprising association data indicative of a predetermined association between an interactor object and an interacted object;
interface circuitry; and
processor circuitry, wherein the processor circuitry is configured to:
generate, based on image data comprising one or more frames indicative of a person and/or the interactor object, a first area indicative of the person;
generate, based on the image data, a second area indicative of the interactor object;
determine, based on the first area, the second area, and the association data, an interaction parameter indicative of an indirect interaction between the person and the interacted object, wherein the interacted object is not directly interacted with by the person;
determine, based on the interaction parameter, a status indicator of the interacted object;
generate a maintenance indicator based on the status indicator; and
output the maintenance indicator.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the electronic device is configured to obtain, from an image sensor, the image data.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein the generation of the first area comprises applying a pose detection scheme to the one or more frames of the image data.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the generation of the second area comprises applying an object extraction scheme to the one or more frames of the image data.

### Claim 5 (depends on 4)

The electronic device of claim 4, wherein the generation of the second area comprises generating, based on the object extraction scheme, a bounding box indicative of the first area.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein the status indicator comprises one or more usage parameters comprising a duration of the interaction and/or a frequency of the interaction and/or a movement of the object and/or a change of size of the second area and/or an effect by the object on another object.

### Claim 7 (depends on 1)

The electronic device of claim 1, wherein the electronic device comprises an image sensor internal to the electronic device and/or external to the electronic device.

### Claim 8 (depends on 1)

The electronic device of claim 1, wherein the electronic device is configured to access an object database, the object database comprising a plurality of objects and a monitoring indicator associated with each of the plurality of objects, wherein the plurality of objects comprises the object.

### Claim 9 (depends on 8)

The electronic device of claim 8, wherein the maintenance indicator is based at least partly on the monitoring indicator of the object.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein the processor circuitry is configured to generate, based on the image data, a surface characteristic indicator of the object, and wherein the maintenance indicator is based on the surface characteristic indicator.

### Claim 11 (depends on 1)

The electronic device of claim 1, wherein the maintenance indicator is indicative of the object needing to be cleaned, moved, replaced, and/or repaired.

### Claim 12 (depends on 1)

The electronic device of claim 1, wherein the determining of the interaction parameter comprises determining a distance between the first area and the second area, and in accordance with the distance being below a distance threshold, the processor circuitry is configured to determine the status indicator.

### Claim 13 (depends on 1)

The electronic device of claim 1, wherein the processor circuitry is configured to determine if the status indicator is above a threshold and, in accordance with the status indicator being above the threshold, the processor circuitry is configured to generate the maintenance indicator.

### Claim 14 (depends on 1)

The electronic device of claim 1, wherein the processor circuitry is configured to determine if the status indicator is below a threshold and, in accordance with the status indicator being below the threshold, the processor circuitry is configured to generate the maintenance indicator.

### Claim 15 (depends on 1)

The electronic device of claim 1, wherein the outputting of the maintenance indicator comprises outputting the maintenance indicator to a server.

### Claim 16 (depends on 1)

The electronic device of claim 1, wherein the outputting of the maintenance indicator comprises outputting the maintenance indicator for an illumination of the object via a projector and/or for viewing a maintenance signal indicative of the maintenance indicator on an augmented reality device.

### Claim 17

A method, performed by an electronic device to monitor an interacted object, the method comprising:
generating, based on image data comprising one or more frames indicative of a person and/or an interactor object associated with the interacted object by association data, a first area indicative of the person;
generating, based on the image data, a second area indicative of the interactor object;
determining, based on the first area, the second area, and the association data, an interaction parameter indicative of an indirect interaction between the person and the interacted object, wherein the interacted object is not directly interacted with by the person;
determining, based on the interaction parameter, a status indicator of the interacted object;
generating a maintenance indicator based on the status indicator; and
outputting the maintenance indicator.

### Claim 18 (depends on 17)

The method of claim 17, the method comprising obtaining, from an image sensor, the image data.

### Claim 19 (depends on 17)

The method of claim 17, wherein the generating of the first area comprises applying a pose detection scheme to the one or more frames of the image data.

### Claim 20 (depends on 17)

The method of claim 17, wherein the generating of the second area comprises applying an object extraction scheme to the one or more frames of the image data.