# US 2021/0232787 A1 · Recording multimodal user interaction with items in real and digital environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232787 A1
- Document ID: US20210232787A1
- Status: Published application
- Kind: A1
- Application number: 17/259,718
- Filed: 2019-10-10
- Published: 2021-07-29
- Earliest claimed: 2018-10-11
- Assignees: Scroble S.A.
- Applicants: Scroble S.A.
- Inventors: Anna Ivanovna Salewski
- Web page: https://patentgrep.com/patent/US20210232787A1

## CPC

- G06K 7/1417: {2D bar codes}
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06K 7/10366: {the interrogation device being adapted for miscellaneous applications}
- G06K 19/06037: {multi-dimensional coding}
- G06Q 10/0833: Tracking
- G06Q 30/0185: {Product, service or business identity fraud}
- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- G06Q 30/0205: {based on location or geographical consideration}
- G06Q 30/06: Buying, selling or leasing transactions
- G06Q 30/0639: {Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility}
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A method and a corresponding system are disclosed for recording multimodal user interaction with an item in a networked system. The networked system includes a server device network- connected to one or more client user device(s). A data record is generated for the item, comprising data representative thereof and including at least image data depicting the item. A unique identifier code is associated with the data record, wherein the association of the unique identifier code with the image data in the record defines an avatar of the item in the system. Copies of the avatar are distributed to client user devices in the network, and the unique identifier code is encoded into a machine-readable tag secured to each physical embodiment of the item outside the network. At each client user device, each digital user interaction with an avatar and each analogue user interaction with a physical item is logged. At the server, each logged user interaction is matched against the item data record and recorded.

## Claims

### Claim 1

A method of recording multimodal user interaction with an item in a networked system, the networked system comprising at least one server device connected to at least one client device across at least one network, the method comprising the steps of generating a data record comprising data representative of the item, the data including at least image data depicting the item;
generating and associating a unique identifier code with the data record, wherein associating at least the image data in the data record with the unique identifier code defines an avatar of the item in the networked system;
distributing one or more copies of the avatar to the or each client device;
encoding the unique identifier code into a machine-readable tag and securing the tag to each physical embodiment of the item outside the networked system;
at the or each client device, logging digital user interaction with the avatar and analogue user interaction with the physical item, wherein logging analogue user interaction comprises reading the machine-readable tag with the client device; and
matching and recording each logged user interaction against the item data record at the or each server device.

### Claim 2 (depends on 1)

A method according to claim 1, wherein the step of logging analogue user interaction comprises the further steps of
determining a geographical location of the client device; and
associating the determined location with the reading of the machine-readable tag.

### Claim 3 (depends on 1)

A method according to claim 1, wherein the step of logging digital user interaction comprises the further step of
determining a network location of the distributed avatar;
optionally determining a geographical location of the network location;
associating at least the network location with the digital user interaction.

### Claim 4 (depends on 1)

A method according to claim 1, comprising the further steps of
generating and storing a user account for each user of the networked system, each user account having at least one respective type selected from a group comprising at least an item provider and an item consumer; and
authenticating access of each client device to the networked system through logging each user's client device into the user's respective user account.

### Claim 5 (depends on 4)

A method according to claim 4, wherein
the item provider of the group of selectable user account types further comprises an item manufacturer type and an item retailer type; and/or
the group of selectable user account types further comprises an item referrer and a third party data supplier.

### Claim 6 (depends on 4)

A method according to claim 4, comprising the further steps of matching and recording each logged user interaction against the respective user account of the client device logging the user interaction.

### Claim 7 (depends on 4)

A method according to claim 4, wherein a data record comprising data representative of the item can only be generated at a client device logged into a user account of the item provider type.

### Claim 8 (depends on 4)

A method according to claim 4, comprising the further step of removably linking at least one data record with at least one user account.

### Claim 9 (depends on 8)

A method according to claim 8, comprising the further steps of generating and storing a list of each data record linked with a user account; and updating the list according to logged user interaction recorded against the user account.

### Claim 10 (depends on 8)

A method according to claim 8, wherein the step of distributing one or more copies of the avatar to the or each client device further comprises distributing the said copies according to the list of data record(s) linked with the user account authenticating a client device.

### Claim 11 (depends on 2, 8)

A method according to claim 8 when dependent on claim 2, comprising the further steps of
determining a first geographical location, being a location of a client device logged into a user account;
determining a second geographical location, being a location of least one physical item having a respective data record linked with the user account; and
when the first and second geographical locations are within a predetermined distance of each other, distributing a copy of the avatar corresponding to the data record of the item to the client device logged into the user account, as a notice of item proximity.

### Claim 12 (depends on 9)

A method according to claim 9, wherein the logged user interaction at the list updating step is digital or analogue user interaction representative of a transaction for a physical embodiment of the item outside the networked system.

### Claim 13 (depends on 1)

A method according to claim 1, comprising the further step of
providing each client device with a messaging interface to all other client devices in the networked system;
allowing each client device to message one or more remote client devices in the networked system through the interface; and
preventing a message from being sent in the networked system until and unless at least one logged user interaction is associated with the message.

### Claim 14 (depends on 1)

A method according to claim 1, wherein
the step of generating a data record comprising data representative of the item comprises the further step of encoding the item data record in a blockchain data structure; and
the step of recording each logged user interaction against the item data record comprises the further step of encoding each matched logged user interaction against the item data record in the blockchain data structure.

### Claim 15 (depends on 14)

A method according to claim 14, wherein data representative of the item in the data record encoded in the Blockchain data structure includes at least image data depicting the item and authentication data guaranteeing a manufacturing origin of each physical embodiment of the item.

### Claim 16

A system for recording multimodal user interaction with an item, the system comprising at least one server device connected to at least one client device across at least one network,
wherein the or each server is configured to
generate a data record comprising data representative of the item, the data including at least image data depicting the item;
generate and associate a unique identifier code with the data record, wherein associating at least the image data in the data record with the unique identifier code defines an avatar of the item in the system;
distribute one or more copies of the avatar to the or each client device; and
match and record each user interaction logged at one or more client device against the item data record;
the system further comprising at least one machine-readable tag, into which the unique identifier code is encoded and which is secured to each physical embodiment of the item outside the system; and
wherein the or each client device is configured both to log digital user interaction with the avatar and to log analogue user interaction with the physical item by reading the machine-readable tag secured thereto.

### Claim 17

A computer program product for recording multimodal user interaction with an item in a networked system comprising at least one server device connected to at least one client device across at least one network, wherein the computer program product is executable by the or each server device and configures same to
generate a data record comprising data representative of the item, the data including at least image data depicting the item;
generate and associate a unique identifier code with the data record, wherein associating at least the image data in the data record with the unique identifier code defines an avatar of the item in the system;
distribute one or more copies of the avatar to the or each client device; and
match and record each user interaction logged at one or more client device against the item data record;
the server-generated unique identifier code being encoded into at least one machine-readable tag secured to each physical embodiment of the item outside the networked system; and
the or each client device being configured both to log digital user interaction with the avatar and to log analogue user interaction with the physical item by reading the machine-readable tag secured thereto.
