# US 12,217,240 B1 · Bar code detection and authentication for contactless transactions using wearable multimedia device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,240 B1
- Document ID: US12217240B1
- Status: Granted
- Kind: B1
- Application number: 17/334,645
- Filed: 2021-05-28
- Granted: 2025-02-04
- Earliest claimed: 2020-05-29
- Assignees: Humane, Inc.
- Applicants: Humane, Inc.
- Inventors: Imran A. Chaudhri; Patrick Gates; Bethany Bongiorno; Yanir Nulman; Monique Relova
- Web page: https://patentgrep.com/patent/US12217240B1

## CPC

- G06Q 20/3276: {using a pictured code, e.g. barcode or QR-code, being read by the M-device}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 21/44: Program or device authentication
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06Q 20/1085: {involving automatic teller machines [ATMs]}
- G06Q 20/18: involving self-service terminals [SST], vending machines, kiosks or multimedia terminals
- G06Q 20/321: {using wearable devices}
- G06Q 20/3278: {RFID or NFC payments by means of M-devices}
- G06Q 20/3821: {Electronic credentials}
- G06Q 20/401: {Transaction verification}
- G06Q 20/4014: {Identity check for transactions}
- G06Q 20/40145: {Biometric identity checks}
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 20/20: in augmented reality scenes
- G06V 20/64: Three-dimensional [3D] objects
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G07F 19/20: {Automatic teller machines [ATMs]}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07F: COIN-FREED OR LIKE APPARATUS (coin sorting G07D3/00; coin testing G07D5/00; {handling coins or paper currencies apart from payment activated apparatus G07D; payment architectures, schemes or protocols G06Q20/00})

## Abstract

In an embodiment, a wearable multimedia device, comprises: a housing configured to attach to clothing of a user; a wide field-of-view camera embedded in the housing; one or more processors; memory storing instructions that when executed by the one or more processors, causes the one or more processors to perform operations comprising: capturing, using the camera, an image of a scene that includes a bar code displayed by a terminal device; determining an intent of the user to read the bar code; in accordance with the user's intent to read the barcode: cropping the barcode from the image of the scene; establishing, using the bar code, a communication channel with the terminal device; receiving, using the communication channel, transaction data from the terminal device; and responsive to the transaction data, sending, using the communication channel, authentication credentials to the terminal device.

## Claims

### Claim 1

An apparatus, comprising:
a wearable multimedia device worn by a user comprising:
a housing;
a wide field-of-view camera embedded in the housing;
a depth sensor;
one or more processors; and
a memory storing instructions that when executed by the one or more processors, causes the one or more processors to perform operations comprising:
capturing, using the camera, an image of a scene that includes a plurality of unlabeled objects and a user gesture;
capturing, using the depth sensor, depth data of the scene;
performing semantic segmentation on the image of the scene to predict an object mask for each unlabeled object;
performing instance segmentation on pixel data within the each object mask;
labeling, based on the instance segmentation, that one of the plurality of unlabeled objects is a contactless terminal device;
associating, with sensor fusion, an intent of the user to engage with the contactless terminal device based at least in part on the labeled contactless terminal device, the depth data, and the user gesture, where the user gesture is the user pointing in a direction of the contactless terminal device;
establishing a communication channel between the wearable multimedia device and the contactless terminal device;
receiving, using the communication channel, data from the contactless terminal device;
responsive to the received data, sending, using the communication channel, authentication credentials to the contactless terminal device;
receiving, using the communication channel, access from the contactless terminal device based at least in part on the authentication credentials; and
responsive to the received access to the contactless terminal device, performing an interaction with the contactless terminal device.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising:
searching for a bar code in the image of the scene; and automatically cropping the bar code from the image of the scene; and
establishing, using the bar code, the communication channel between the wearable multimedia device and the terminal device.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the bar code is a quick response (QR) code.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein the bar code is displayed on a monitor of the contactless terminal device.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the image of the scene captured by the camera and the depth data of the scene captured by a depth sensor of the wearable multimedia device are used to determine that the user is gesturing.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein determining the intent of the user to engage with the contactless terminal device further comprises determining that the gesture is the user pointing their hand or finger in the direction of the contactless terminal device.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the image of the scene captured by the camera and user speech captured by a microphone of the wearable multimedia device at the scene are used to determine the intent of the user to engage with the contactless terminal device.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the image of the scene captured by the camera, user speech captured by a microphone of the wearable multimedia device, and depth data captured by the depth sensor of the wearable multimedia device at the scene are used to determine the intent of the user to engage with the contactless terminal device.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein at least one of location, speed or direction of the user captured by at least one sensor of the wearable multimedia device is used to determine the intent of the user to engage with the contactless terminal device.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the image of the scene captured by the camera or depth data captured by a depth sensor of the wearable multimedia device is used to determine a type of the contactless terminal device.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein a past history of communications with the contactless terminal device is used to determine the intent of the user to engage with the contactless terminal device.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the authentication credentials include biometric sensor data.

### Claim 13 (depends on 1)

The apparatus of claim 1, further comprising:
processing, using a machine learning model, the image of the scene to detect and label the contactless terminal device in the scene.

### Claim 14 (depends on 1)

The apparatus of claim 1, further comprising:
after authentication of the wearable multimedia device by the contactless terminal device using the authentication credentials, capturing a control gesture of the user; and
sending control data to the contactless terminal device for controlling the contactless terminal device based on the control gesture.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the contactless terminal device is a media playback device and the control gesture is a swipe gesture for changing content played on the media playback device.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein a speed of the swipe gesture controls a speed of the content changing.

### Claim 17

A method comprising:
capturing, using a wide field-of-view camera of a wearable multimedia device worn by a user, an image of a scene that includes a plurality of unlabeled objects and a user gesture, wherein the wearable multimedia device comprises a housing, the wide field-of-view camera embedded in the housing, a depth sensor, one or more processors, and a memory storing instructions that are executed by the one or more processors;
capturing, by the one or more processors using the depth sensor, depth data of the scene;
performing, by the one or more processors, semantic segmentation on the image of the scene to predict an object mask for each unlabeled object;
performing, by the one or more processors, instance segmentation on pixel data within the each object mask;
labeling, by the one or more processors based on the instance segmentation, that one of the plurality of unlabeled objects is a contactless terminal device;
associating, by the one or more processors with sensor fusion, an intent of the user to engage with the contactless terminal device based at least in part on the user gesture, the depth data and the labeled contactless terminal device, where the user gesture is the user pointing in a direction of the contactless terminal device;
establishing, by the one or more processors, a communication channel between the wearable multimedia device and the contactless terminal device;
receiving, by the one or more processors using the communication channel, data from the contactless terminal device;
responsive to the received data, sending, using the communication channel, authentication credentials to the contactless terminal device;
receiving, by the one or more processors using the communication channel, access from the contactless terminal device based at least in part on the authentication credentials; and
responsive to the received access to the contactless terminal device, performing, by the one or more processors, an interaction with the contactless terminal device.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
searching, by the one or more processors, for a bar code in the image of the scene;
automatically cropping, by the one or more processors, the bar code from the image of the scene; and
establishing, by the one or more processors using the bar code, the communication channel between the wearable multimedia device and the terminal device.

### Claim 19 (depends on 17)

The method of claim 17, wherein determining the intent of the user to engage with the contactless terminal device further comprises determining, by the one or more processors, that the gesture is the user pointing their hand or finger in the direction of the contactless terminal device.

### Claim 20 (depends on 17)

The method of claim 17, wherein the image of the scene captured the camera and user speech captured by a microphone of the wearable multimedia device at the scene are used to determine, by the one or more processors, the intent of the user to engage with the contactless terminal device.