# US 2023/0267299 A1 · Artificial intelligence assisted wearable

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267299 A1
- Document ID: US20230267299A1
- Status: Published application
- Kind: A1
- Application number: 18/142,401
- Filed: 2023-05-02
- Published: 2023-08-24
- Earliest claimed: 2019-09-13
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Brian Stephen CLAIRE
- Web page: https://patentgrep.com/patent/US20230267299A1

## CPC

- G06F 3/005: {Input arrangements through a video camera}
- G06N 3/004: Artificial life, i.e. computing arrangements simulating life
- A45C 15/00: Purses, bags, luggage or other receptacles covered by groups A45C1/00 - A45C11/00, combined with other {objects or} articles (A45C1/08, A45C3/14, {A45C11/321, A45C13/28} take precedence)
- A45F 3/04: Sacks or packs carried on the body by means of two straps passing over the two shoulders
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/3329: Natural language query formulation
- G06V 40/172: {Classification, e.g. identification}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- H04R 1/028: {associated with devices performing functions other than acoustics, e.g. electric candles}
- H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor
- G01S 19/13: Receivers
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- A45C: PURSES; LUGGAGE; HAND CARRIED BAGS
- A45F: TRAVELLING OR CAMP EQUIPMENT: SACKS OR PACKS CARRIED ON THE BODY
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

The description relates to artificial intelligence assisted wearables, such as backpacks. An example backpack may include sensors, such as a microphone and a camera. The backpack may receive a contextual voice command from a user. The contextual voice command may include a non-explicit reference to an object in an environment. The backpack may use the sensors to sense the environment, use an artificial intelligence engine to identify the object in the environment, and use a digital assistant to perform a contextual task in response to the contextual voice command. The contextual task may relate to the object in the environment. The backpack may output a response to the contextual voice command to the user.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A hands-free digital assistant device, comprising:
a camera;
a microphone;
a speaker;
a network interface;
a processor; and
a storage having instructions which, when executed by the processor, cause the processor to:
receive a contextual voice command from the user via the microphone, wherein the contextual voice command uses a non-explicit reference to an object in an environment;
capture an image of the environment including the object via the camera;
cause transmission of the contextual voice command and the image to an engine via the network interface to cause a contextual task to be performed, the contextual task including a computerized action relating to the object;
receive a response associated with the contextual task that was performed based at least on the contextual voice command; and
output the response to the user via the speaker.

### Claim 22 (depends on 21)

The device of claim 21, further comprising:
a compass,
wherein the instructions further cause the processor to sense a direction that the user is facing via the compass.

### Claim 23 (depends on 21)

The device of claim 21, further comprising:
a global positioning system (GPS) unit,
wherein the instructions further cause the processor to determine a location of the user via the GPS unit.

### Claim 24

A system for hands-free digital assistant device, comprising:
a wearable device;
a sensor attached to the wearable device, the sensor being fixed relative to a body of a user and capable of sensing an environment;
a processor; and
a storage having instructions which, when executed by the processor, cause the processor to:
receive a contextual voice command that includes a pronoun for a contextual signal that refers to an object in the environment;
detect the object in the environment using the sensor;
cause an engine to perform a contextual task relating to the object in response to the contextual voice command; and
output a response associated with the contextual task to the user.

### Claim 25 (depends on 24)

The system of claim 24, wherein the wearable device includes a backpack.

### Claim 26 (depends on 24)

The system of claim 24, wherein the sensor includes a camera.

### Claim 27 (depends on 26)

The system of claim 26, wherein the camera is located in a strap of the wearable device and facing a front direction of the user.

### Claim 28 (depends on 24)

The system of claim 24, further comprising:
a speaker for outputting the response, wherein the response includes auditory feedback.

### Claim 29 (depends on 24)

The system of claim 24, further comprising:
a light emitting diode for outputting the response, wherein the response includes visual feedback.

### Claim 30 (depends on 24)

The system of claim 24, further comprising:
a haptic actuator for outputting the response, wherein the response includes haptic feedback.

### Claim 31 (depends on 24)

The system of claim 24, further comprising:
a network interface for connecting to a network through a companion device that is capable of connecting to the network.

### Claim 32 (depends on 24)

The system of claim 24, further comprising:
a battery for charging a companion device.

### Claim 33

A method for hands-free digital assistance, comprising:
receiving a contextual voice command that makes a reference to an object in an environment without explicitly identifying the object;
capturing a recording of the environment including the object;
using an engine to determine an identification of the object and to interpret the contextual voice command based at least on the identification of the object; and
causing a contextual task to be performed in response to the contextual voice command, the contextual task including a computerized action relating to the object in the environment.

### Claim 34 (depends on 33)

The method of claim 33, further comprising:
using a speech recognition module to interpret the contextual voice command.

### Claim 35 (depends on 33)

The method of claim 33, wherein the recording includes one or more of: an audio recording, an image recording, and/or a video recording.

### Claim 36 (depends on 33)

The method of claim 33, further comprising:
using an image recognition module to determine the identification of the object in the recording.

### Claim 37 (depends on 33)

The method of claim 33, further comprising:
using a text recognition module to determine the identification of the object in the recording.

### Claim 38 (depends on 33)

The method of claim 33, further comprising:
using a facial recognition module to determine the identification of the object in the recording.

### Claim 39 (depends on 33)

The method of claim 33, further comprising:
using a cognitive module to determine the contextual task to be performed in response to the contextual voice command.

### Claim 40 (depends on 33)

The method of claim 33, further comprising:
generating a response associated with the contextual task; and
transmitting the response to be output to a user.
