# US 2021/0081749 A1 · Artificial intelligence assisted wearable

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081749 A1
- Document ID: US20210081749A1
- Status: Published application
- Kind: A1
- Application number: 16/570,058
- Filed: 2019-09-13
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Brian Stephen CLAIRE
- Web page: https://patentgrep.com/patent/US20210081749A1

## CPC

- G06F 3/005: {Input arrangements through a video camera}
- G06N 3/004: Artificial life, i.e. computing arrangements simulating life
- A45C 15/00: Purses, bags, luggage or other receptacles covered by groups A45C1/00 - A45C11/00, combined with other {objects or} articles (A45C1/08, A45C3/14, {A45C11/321, A45C13/28} take precedence)
- A45F 3/04: Sacks or packs carried on the body by means of two straps passing over the two shoulders
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/3329: Natural language query formulation
- G06V 40/172: {Classification, e.g. identification}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- H04R 1/028: {associated with devices performing functions other than acoustics, e.g. electric candles}
- H04R 1/08: Mouthpieces; {Microphones;} Attachments therefor
- G01S 19/13: Receivers
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L 2015/228: {of application context}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- A45C: PURSES; LUGGAGE; HAND CARRIED BAGS
- A45F: TRAVELLING OR CAMP EQUIPMENT: SACKS OR PACKS CARRIED ON THE BODY
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

The description relates to artificial intelligence assisted wearables, such as backpacks. An example backpack may include sensors, such as a microphone and a camera. The backpack may receive a contextual voice command from a user. The contextual voice command may include a non-explicit reference to an object in an environment. The backpack may use the sensors to sense the environment, use an artificial intelligence engine to identify the object in the environment, and use a digital assistant to perform a contextual task in response to the contextual voice command. The contextual task may relate to the object in the environment. The backpack may output a response to the contextual voice command to the user.

## Claims

### Claim 1

A backpack, comprising:
a strap including a camera that faces a front direction of a user when the user wears the backpack;
a microphone;
a speaker;
a network interface;
a processor; and
a storage having instructions which, when executed by the processor, cause the processor to:
receive a contextual voice command from the user via the microphone, the contextual voice command using a non-explicit reference to an object in an environment;
capture an image of the environment including the object via the camera;
transmit the contextual voice command and the image to an artificial intelligence engine via the network interface to cause a contextual task to be performed, the contextual task including a computerized action relating to the object;
receive a response associated with the contextual task that was performed based at least on the contextual voice command; and
output the response to the user via the speaker.

### Claim 2 (depends on 1)

The backpack of claim 1, further comprising:
a compass,
wherein the instructions further cause the processor to sense a direction that the user is facing via the compass.

### Claim 3 (depends on 1)

The backpack of claim 1, further comprising:
a global positioning system (GPS) unit,
wherein the instructions further cause the processor to determine a location of the user via the GPS unit.

### Claim 4

A system, comprising:
a wearable;
a sensor attached to the wearable, the sensor being fixed relative to a body of a user and capable of sensing an environment;
a processor; and
a storage having instructions which, when executed by the processor, cause the processor to:
receive a contextual voice command that includes a pronoun to refer to an object in the environment;
detect the object in the environment using the sensor;
cause an artificial intelligence engine to perform a contextual task relating to the object in response to the contextual voice command; and
output a response associated with the contextual task to the user.

### Claim 5 (depends on 4)

The system of claim 4, wherein the wearable includes a backpack.

### Claim 6 (depends on 4)

The system of claim 4, wherein the sensor includes a camera.

### Claim 7 (depends on 6)

The system of claim 6, wherein the camera is located in a strap of the wearable and facing a front direction of the user.

### Claim 8 (depends on 4)

The system of claim 4, further comprising:
a speaker for outputting the response, wherein the response includes auditory feedback.

### Claim 9 (depends on 4)

The system of claim 4, further comprising:
a light emitting diode for outputting the response, wherein the response includes visual feedback.

### Claim 10 (depends on 4)

The system of claim 4, further comprising:
a haptic actuator for outputting the response, wherein the response includes haptic feedback.

### Claim 11 (depends on 4)

The system of claim 4, further comprising:
a network interface for connecting to a network through a companion device that is capable of connecting to the network.

### Claim 12 (depends on 4)

The system of claim 4, further comprising:
a battery for charging a companion device.

### Claim 13

A method, comprising:
receiving a contextual voice command that references an object in an environment without explicitly identifying the object;
capturing a recording of the environment including the object;
using an artificial intelligence engine to determine an identification of the object and to interpret the contextual voice command based at least on the identification of the object; and
causing a contextual task to be performed in response to the contextual voice command, the contextual task including a computerized action relating to the object in the environment.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
using a speech recognition module to interpret the contextual voice command.

### Claim 15 (depends on 13)

The method of claim 13, wherein the recording includes one or more of: an audio recording, an image recording, and/or a video recording.

### Claim 16 (depends on 13)

The method of claim 13, further comprising:
using an image recognition module to determine the identification of the object in the recording.

### Claim 17 (depends on 13)

The method of claim 13, further comprising:
using a text recognition module to determine the identification of the object in the recording.

### Claim 18 (depends on 13)

The method of claim 13, further comprising:
using a facial recognition module to determine the identification of the object in the recording.

### Claim 19 (depends on 13)

The method of claim 13, further comprising:
using a cognitive module to determine the contextual task to be performed in response to the contextual voice command.

### Claim 20 (depends on 13)

The method of claim 13, further comprising:
generating a response associated with the contextual task; and
transmitting the response to be output to a user.
