# US 2013/0155237 A1 · Interacting with a mobile device within a vehicle using gestures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0155237 A1
- Document ID: US20130155237A1
- Status: Published application
- Kind: A1
- Application number: 13/327,787
- Filed: 2011-12-16
- Published: 2013-06-20
- Earliest claimed: 2011-12-16
- Assignees: Microsoft Corporation
- Inventors: Timothy S. Paek; Paramvir Bahl; Oliver H. Foehr
- Web page: https://patentgrep.com/patent/US20130155237A1

## CPC

- G06F 1/1632: {External expansion units, e.g. docking stations}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/26: using acoustic output
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- B60K 35/265: {Voice}
- B60K 35/80: Arrangements for controlling instruments
- B60K 35/85: Arrangements for transferring vehicle- or driver-related data
- B60K 2360/146: Instrument input by gesture
- B60K 2360/1464: 3D-gesture
- B60K 2360/148: Instrument input by voice
- B60K 2360/164: Infotainment
- B60K 2360/166: Navigation
- B60K 2360/167: Vehicle dynamics information
- B60K 2360/21: using cameras
- B60K 2360/334: Projection means
- B60K 2360/566: Mobile devices displaying vehicle information
- B60K 2360/573: Mobile devices controlling vehicle functions
- B60K 2360/583: Data transfer between instruments
- B60K 2360/5899: Internet
- B60K 2360/595: Data transfer involving internal databases
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A mobile device is described herein which includes functionality for recognizing gestures made by a user within a vehicle. The mobile device operates by receiving image information that captures a scene including objects within an interaction space. The interaction space corresponds to a volume that projects out from the mobile device in a direction of the user. The mobile device then determines, based on the image information, whether the user has performed a recognizable gesture within the interaction space, without touching the mobile device. The mobile device can receive the image information from a camera device that is an internal component of the mobile device and/or a camera device that is a component of a mount which secures the mobile device within the vehicle. In some implementations, one or more projectors provided by the mobile device and/or the mount may illuminate the interaction space.

## Claims

### Claim 1

A method for recognizing gestures using a mobile device that is mounted in a vehicle, the mobile device functioning as a handheld mobile device when not mounted in the vehicle, comprising:
receiving image information from at least one camera device,
the image information capturing a scene that includes an interaction space as part thereof, the interaction space comprising a volume having prescribed dimensions that projects out from the mobile device in a direction of a user who is operating the vehicle; and
determining, using a gesture recognition module, whether the user has performed a recognizable gesture within the interaction space, based on the image information,
wherein the gesture comprises one or more of: (a) a static pose made with at least one hand of the user without touching the mobile device; and (b) a dynamic movement made with said at least one hand of the user without touching the mobile device.

### Claim 2 (depends on 1)

The method of claim 1, wherein said determining comprises:
generating depth information based on the image information using a depth reconstruction technique; and
extracting a representation of said at least one hand that is positioned within the interaction space, based on the depth information.

### Claim 3 (depends on 1)

The method of claim 1, wherein said determining comprises:
projecting one or more beams of electromagnetic radiation, said one or more beams defining a region of increased relative illumination; and
extracting a representation of said at least one hand that is positioned within the interaction space by detecting an object having increased relative brightness in the image information.

### Claim 4 (depends on 1)

The method of claim 1, wherein said at least one camera is a component of the mobile device.

### Claim 5 (depends on 1)

The method of claim 1, wherein said at least one camera is a component of a mount that secures the mobile device within the vehicle.

### Claim 6 (depends on 1)

The method of claim 1, wherein said receiving of image information is performed in conjunction with irradiating the interaction space with electromagnetic radiation, using at least one projector.

### Claim 7 (depends on 6)

The method of claim 6, wherein said at least one projector is a component of the mobile device.

### Claim 8 (depends on 6)

The method of claim 6, wherein said at least one projector is a component of a mount that secures the mobile device within the vehicle.

### Claim 9 (depends on 1)

The method of claim 1, wherein said at least one camera device produces the image information in response to receipt of infrared spectrum radiation.

### Claim 10 (depends on 1)

The method of claim 1, wherein said at least one camera device contains a bandpass filter that diminishes visible spectrum radiation.

### Claim 11 (depends on 1)

The method of claim 1, further comprising defining the interaction space in a calibration procedure prior to said determining of the recognizable gesture.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
assessing performance of the gesture recognition module, to provide an assessed performance; and
dynamically adjusting at least one operational setting of the gesture recognition module based on the assessed performance.

### Claim 13 (depends on 12)

The method of claim 12, wherein said at least one operational setting is selected from:
at least one parameter that affects projection of electromagnetic radiation into the interaction space by at least one projector;
at least one parameter that affects receipt of the image information by said at least one camera device; and
a mode of image capture used by the gesture recognition module to recognize gestures.

### Claim 14 (depends on 1)

The method of claim 1, further comprising performing a control action in response to determining that the user has performed the gesture, the control action affecting a manner of operation of the mobile device.

### Claim 15 (depends on 14)

The method of claim 14, wherein the gesture is associated with a voice recognition mode, and wherein said performing of the control action comprises activating the voice recognition mode in response to determining that the user has performed the gesture.

### Claim 16

A mobile device for use within a vehicle, comprising:
input functionality configured to receive image information regarding objects within a scene, the scene including, as part thereof, an interaction space, the interaction space projecting out a prescribed distance from the mobile device within the vehicle,
the image information originating from one or more of:
an internal camera device that is an internal component of the mobile device; and
an external camera device that is a component of a mount which secures the mobile device within the vehicle; and
the input functionality also including a gesture recognition module configured to determine whether a user has made a gesture within the interaction space, based on one or more of:
depth information that is generated from the image information using a depth reconstruction technique; and
the image information itself without consideration of the depth information,
wherein the gesture comprises one or more of: (a) a static pose made with at least one hand of the user without touching the mobile device; and (b) a dynamic movement made with said at least one hand of the user without touching the mobile device.

### Claim 17

A mount for holding a mobile device, comprising:
a cradle for securing the mobile device; and
an imaging member including external camera functionality, the external camera functionality comprising:
at least one external camera device for receiving image information, the image information capturing a scene that includes an interaction space as part thereof, the interaction space comprising a volume having prescribed dimensions that projects out from the mobile device; and
an interface for providing the image information to input functionality provided by the mobile device.

### Claim 18 (depends on 17)

The mount of claim 17, further comprising at least one projector for projecting electromagnetic radiation into the interaction space.

### Claim 19 (depends on 17)

The mount of claim 17, further comprising image processing functionality for processing the image information.

### Claim 20 (depends on 19)

The mount of claim 19, wherein the image processing functionality is configured to generate depth information based on the image information using a depth reconstruction technique.
