# US 8,784,203 B2 · Determination of controller three-dimensional location using image analysis and ultrasonic communication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,784,203 B2
- Document ID: US8784203B2
- Status: Granted
- Kind: B2
- Application number: 13/615,355
- Filed: 2012-09-13
- Published: 2013-01-03
- Granted: 2014-07-22
- Earliest claimed: 2008-05-30
- Assignees: Sony Computer Entertainment America LLC
- Applicants: Gary M. Zalewski
- Inventors: Gary M. Zalewski
- Web page: https://patentgrep.com/patent/US8784203B2

## CPC

- A63F 13/424: involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone
- A63F 13/28: responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light
- A63F 13/833: Hand-to-hand fighting, e.g. martial arts competition
- A63F 13/843: involving concurrently two or more players on the same game device, e.g. requiring the use of a plurality of controllers or of a specific view of game data for each player
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/033: Pointing devices displaced or positioned by the user {, e.g. mice, trackballs, pens or joysticks}; Accessories therefor (digitisers characterised by the transducing means G06F3/041)
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- A63F 2300/1081: Input via voice recognition
- A63F 2300/1087: comprising photodetecting means, e.g. a camera
- A63F 2300/8029: Fighting without shooting
- A63F 2300/8088: involving concurrently several players in a non-networked game, e.g. on the same game console
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods, systems, and computer programs are presented for determining the location of a controller. One method includes an operation for capturing image data of a capture area in front of a display. Additionally, the method includes another operation for capturing sound data emitted by the controller in the capture area in front of the display. The two-dimensional location of the controller is calculated based on the captured image data, and the third dimensional location of the controller is calculated based on the captured sound data.

## Claims

### Claim 1

A method for determining location of a controller, the method comprising:
capturing image data of a capture area in front of a display;
capturing sound data emitted by the controller in the capture area in front of the display;
calculating a two-dimensional location of the controller based on the captured image data; and
calculating a third dimensional location of the controller based on the captured sound data, wherein calculating the third dimensional location further includes:
measuring a time of flight of the captured sound data; and
calculating a depth within the capture area based on the time of flight of the captured sound data, wherein the depth is an orthogonal measure to both dimensions of the two-dimensional location, wherein operations of the method are executed by a processor.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein calculating the two-dimensional location of the controller further includes:
detecting a spherical shape within the image data, the spherical shape being associated with a spherical-type section in the controller.

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein the spherical-type section is operable to emit one of a plurality of light colors to assist in identifying the spherical shape in the captured image data.

### Claim 4 (depends on 3)

The method as recited in claim 3, further including:
sending from a processor to the controller a choice of light color from the plurality of light colors, wherein the processor calculates the two-dimensional location, wherein the choice of light color is utilized to identify the spherical shape in the captured image data.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein calculating the third dimensional location further includes:
measuring and a phase coherence of the captured sound data; and
calculating the depth within the capture area based on the time of flight and the phase coherence of the captured sound data.

### Claim 6 (depends on 1)

The method as recited in claim 1, further including,
presenting in a graphical user interface in the display an object based on the two-dimensional location and the third dimensional location.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein the sound data is captured with a sound capture device, the method further including:
establishing ultrasound communication between the sound capture device and the controller.

### Claim 8 (depends on 1)

The method as recited in claim 1, further including:
detecting a position of a user head within the captured image data or a position of a user torso within the captured image data.

### Claim 9 (depends on 1)

The method as recited in claim 1, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

### Claim 10

A system for determining location of a controller, comprising:
an image capture device operable to capture image data of a capture area in front of a display;
a sound capture device operable to capture sound data emitted by the controller in the capture area in front of the display; and
a processor in communication with the image capture device and the sound capture device, wherein the processor is operable to measure a time of flight of the captured sound data, wherein the processor is operable to calculate a two-dimensional location of the controller based on the captured image data, and the processor is operable to calculate a third dimensional location of the controller based on the captured sound data and the time of flight, wherein the third dimensional location is orthogonal to both dimensions of the two-dimensional location.

### Claim 11 (depends on 10)

The system as recited in claim 10, wherein the controller includes:
an elongated body with a first end and a second end;
a grip area at about the first end;
an input pad defined between the first end and the second end;
a spherical-type section defined at the second end; and
a sound emitting device.

### Claim 12 (depends on 11)

The system as recited in claim 11, wherein the controller further includes an acoustical chamber, the acoustical chamber defined to direct and receive sound in one or more directions.

### Claim 13 (depends on 12)

The system as recited in claim 12, wherein the controller further includes an inertial analyzer for tracking inertial activity of the controller, wherein the controller is operable to transmit the inertial activity to the system, wherein the processor is further operable to include the inertial activity when calculating the two-dimensional location and the third dimensional location of the controller.

### Claim 14 (depends on 10)

The system as recited in claim 10, wherein the image capture device includes a visible-spectrum camera, wherein the image capture device further includes an infrared camera, wherein the image data from the infrared camera and the visible-spectrum camera are combined in the calculation of the two-dimensional location.

### Claim 15 (depends on 10)

The system as recited in claim 10 further including: a speaker connected to the processor, wherein the controller further includes a controller sound capturing device, wherein the controller and the system are operable to establish sound communication via the speaker, the sound capture device, and the controller sound capturing device.

### Claim 16 (depends on 10)

The system as recited in claim 10, wherein the image data corresponds to one of either an red-green-blue (RGB) pixel image data, a black and white pixel image data, or infrared data.

### Claim 17 (depends on 10)

The system as recited in claim 10, wherein the processor is operable to measure a phase coherence of the captured sound data, and the processor is operable to calculate the third dimensional location of the controller within the capture area based on the time of flight and the phase coherence of the captured sound data.

### Claim 18

A system comprising:
a controller including an image capture device;
one or more light emitters facing a capture area in front of a display, the light emitters located near the display;
a sound capture device operable to capture sound data emitted by the controller when the controller is in the capture area in front of the display, and
a processor in communication with the controller, the one or more light emitters and the sound capture device, wherein the processor is operable to receive from the controller image data captured with the image capture device, wherein the processor determines a position of the light emitters within the image captured data and calculates calculate a two-dimensional location of the controller based on the determined position, wherein the processor calculates a third dimensional location of the controller based on the captured sound data, wherein the processor is operable to measure a time of flight of the captured sound data, and the processor is operable to calculate a depth within the capture area based on the time of flight of the captured sound data.

### Claim 19 (depends on 18)

The system as recited in claim 18, wherein the processor is operable to measure a phase coherence of the captured sound data, and the processor is operable to calculate the depth within the capture area based on the time of flight and the phase coherence of the captured sound data.