# US 12,118,136 B2 · Air floating video display apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,136 B2
- Document ID: US12118136B2
- Status: Granted
- Kind: B2
- Application number: 18/265,266
- Filed: 2021-12-20
- Published: 2024-02-01
- Granted: 2024-10-15
- Earliest claimed: 2021-01-25
- Assignees: Maxell, Ltd.
- Applicants: Maxell, Ltd.
- Inventors: Hiroaki Takahashi; Koji Hirata; Koji Fujita; Toshinori Sugiyama
- Web page: https://patentgrep.com/patent/US12118136B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G02F 1/133603: {with LEDs}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/0421: {by interrupting or reflecting a light beam, e.g. optical touch-screen}
- G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G09F 9/00: Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements (in which the variable information is permanently attached to a movable support G09F11/00; abacus G06C1/00; slide-rules G06G1/00)
- H04N 13/346: using prisms or semi-transparent mirrors
- H04R 1/403: {loud-speakers}
- G02B 2027/0145: {creating an intermediate image}
- G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
- H04R 2499/15: Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02F: OPTICAL DEVICES OR ARRANGEMENTS FOR THE CONTROL OF LIGHT BY MODIFICATION OF THE OPTICAL PROPERTIES OF THE MEDIA OF THE ELEMENTS INVOLVED THEREIN; NON-LINEAR OPTICS; FREQUENCY-CHANGING OF LIGHT; OPTICAL LOGIC ELEMENTS; OPTICAL ANALOGUE/DIGITAL CONVERTERS
- G09F: DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A technique suitable when an air floating video is used as a non-contact user interface is provided. An air floating video display apparatus includes: a sensor for detecting a state of operation including a position of a hand finger of a user with respect to an air region including a plane of the air floating video or an object displayed on the plane; and a fingertip tactile sense generator which generates texture of a sense of touch on the hand finger by forming, based on information detected by the sensor, sound pressure based on an ultrasonic wave in vicinity of a position of the hand finger.

## Claims

### Claim 1

An air floating video display apparatus configured to form an air floating video, comprising:
a display apparatus configured to display video;
a retroreflector configured to retroreflect video light emitted from the display apparatus;
the air floating video being formed based on reflection light reflected from the retroreflector,
a sensor configured to detect a state of operation including a position of a hand finger of a user or a substance held by the user with respect to a space region including a plane of the air floating video or an object displayed on the plane;
a tactile sense generator configured to generate texture of a sense of touch on the hand finger or the substance held by the user by forming, based on information detected by the sensor, sound pressure based on an ultrasonic wave in vicinity of the position of the hand finger or the substance held by the user; and
a super-directive loudspeaker configured to output super-directive audio toward vicinity of a face of the user, wherein
when audio of a type requiring high confidentiality is output, a position of the face of the user is detected based on an image of the user captured by an image-capturing apparatus, and super-directive audio is output toward the vicinity of the face of the user by the super-directive loudspeaker, and
when audio of a type not requiring the high confidentiality is output, the audio is output by the tactile sense generator from the vicinity of the position of the hand finger of the user or the substance held by the user.

### Claim 2 (depends on 1)

The air floating video display apparatus according to claim 1,
wherein the tactile sense generator modulates a signal of the ultrasonic wave with an audio signal to produce audio from the vicinity of the position of the hand finger or the substance held by the user.

### Claim 3 (depends on 1)

The air floating video display apparatus according to claim 1,
wherein the tactile sense generator has
an ultrasonic element array with a plurality of ultrasonic elements arranged thereon, and
an ultrasonic signal generation circuit configured to generate an ultrasonic driving signal to be input to each of the ultrasonic elements of the ultrasonic element array based on the information detected by the sensor, and
the ultrasonic signal generation circuit generates, for each of the ultrasonic elements, the ultrasonic driving signal so as to provide one ultrasonic signal selected from ultrasonic signals having phases of a plurality of different types.

### Claim 4 (depends on 1)

The air floating video display apparatus according to claim 1,
wherein, as the object, an object which accepts touch operation with the hand finger or the substance held by the user is provided, and
sound pressure based on the ultrasonic wave is formed on the plane of the air floating video in vicinity of the hand finger or the touch position of the substance held by the user.

### Claim 5 (depends on 3)

The air floating video display apparatus according to claim 3 comprising:
a housing configured to house the display apparatus and the retroreflector; and
a transparent member provided to part of the housing and configured to transmit the reflection light reflected from the retroreflector,
wherein the ultrasonic element array is arranged at a position outside the transparent member so as to form a path in which the ultrasonic wave emitted from the ultrasonic element array is reflected by the transparent member and is then applied from a back side of the plane of the air floating video.

### Claim 6 (depends on 3)

The air floating video display apparatus according to claim 3 comprising:
a housing configured to house the display apparatus and the retroreflector; and
a transparent member provided to part of the housing and configured to transmit the reflection light reflected from the retroreflector,
wherein the ultrasonic element array is arranged at a position outside the transparent member so as to form a path in which the ultrasonic wave emitted from the ultrasonic element array is not reflected by the transparent member but is applied from a back side of the plane of the air floating video.

### Claim 7 (depends on 2)

The air floating video display apparatus according to claim 2,
wherein the modulation is amplification modulation.

### Claim 8 (depends on 2)

The air floating video display apparatus according to claim 2,
wherein the audio signal is an audio signal of predetermined audio associated with the plane of the air floating video or the object, or operation on the plane or the object.

### Claim 9 (depends on 1)

The air floating video display apparatus according to claim 1,
wherein the texture of the sense of touch is generated on the hand finger of the user or the substance held by the user by the tactile sense generator.

### Claim 10 (depends on 1)

The air floating video display apparatus according to claim 1,
wherein the display apparatus includes:
a liquid-crystal display panel configured to display the video; and
a light source configured to supply light in a specific polarizing direction to the liquid-crystal display panel,
the retroreflector retroreflects a video luminous flux having a narrow diffuse angle as video light emitted from the liquid-crystal display panel, and
the apparatus includes a light-blocking member arranged in space on an optical path connecting the liquid-crystal display panel and the retroreflector to block video light having a diffuse angle exceeding a specific angle from the liquid-crystal display panel from entering the retroreflector.

### Claim 11

A method of forming an air floating video, the method comprising:
detecting a state of operation including a position of a hand finger of a user or a substance held by the user with respect to a space region including a plane of an air floating video or an object displayed on the plane;
detecting an ultrasonic wave in vicinity of the position of the hand finger or the substance held by the user;
generating texture of a sense of touch on the hand finger or the substance held by the user by forming sound pressure based on the ultrasonic wave in vicinity of the position of the hand finger or the substance held by the user; and
controlling output of audio including determining whether audio to be output requires high confidentiality, wherein
in response to the audio requiring high confidentiality, detecting a position of a face of the user based on an image of the user captured by an image-capturing apparatus, and outputting audio toward the vicinity of the face of the user, and
in response to the audio not requiring high confidentiality, outputting the audio to the vicinity of the position of the hand finger of the user or the substance held by the user.

### Claim 12 (depends on 11)

The method according to claim 11, wherein generating texture includes modulating a signal of the ultrasonic wave with an audio signal to produce audio from the vicinity of the position of the hand finger or the substance held by the user.

### Claim 13 (depends on 11)

The method according to claim 11, wherein outputting audio toward the vicinity of the face of the user includes using a super-directive loudspeaker.

### Claim 14 (depends on 13)

The method according to claim 13, wherein outputting audio toward the vicinity of the finger of the user includes using a different loudspeaker than the super-directive loudspeaker.

### Claim 15

A non-transitory computer readable storage device having computer readable instructions that when executed by circuitry cause the circuitry to:
detect a state of operation including a position of a hand finger of a user or a substance held by the user with respect to a space region including a plane of an air floating video or an object displayed on the plane;
detect an ultrasonic wave in vicinity of the position of the hand finger or the substance held by the user;
generate texture of a sense of touch on the hand finger or the substance held by the user by forming sound pressure based on the ultrasonic wave in vicinity of the position of the hand finger or the substance held by the user; and
control output of audio including determining whether audio to be output requires high confidentiality, wherein
in response to the audio requiring high confidentiality, detect a position of a face of the user based on an image of the user captured by an image-capturing apparatus, and output audio toward the vicinity of the face of the user using a super-directive loudspeaker, and
in response to the audio not requiring high confidentiality, output the audio from the vicinity of the position of the hand finger of the user or the substance held by the user.

### Claim 16 (depends on 15)

The non-transitory computer readable storage device according to claim 15, wherein the circuitry is caused to generate texture by modulating a signal of the ultrasonic wave with an audio signal to produce audio from the vicinity of the position of the hand finger or the substance held by the user.

### Claim 17 (depends on 15)

The non-transitory computer readable storage device according to claim 15, wherein the circuitry is caused to output audio toward the vicinity of the face of the user using a super-directive loudspeaker.

### Claim 18 (depends on 17)

The non-transitory computer readable storage device according to claim 17, wherein the circuitry is caused to output audio toward the vicinity of the finger of the user includes using a different loudspeaker than the super-directive loudspeaker.