# US 12,008,168 B2 · Method for operating a mobile terminal using a gesture recognition and control device, gesture recognition and control device, motor vehicle, and an output apparatus that can be worn on the head

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,168 B2
- Document ID: US12008168B2
- Status: Granted
- Kind: B2
- Application number: 17/627,304
- Filed: 2020-07-01
- Published: 2022-08-04
- Granted: 2024-06-11
- Earliest claimed: 2019-07-15
- Assignees: Audi AG
- Applicants: Audi AG
- Inventors: Norbert Kulbat
- Web page: https://patentgrep.com/patent/US12008168B2

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/80: Arrangements for controlling instruments
- B60W 50/10: Interpretation of driver requests or demands
- B60W 60/001: {Planning or execution of driving tasks}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04817: using icons (graphical or visual programming using iconic symbols G06F8/34)
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 2360/1464: 3D-gesture
- B60K 2360/1468: Touch gesture
- B60K 2360/175: Autonomous driving
- B60K 2360/177: Augmented reality
- B60K 2360/563: Vehicle displaying mobile device information
- B60K 2360/569: Vehicle controlling mobile device functions
- B60K 2360/577: Mirror link with mobile devices
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2540/00: Input parameters relating to occupants
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A gesture recognition and control device recognizes a mobile terminal and ascertains a current graphical user interface generated by a display device of the mobile terminal. The gesture recognition and control device provides an output signal describing, as display content, the graphical user interface generated by the display device of the mobile terminal, and transmits the output signal to an output apparatus that can be worn on the head for outputting the display content in a predefined output region in the interior of the motor vehicle as part of augmented reality or virtual reality output by the output apparatus. During the process of outputting the display content, the gesture recognition and control device recognizes a spatial gesture of the user, generates a remote control signal for triggering an operating function of the mobile terminal, and transmits the remote control signal to a control device of the recognized mobile terminal.

## Claims

### Claim 1

A method for operating a mobile terminal using a gesture recognition and control device, the method comprising:
recognizing the mobile terminal;
ascertaining a graphical user interface as a GUI of the mobile terminal generated to be displayed by a display device of the mobile terminal, the GUI providing an operating option that involves an operating function of the mobile terminal being triggerable;
providing an output signal describing the GUI of the mobile terminal as a display content of the mobile terminal;
mirroring the display content of the mobile terminal in form of a virtual display content of the mobile terminal in an output region in an interior of a motor vehicle as part of an augmented reality or a virtual reality, output by a wearable output apparatus, based on the output signal describing the GUI of the mobile terminal as the display content of the mobile terminal;
while the virtual display content of the mobile terminal is mirrored in the output region, recognizing a contactless operating gesture of a user based on the operating option of the GUI of the mobile terminal;
generating a remote control signal based on the contactless operating gesture, the remote control signal describing triggering of the operating function of the mobile terminal assigned to the contactless operating gesture; and
transmitting the remote control signal to a control device of the mobile terminal for triggering the operating function of the mobile terminal.

### Claim 2 (depends on 1)

The method as claimed in claim 1, further comprising:
recognizing a further contactless operating gesture to predefine a positioning of the virtual display content in the interior; and
predefining the output region in the interior based on the further contactless operating gesture.

### Claim 3 (depends on 2)

The method as claimed in claim 2, further comprising:
providing a further output signal describing a further graphical user interface as a further GUI providing an operating menu for predefining the positioning of the virtual display content.

### Claim 4 (depends on 1)

The method as claimed in claim 1, further comprising:
recognizing a further contactless operating gesture describing a scaling of the virtual display content; and
scaling an image showing the virtual display content based on the further contactless operating gesture.

### Claim 5 (depends on 4)

The method as claimed in claim 4, further comprising:
providing a further output signal describing a further graphical user interface as a further GUI providing an operating menu for scaling the virtual display content.

### Claim 6 (depends on 1)

The method as claimed in claim 1, wherein the method is performed by the gesture recognition and control device based on at least one of an activation of a fully autonomous driving mode and a start signal of the motor vehicle.

### Claim 7

A gesture recognition and control device, comprising:
a memory configured to store instructions; and
a processor configured to execute the instructions stored in the memory to:
recognize a mobile terminal,
ascertain a graphical user interface as a GUI of the mobile terminal generated to be displayed by a display device of the mobile terminal, the GUI providing an operating option that involves an operating function of the mobile terminal being triggerable,
provide an output signal describing the GUI of the mobile terminal as a display content of the mobile terminal,
mirroring the display content of the mobile terminal in form of a virtual display content of the mobile terminal in an output region in an interior of a motor vehicle as part of an augmented reality or a virtual reality, output by a wearable output apparatus, based on the output signal describing the GUI of the mobile terminal as the display content of the mobile terminal,
while the virtual display content of the mobile terminal is mirrored in the output region, recognize a contactless operating gesture of a user based on the operating option of the GUI of the mobile terminal,
generate a remote control signal based on the contactless operating gesture, the remote control signal describing triggering of the operating function of the mobile terminal assigned to the contactless operating gesture, and
transmit the remote control signal to a control device of the mobile terminal to trigger the operating function of the mobile terminal.

### Claim 8 (depends on 7)

The gesture recognition and control device as claimed in claim 7, wherein the processor is configured to execute the instructions stored in the memory to:
recognize a further contactless operating gesture to predefine a positioning of the virtual display content in the interior,
predefine the output region in the interior based on the further contactless operating gesture.

### Claim 9 (depends on 8)

The gesture recognition and control device as claimed in claim 8, wherein the processor is configured to execute the instructions stored in the memory to:
provide a further output signal describing a further graphical user interface as a further GUI providing an operating menu for predefining the positioning of the virtual display content.

### Claim 10 (depends on 7)

The gesture recognition and control device as claimed in claim 7, wherein the processor is configured to execute the instructions stored in the memory to:
recognize a further contactless operating gesture describing a scaling of the virtual display content, and
scale an image showing the virtual display content based on the further contactless operating gesture.

### Claim 11 (depends on 10)

The gesture recognition and control device as claimed in claim 10, wherein the processor is configured to execute the instructions stored in the memory to:
provide a further output signal describing a further graphical user interface as a further GUI providing an operating menu for scaling the virtual display content.

### Claim 12 (depends on 7)

The gesture recognition and control device as claimed in claim 7, wherein the processor is configured to execute the instructions stored in the memory to recognize the mobile terminal based on at least one of an activation of a fully autonomous driving mode and a start signal of the motor vehicle.

### Claim 13 (depends on 7)

A motor vehicle comprising the gesture recognition and control device claimed in claim 7.

### Claim 14

An output apparatus wearable on a head of a user for outputting augmented reality and/or virtual reality, the output apparatus comprising:
a display; and
a gesture recognition and control device, including:
a memory configured to store instructions, and
a processor configured to execute the instructions stored in the memory to:
recognize a mobile terminal,
ascertain a graphical user interface as a GUI of the mobile terminal generated to be displayed by a display device of the mobile terminal, the GUI providing an operating option that involves an operating function of the mobile terminal being triggerable,
generate an output signal describing the GUI of mobile terminal as a display content of the mobile terminal,
mirror the display content of the mobile terminal in form of a virtual display content of the mobile terminal in an output region in an interior of a motor vehicle as part of an augmented reality or a virtual reality, output by the output apparatus, based on the output signal describing the GUI of the mobile terminal as the display content of the mobile terminal,
while the virtual display content of the mobile terminal is mirrored in the output region, recognize a contactless operating gesture of user based on the operating option of the GUI of the mobile terminal,
generate a remote control signal based on the contactless operating gesture, the remote control signal describing triggering of the operating function of the mobile terminal assigned to the contactless operating gesture, and
transmit the remote control signal to a control device of the mobile terminal to trigger the operating function of the mobile terminal.

### Claim 15 (depends on 14)

The output apparatus as claimed in claim 14, further comprising a camera to detect the contactless operating gesture of the user.