# US 2023/0273674 A1 · Displaying physical input devices as virtual objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0273674 A1
- Document ID: US20230273674A1
- Status: Published application
- Kind: A1
- Application number: 17/985,040
- Filed: 2022-11-10
- Published: 2023-08-31
- Earliest claimed: 2018-06-05
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Pol PLA I. CONESA; Earl M. OLSON; Aaron P. THOMPSON
- Web page: https://patentgrep.com/patent/US20230273674A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/023: Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/03547: {Touch pads, in which fingers can move on a surface}
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure relates to techniques for displaying representations of physical input devices and overlaying visual features on the representations of physical input devices in a computer-generated reality (CGR) environment. The techniques include displaying a virtual application in a CGR environment and, in response to detecting an input field in the displayed virtual application, displaying at least a portion of the displayed application on a representation of a physical input device. The at least a portion of the displayed application includes the detected input field. In response to detecting an input received at the physical input device, the input field is updated with the input, and the updated input field is displayed.

## Claims

### Claim 1

(canceled)

### Claim 2

An electronic device, comprising:
a display;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
presenting, via the display, a user interface for a virtual application;
presenting, via the display, a representation of a physical input device;
detecting an application state corresponding to the user interface for the virtual application; and
in response to detecting the application state:
in accordance with a determination that the application state corresponds to a first application state, presenting a first set of input controls overlaid on the representation of the physical input device; and
in accordance with a determination that the user interface for the virtual application is in a second application state, different from the first application state, presenting a second set of input controls overlaid on the representation of the physical input device, wherein the second set of input controls is different from the first set of input controls.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the user interface for a virtual application is displayed concurrently with the representation of a physical input device.

### Claim 4 (depends on 2)

The electronic device of claim 2, wherein the physical input device does not include a display component.

### Claim 5 (depends on 2)

The electronic device of claim 2, wherein the presented respective set of input controls are based on a context of the virtual application.

### Claim 6 (depends on 2)

The electronic device of claim 2, wherein the physical input device includes a third set of input controls that is different from the presented respective set of input controls.

### Claim 7 (depends on 2)

The electronic device of claim 2, the one or more programs further including instructions for:
while presenting the respective set of input controls, detecting an input directed to a key on the physical input device;
in response to detecting the input directed to the key on the physical input device:
in accordance with a determination that a first function of the key on the physical input device corresponds to the first function of a key at a corresponding location on the representation of the physical input device, performing the first function; and
in accordance with a determination that the first function of the key on the physical input device does not correspond to the first function of a key at the corresponding location on the representation of the physical input device, forgoing performing the first function.

### Claim 8 (depends on 2)

The electronic device of claim 2, the one or more programs further including instructions for:
while presenting the respective set of input controls, detecting an input directed to a key on the physical input device, wherein the key corresponds to a second function; and
in response to detecting the input directed to the key on the physical input device, performing a third function, different from the second function.

### Claim 9

A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device with a display, the one or more programs including instructions for:
presenting, via the display, a user interface for a virtual application;
presenting, via the display, a representation of a physical input device;
detecting an application state corresponding to the user interface for the virtual application; and
in response to detecting the application state:
in accordance with a determination that the application state corresponds to a first application state, presenting a first set of input controls overlaid on the representation of the physical input device; and
in accordance with a determination that the user interface for the virtual application is in a second application state, different from the first application state, presenting a second set of input controls overlaid on the representation of the physical input device, wherein the second set of input controls is different from the first set of input controls.

### Claim 10

A method, comprising:
at an electronic device having a display:
presenting, via the display, a user interface for a virtual application;
presenting, via the display, a representation of a physical input device;
detecting an application state corresponding to the user interface for the virtual application; and
in response to detecting the application state:
in accordance with a determination that the application state corresponds to a first application state, presenting a first set of input controls overlaid on the representation of the physical input device; and
in accordance with a determination that the user interface for the virtual application is in a second application state, different from the first application state, presenting a second set of input controls overlaid on the representation of the physical input device, wherein the second set of input controls is different from the first set of input controls.
