# US 2017/0285931 A1 · Operating visual user interface controls with ink commands

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0285931 A1
- Document ID: US20170285931A1
- Status: Published application
- Kind: A1
- Application number: 15/084,272
- Filed: 2016-03-29
- Published: 2017-10-05
- Earliest claimed: 2016-03-29
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: David Walker Duhon; Yibo Sun; Xiao Tu; Francis Zhou
- Web page: https://patentgrep.com/patent/US20170285931A1

## CPC

- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/03543: {Mice or pucks (G06F3/03541 takes precedence)}
- G06F 3/03545: {Pens or stylus}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0486: Drag-and-drop
- G06F 40/171: by use of digital ink
- G06V 10/235: {based on user input or interaction}
- G06V 30/32: Digital ink
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A facility enabling a user to operate visual user interface controls with ink commands is described. The facility causes to be displayed a control operable with mouse input, and receives an ink command directed to the displayed control. In response, the facility operates the displayed control in accordance with the received ink command.

## Claims

### Claim 1

A processor-based device, comprising:
a processor; and
a memory having contents that cause the processor to, in response to execution of an operating system by the processor:
cause to be displayed a control operable with mouse and/or keyboard input;
receive ink input proximate to the displayed control;
interpret the received ink input as an ink command; and
operate the displayed control in accordance with the interpreted ink command.

### Claim 2 (depends on 1)

The device of claim 1 wherein the displayed control is a one-dimensional value-setting control for setting an index value, and wherein the received ink input comprises a handwritten number, and wherein the interpreted ink command specifies an index value that is based on the handwritten number.

### Claim 3 (depends on 2)

The device of claim 2 wherein the displayed control is a one-dimensional value-setting control for setting a media playback index time, a page number, or an audio volume level.

### Claim 4 (depends on 1)

The device of claim 1 wherein the displayed control is for setting a quantity, and has
a handle and a track, the track having a first end and a second end, the handle being repositionable to different points on the track corresponding to different values of the quantity,
wherein the received ink input comprises ink that crosses the control's track a particular percentage of the distance from the first end of the track to the second end of the track,
and wherein the interpreted ink command specifies a value for the quantity that is based upon the percentage.

### Claim 5 (depends on 4)

The device of claim 4 wherein the displayed control is a slider, scrollbar, or ruler.

### Claim 6 (depends on 1)

The device of claim 1 wherein the displayed control is a button bearing a visual symbol,
and wherein the received ink input comprises an approximation of the visual symbol borne by the button,
and wherein operating the displayed control in accordance with the interpreted ink command comprises activating the button.

### Claim 7 (depends on 1)

The device of claim 1 one wherein the displayed control is one or more electronic document reader pages,
and wherein the received ink input comprises a selected number of hand-drawn carets pointing in a selected direction,
and wherein operating the displayed control in accordance with the interpreted ink command comprises performing the selected number of page flips in the selected direction.

### Claim 8 (depends on 1)

The device of claim 1 wherein the displayed control is a date picker control for setting a date, the control presenting a plurality of dates arranged in rows each corresponding to a week,
and wherein the received ink input is a containing shape that contains a selected date presented by the control,
and wherein operating the displayed control in accordance with the interpreted ink command comprises setting the selected date.

### Claim 9 (depends on 1)

The device of claim 1 wherein the displayed control is a date picker control for setting a date, the control that present either (1) a plurality of dates arranged in rows each corresponding to a week, and/or (2) two or more controls for independently setting different components of a date,
and wherein the received ink input is text indicating a date,
and wherein operating the displayed control in accordance with the interpreted ink command comprises setting the indicated date.

### Claim 10 (depends on 1)

The device of claim 1 wherein the displayed control is an enumerated textual string picker control for setting one of a plurality of the enumerated textual strings each comprised of a sequence of characters,
and wherein the received ink input is a handwritten sequence of characters matching the beginning of the character sequence of exactly one of the enumerated textual strings,
and wherein operating the displayed control in accordance with the interpreted ink input comprises setting the enumerated textual string the beginning of whose sequence matches the handwritten sequence of characters.

### Claim 11 (depends on 1)

The device of claim 1 wherein the displayed control is a container control for selecting items in a container, the container control containing a visual representation of each of at least a portion of the items in the container,
and wherein the received ink input is an enclosing shape surrounding one or more of the item visual representations,
and wherein operating the displayed control in accordance with the interpreted ink input comprises selecting the items whose visual representations are surrounded by the enclosing shape.

### Claim 12 (depends on 11)

The device of claim 11 wherein the displayed container control shows items in the container in a grid, list, or tree.

### Claim 13 (depends on 1)

The device of claim 1 wherein the displayed control is a container control for selecting items in a container, the container control containing a visual representation of each of at least a portion of the items in the container, each of the items in the container having a textual string associated with it,
and wherein the received ink input is a handwritten textual string comprising a sequence of characters,
and wherein operating the displayed control in accordance with the interpreted ink input comprises subsetting the items for which the container control contains visual representations to only those items whose associated textual strings match the handwritten textual string.

### Claim 14 (depends on 1)

The device of claim 1 wherein the displayed control is a container control containing a visual representation of each of at least a portion of the items in a source container, at least one of the item visual representations indicating that the represented item is selected,
and wherein the received ink input is an arrow beginning near one of the item visual representations that indicates that the represented item is selected, and ending near a visual indication of a target container,
and wherein operating the displayed control in accordance with the interpreted ink input comprises transferring the selected items from the source container to the target container.

### Claim 15 (depends on 1)

The device of claim 1 wherein the displayed control is a container control containing a visual representation of each of at least a portion of the items in a source container, at least one of the item visual representations indicating that the represented item is selected,
and wherein the received ink input is an arrow beginning near one of the item visual representations that indicates that the represented item is selected, and ending near a visual indication of an operation that can be performed against items,
and wherein operating the displayed control in accordance with the interpreted ink input comprises subjecting the selected items to the indicated operation.

### Claim 16 (depends on 1)

The device of claim 1 wherein the displayed control is a container control containing a visual representation of each of at least a portion of the items in the container,
and wherein the received ink input comprises a shape drawn over at least a portion of the representation of the selected item,
and wherein operating the displayed control in accordance with the interpreted ink input comprises deleting the selected item from the container.

### Claim 17 (depends on 16)

The device of claim 16 wherein the shape comprised by the received ink input is a horizontal segment, a horizontal segment with a loop at one end, or a scribble.

### Claim 18 (depends on 1)

The device of claim 1 wherein the displayed control is a container control for selecting items in a container, the container control containing a visual representation of each of at least a portion of the items in the container,
and wherein the received ink input comprises a check mark, a star shape, or an asterisk drawn near the representation of each of one or more distinguished items,
and wherein operating the displayed control in accordance with the interpreted ink input comprises selecting the distinguished items.

### Claim 19

A computer-readable medium having contents configured to cause a computing system to:
cause to be displayed a control operable with mouse input;
receive an ink command directed to the displayed control; and
operate the displayed control in accordance with the received ink command.

### Claim 20

A method in a computing system, the method comprising:
receiving ink input at a particular location relative to a display region;
based on the received ink input's shape and location, interpreting the received ink input as an ink command; and
performing the interpreted ink command.
