# US 2017/0286385 A1 · Ink in an Electronic Document

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286385 A1
- Document ID: US20170286385A1
- Status: Published application
- Kind: A1
- Application number: 15/199,539
- Filed: 2016-06-30
- Published: 2017-10-05
- Earliest claimed: 2016-03-29
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Ryan Lucas Hastings; Daniel McCulloch; Michael John Patten
- Web page: https://patentgrep.com/patent/US20170286385A1

## CPC

- G06F 40/14: Tree-structured documents (parsing G06F40/205; validation G06F40/226)
- G06F 3/03545: {Pens or stylus}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 40/117: Tagging; Marking up (details of markup languages G06F40/143); Designating a block; Setting of attributes (style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154)
- G06F 40/151: Transformation
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/171: by use of digital ink
- G06F 40/197: Version control (for software G06F8/71)
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 2203/04807: Pen manipulated menu
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques for ink in an electronic document are described. According to various implementations, techniques described herein provide a rich set of tools which allow a user to markup an electronic document such as a web page, not only in static 2D where the user writes on top of a document, but in dynamic 3D. In addition, when adding 3D elements to an electronic document, the 3D elements are added based on awareness of the content of the electronic document and can adapt its content in relationship to the document.

## Claims

### Claim 1

A system comprising:
a display;
one or more processors; and
one or more computer-readable storage media storing computer-executable instructions that, responsive to execution by the one or more processors, cause the system to perform operations including:
transforming an electronic document into a 3-dimensional (3D) canvas that represents a 2-dimensional (2D) representation of a 3D version of the electronic document;
enabling a graphical element of the electronic document to be manipulated in 3D relative to the 3D canvas via ink input;
receiving ink input manipulating a characteristic of the graphical element; and
modifying a stored version of the graphical element based on the ink input.

### Claim 2 (depends on 1)

The system as described in claim 1, wherein the electronic document comprises a web page, and the graphical element comprises a graphical element of the web page.

### Claim 3 (depends on 1)

The system as described in claim 1, wherein the electronic document comprises a web page, and said transforming is performed responsive to detecting that a smart ink mode is invoked for the web page.

### Claim 4 (depends on 1)

The system as described in claim 1, wherein said enabling is performed responsive to a selection of the graphical element via a pen.

### Claim 5 (depends on 1)

The system as described in claim 1, wherein the operations further comprise:
receiving ink input to the 3D canvas; and
adding the ink input to the 3D canvas in a context of a structure of the electronic document.

### Claim 6 (depends on 1)

The system as described in claim 1, wherein the electronic document includes a further graphical element, and wherein the operations further comprise:
receiving ink input adding an ink object to the 3D canvas; and
applying a visual effect to the further graphical element based on the ink object.

### Claim 7 (depends on 1)

The system as described in claim 1, wherein the ink input comprises an annotation of the graphical element, and wherein said modifying comprises adding the annotation to the stored version of the graphical element.

### Claim 8 (depends on 1)

The system as described in claim 1, wherein the electronic document comprises a web page, and wherein the operations further comprise:
updating the web page to replace the graphical element with the modified stored version of the graphical element, said updating causing a published version of the web page to be updated.

### Claim 9 (depends on 1)

The system as described in claim 1, wherein the operations further comprise causing the 3D canvas to be shared to a different system as an editable version of the 3D canvas.

### Claim 10

A method comprising:
detecting that a smart ink mode is invoked for an electronic document;
transforming, responsive to said detecting, the electronic document into a 3-dimensional (3D) canvas that represents a 2-dimensional (2D) representation of a 3D version of the electronic document; and
enabling an element of the electronic document to be manipulated in 3D relative to the 3D canvas via ink input.

### Claim 11 (depends on 10)

The method as described in claim 10, wherein the electronic document comprises a web page, and the element comprises a graphical element of the web page.

### Claim 12 (depends on 10)

The method as described in claim 10, wherein the electronic document comprises a web page, the graphical element comprises a graphical element of the web page, and wherein said enabling comprises enabling the graphical element to be rotated within the 3D canvas relative to multiple axes of the graphical element to simulate a 3D rotation of the graphical element.

### Claim 13 (depends on 10)

The method as described in claim 10, further comprising:
receiving a selection of the element from the 3D canvas; and
providing, responsive to said receiving, a visual cue that the element is able to be manipulated and edited to affect an appearance of the element.

### Claim 14 (depends on 10)

The method as described in claim 10, further comprising:
receiving an edit to the element via the 3D canvas;
causing the element to be modified based on the edit; and
causing a published version of the electronic document to be updated with the modified element.

### Claim 15 (depends on 10)

The method as described in claim 10, wherein the ink input comprises an annotation of the element, and wherein said modifying comprises adding the annotation to the stored version of the electronic document.

### Claim 16 (depends on 10)

The method as described in claim 10, further comprising:
receiving ink input to the 3D canvas;
adding the ink input as an ink object to the 3D canvas; and
applying a visual effect to a further graphical element of the electronic document based on the ink object.

### Claim 17 (depends on 10)

The method as described in claim 10, further comprising:
receiving, via a first device, an edit to the element via the 3D canvas;
causing the element to be modified based on the edit; and
causing the 3D canvas with the modified element to be shared to a second device as an editable version of the 3D canvas.

### Claim 18

A method comprising:
transforming an electronic document into a 3-dimensional (3D) canvas that represents a 2-dimensional (2D) representation of a 3D version of the electronic document;
enabling a graphical element of the electronic document to be manipulated in 3D relative to the 3D canvas via ink input;
receiving, at a first device, ink input manipulating a characteristic of the graphical element to generate a modified graphical element; and
causing the 3D canvas with the modified graphical element to be shared to a second device such that the modified graphical element is editable at the second device via interaction with the 3D canvas.

### Claim 19 (depends on 18)

The method as described in claim 18, wherein said causing comprises causing the 3D canvas to be published to a network location such that the 3D canvas is accessible and editable via the network location.

### Claim 20 (depends on 18)

The method as described in claim 18, wherein the electronic document comprises a web page, and wherein said causing comprises sharing the web page in an editable form to the second device.