# US 10,503,396 B2 · Device, method, and graphical user interface for simulating and interacting with handwritten text > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,396 B2 - Document ID: US10503396B2 - Status: Granted - Kind: B2 - Application number: 15/978,127 - Filed: 2018-05-12 - Published: 2018-12-06 - Granted: 2019-12-10 - Earliest claimed: 2017-06-02 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Daniel E. Gobera Rubalcava; Ryan S. Dixon - Web page: https://patentgrep.com/patent/US10503396B2 ## CPC - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/03545: {Pens or stylus} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 2203/04105: Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position - G06F 2203/04106: Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A device displays a drawing region. While displaying the drawing region, the device detects a sequence of drawing inputs on a touch-sensitive display. In response to the sequence of drawing inputs, the device draws a plurality of strokes in the drawing region. The plurality of strokes correspond to a plurality of characters. After detecting the sequence of drawing inputs, the device detects a predefined gesture that corresponds to a request to perform an operation based on the plurality of characters represented by the plurality of strokes. In response to detecting the predefined gesture, the device concurrently displays a first visual prompt indicating that a first subset of one or more characters in the plurality of characters can be used to perform the operation and a second visual prompt indicating that a second subset of one or more characters in the plurality of characters can be used to perform the operation. ## Claims ### Claim 1 A method comprising: at a device including a touch-sensitive display: displaying, on the touch-sensitive display, a drawing region; while displaying, the drawing region, detecting a sequence of drawing inputs on the touch-sensitive display; in response to detecting the sequence of drawing inputs, drawing a plurality of strokes in the drawing region, wherein the plurality of strokes correspond to a plurality of characters; after detecting the sequence of drawing inputs on the touch-sensitive display, detecting, on the touch-sensitive display, a predefined gesture that corresponds to a request to perform an operation based on the plurality of characters represented by the plurality of strokes; and in response to detecting the predefined gesture, concurrently displaying, on the touch-sensitive display: a first visual prompt indicating that a first subset of one or more characters in the plurality of characters can be used to perform the operation; and a second visual prompt indicating that a second subset of one or more characters in the plurality of characters can be used to perform the operation. ### Claim 2 (depends on 1) The method of claim 1, further comprising: displaying, on the touch-sensitive display, a result field that includes a resultant value for the operation, wherein the resultant value is a function of the first subset of one or more characters and the second subset of one or more characters. ### Claim 3 (depends on 2) The method of claim 2, wherein the predefined gesture is detected at a particular location, and wherein the result field is displayed at the particular location. ### Claim 4 (depends on 2) The method of claim 2, further comprising: detecting a user selection of the result field; in response to receiving the user selection of the result field, displaying an overlay that includes: one or more mathematical symbols that are associated with the operation; a first numerical value that corresponds with the first subset of one or more characters; and a second numerical value that corresponds with the second subset of one or more characters. ### Claim 5 (depends on 4) The method of claim 4, further comprising: receiving a user input changing one or more of the one or more mathematical symbols, the first numerical value, and the second numerical value; and in response to the user input, updating the resultant value based on the changes to the one or more of the one or more mathematical symbols, the first numerical value and the second numerical value. ### Claim 6 (depends on 2) The method of claim 2, wherein displaying the result field comprises: rendering the resultant value in a simulated handwritten text that is within a degree of similarity to the plurality of strokes displayed in the drawing region. ### Claim 7 (depends on 2) The method of claim 2, wherein the first subset of one or more characters and the second subset of one or more characters are associated with one or more of a symbol and a unit of measurement, and wherein the result field further includes one or more of the symbol and the unit of measurement. ### Claim 8 (depends on 2) The method of claim 2, wherein the first subset of one or more characters is associated with a first unit of measurement and the second subset of one or more characters is associated with a second unit of measurement that is different from the first unit of measurement, and wherein displaying the result field comprises: converting the first subset of one or more characters from the first unit of measurement into the second unit of measurement; and determining the resultant value based on a function of the converted first subset of one or more characters and the second subset of one or more characters. ### Claim 9 (depends on 2) The method of claim 2, further comprising: receiving a request to perform a subsequent operation; and performing the subsequent operation based on a function of the resultant value. ### Claim 10 (depends on 1) The method of claim 1, wherein the detecting the sequence of drawing inputs includes receiving the sequence of drawing inputs via a stylus, and wherein detecting the predefined gesture includes receiving the predefined gesture via a digit of a user. ### Claim 11 (depends on 1) The method of claim 1, wherein: the operation includes a formula; the first visual prompt indicates that the first subset of one or more characters represents a first numerical value that can be included in the formula; and the second visual prompt indicates that the second subset of one or more characters represents a second numerical value that can be included in the formula. ### Claim 12 (depends on 11) The method of claim 11, further comprising: receiving a user input selecting the first visual prompt; in response to the user input selecting the first visual prompt, including the first numerical value in the formula; receiving a user input selecting the second visual prompt; and in response to the user input selecting the second visual prompt, including the second numerical value in the formula. ### Claim 13 (depends on 11) The method of claim 11, wherein the first visual prompt includes a first affordance for changing the first numerical value, and wherein the second visual prompt includes a second affordance for changing the second numerical value. ### Claim 14 (depends on 13) The method of claim 13, wherein the first affordance for changing the first numerical value includes the first numerical value, and wherein the second affordance for changing the second numerical value includes the second numerical value. ### Claim 15 (depends on 13) The method of claim 13, further comprising: in response to a user input changing one or more of the first numerical value and the second numerical value, updating the formula based on a change to one or more of the first numerical value and the second numerical value. ### Claim 16 (depends on 1) The method of claim 1, further comprising: detecting, via a microphone, a voice input that indicates the operation. ### Claim 17 An electronic device comprising: a display; an input device; one or processors; non-transitory memory storing wherein the one or more programs, the one or more programs including instructions that, when executed by the one or more processors, cause the electronic device to: display, on the touch-sensitive display, a drawing region; while displaying, the drawing region, detect a sequence of drawing inputs on the touch-sensitive display; in response to detecting the sequence of drawing inputs, drawing a plurality of strokes in the drawing region, wherein the plurality of strokes correspond to a plurality of characters; after detecting the sequence of drawing inputs on the touch-sensitive display, detect, on the touch-sensitive display, a predefined gesture that corresponds to a request to perform an operation based on the plurality of characters represented by the plurality of strokes; and in response to detecting the predefined gesture, concurrently display, on the touch-sensitive display: a first visual prompt indicating that a first subset of one or more characters in the plurality of characters can be used to perform the operation; and a second visual prompt indicating that a second subset of one or more characters in the plurality of characters can be used to perform the operation. ### Claim 18 (depends on 17) The electronic device of claim 17, wherein the instructions further cause the electronic device to: display, on the touch-sensitive display, a result field that includes a resultant value for the operation, wherein the resultant value is a function of the first subset of one or more characters and the second subset of one or more characters. ### Claim 19 (depends on 18) The electronic device of claim 18, wherein the instructions further cause the electronic device to: detect a user selection of the result field; in response to receiving the user selection of the result field, display an overlay that includes: one or more mathematical symbols that are associated with the operation; a first numerical value that corresponds with the first subset of one or more characters; and a second numerical value that corresponds with the second subset of one or more characters. ### Claim 20 (depends on 19) The electronic device of claim 19, wherein the instructions further cause the electronic device to: receive a user input changing one or more of the one or more mathematical symbols, the first numerical value, and the second numerical value; and in response to the user input, update the resultant value based on the changes to the one or more of the one or more mathematical symbols, the first numerical value and the second numerical value. ### Claim 21 (depends on 18) The electronic device of claim 18, wherein the predefined gesture is detected at a particular location, and wherein the result field is displayed at the particular location. ### Claim 22 (depends on 18) The electronic device of claim 18, wherein displaying the result field comprises: rendering the resultant value in a simulated handwritten text that is within a degree of similarity to the plurality of strokes displayed in the drawing region. ### Claim 23 (depends on 18) The electronic device of claim 18, wherein the first subset of one or more characters and the second subset of one or more characters are associated with one or more of a symbol and a unit of measurement, and wherein the result field further includes one or more of the symbol and the unit of measurement. ### Claim 24 (depends on 18) The electronic device of claim 18, wherein the first subset of one or more characters is associated with a first unit of measurement and the second subset of one or more characters is associated with a second unit of measurement that is different from the first unit of measurement, and wherein displaying the result field comprises: converting the first subset of one or more characters from the first unit of measurement into the second unit of measurement; and determining the resultant value based on a function of the converted first subset of one or more characters and the second subset of one or more characters. ### Claim 25 (depends on 18) The electronic device of claim 18, wherein the instructions further cause the electronic device to: receive a request to perform a subsequent operation; and perform the subsequent operation based on a function of the resultant value. ### Claim 26 (depends on 17) The electronic device of claim 17, wherein the detecting the sequence of drawing inputs includes receiving the sequence of drawing inputs via a stylus, and wherein detecting the predefined gesture includes receiving the predefined gesture via a digit of a user. ### Claim 27 (depends on 17) The electronic device of claim 17, wherein: the operation includes a formula; the first visual prompt indicates that the first subset of one or more characters represents a first numerical value that can be included in the formula; and the second visual prompt indicates that the second subset of one or more characters represents a second numerical value that can be included in the formula. ### Claim 28 (depends on 27) The electronic device of claim 27, wherein the instructions further cause the electronic device to: receive a user input selecting the first visual prompt; in response to the user input selecting the first visual prompt, include the first numerical value in the formula; receive a user input selecting the second visual prompt; and in response to the user input selecting the second visual prompt, include the second numerical value in the formula. ### Claim 29 (depends on 27) The electronic device of claim 27, wherein the first visual prompt includes a first affordance for changing the first numerical value, and wherein the second visual prompt includes a second affordance for changing the second numerical value. ### Claim 30 (depends on 29) The electronic device of claim 29, wherein the first affordance for changing the first numerical value includes the first numerical value, and wherein the second affordance for changing the second numerical value includes the second numerical value. ### Claim 31 (depends on 29) The electronic device of claim 29, wherein the instructions further cause the electronic device to: in response to a user input changing one or more of the first numerical value and the second numerical value, update the formula based on a change to one or more of the first numerical value and the second numerical value. ### Claim 32 (depends on 17) The electronic device of claim 17, wherein the instructions further cause the electronic device to: detect, via a microphone, a voice input that indicates the operation. ### Claim 33 A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with a display, and an input device, cause the electronic device to: display, on the touch-sensitive display, a drawing region; while displaying, the drawing region, detect a sequence of drawing inputs on the touch-sensitive display; in response to detecting the sequence of drawing inputs, drawing a plurality of strokes in the drawing region, wherein the plurality of strokes correspond to a plurality of characters; after detecting the sequence of drawing inputs on the touch-sensitive display, detect, on the touch-sensitive display, a predefined gesture that corresponds to a request to perform an operation based on the plurality of characters represented by the plurality of strokes; and in response to detecting the predefined gesture, concurrently display, on the touch-sensitive display: a first visual prompt indicating that a first subset of one or more characters in the plurality of characters can be used to perform the operation; and a second visual prompt indicating that a second subset of one or more characters in the plurality of characters can be used to perform the operation. ### Claim 34 (depends on 33) The non-transitory computer readable storage medium of claim 33, wherein the instructions further cause the electronic device to: display, on the touch-sensitive display, a result field that includes a resultant value for the operation, wherein the resultant value is a function of the first subset of one or more characters and the second subset of one or more characters. ### Claim 35 (depends on 34) The non-transitory computer readable storage medium of claim 34, wherein the predefined gesture is detected at a particular location, and wherein the result field is displayed at the particular location. ### Claim 36 (depends on 34) The non-transitory computer readable storage medium of claim 34, wherein the instructions further cause the electronic device to: detect a user selection of the result field; in response to receiving the user selection of the result field, display an overlay that includes: one or more mathematical symbols that are associated with the operation; a first numerical value that corresponds with the first subset of one or more characters; and a second numerical value that corresponds with the second subset of one or more characters. ### Claim 37 (depends on 36) The non-transitory computer readable storage medium of claim 36, wherein the instructions further cause the electronic device to: receive a user input changing one or more of the one or more mathematical symbols, the first numerical value, and the second numerical value; and in response to the user input, update the resultant value based on the changes to the one or more of the one or more mathematical symbols, the first numerical value and the second numerical value. ### Claim 38 (depends on 34) The non-transitory computer readable storage medium of claim 34, wherein displaying the result field comprises: rendering the resultant value in a simulated handwritten text that is within a degree of similarity to the plurality of strokes displayed in the drawing region. ### Claim 39 (depends on 34) The non-transitory computer readable storage medium of claim 34, wherein the first subset of one or more characters and the second subset of one or more characters are associated with one or more of a symbol and a unit of measurement, and wherein the result field further includes one or more of the symbol and the unit of measurement. ### Claim 40 (depends on 34) The non-transitory computer readable storage medium of claim 34, wherein the first subset of one or more characters is associated with a first unit of measurement and the second subset of one or more characters is associated with a second unit of measurement that is different from the first unit of measurement, and wherein displaying the result field comprises: converting the first subset of one or more characters from the first unit of measurement into the second unit of measurement; and determining the resultant value based on a function of the converted first subset of one or more characters and the second subset of one or more characters. ### Claim 41 (depends on 34) The non-transitory computer readable storage medium of claim 34, wherein the instructions further cause the electronic device to: receive a request to perform a subsequent operation; and perform the subsequent operation based on a function of the resultant value. ### Claim 42 (depends on 33) The non-transitory computer readable storage medium of claim 33, wherein the detecting the sequence of drawing inputs includes receiving the sequence of drawing inputs via a stylus, and wherein detecting the predefined gesture includes receiving the predefined gesture via a digit of a user. ### Claim 43 (depends on 33) The non-transitory computer readable storage medium of claim 33, wherein: the operation includes a formula; the first visual prompt indicates that the first subset of one or more characters represents a first numerical value that can be included in the formula; and the second visual prompt indicates that the second subset of one or more characters represents a second numerical value that can be included in the formula. ### Claim 44 (depends on 43) The non-transitory computer readable storage medium of claim 43, wherein the instructions further cause the electronic device to: receive a user input selecting the first visual prompt; in response to the user input selecting the first visual prompt, include the first numerical value in the formula; receiving a user input selecting the second visual prompt; and in response to the user input selecting the second visual prompt, include the second numerical value in the formula. ### Claim 45 (depends on 43) The non-transitory computer readable storage medium of claim 43, wherein the first visual prompt includes a first affordance for changing the first numerical value, and wherein the second visual prompt includes a second affordance for changing the second numerical value. ### Claim 46 (depends on 45) The non-transitory computer readable storage medium of claim 45, wherein the first affordance for changing the first numerical value includes the first numerical value, and wherein the second affordance for changing the second numerical value includes the second numerical value. ### Claim 47 (depends on 45) The non-transitory computer readable storage medium of claim 45, wherein the instructions further cause the electronic device to: in response to a user input changing one or more of the first numerical value and the second numerical value, update the formula based on a change to one or more of the first numerical value and the second numerical value. ### Claim 48 (depends on 33) The non-transitory computer readable storage medium of claim 33, wherein the instructions further cause the electronic device to: detect, via a microphone, a voice input that indicates the operation.