# US 11,113,517 B2 · Object detection and segmentation for inking applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,517 B2 - Document ID: US11113517B2 - Status: Granted - Kind: B2 - Application number: 16/360,006 - Filed: 2019-03-20 - Published: 2020-09-24 - Granted: 2021-09-07 - Earliest claimed: 2019-03-20 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Yixin Shi; Aman Orazaev; Tianyi Chen; Sheng Yi - Web page: https://patentgrep.com/patent/US11113517B2 ## CPC - G06V 30/347: {Sampling; Contour coding; Stroke extraction} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0418: {using chaos or fractal principles} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/09: Supervised learning - G06T 7/11: Region-based segmentation - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/273: {removing elements interfering with the pattern to be recognised} - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06V 30/153: {using recognition of characters or words} - G06V 30/36: {Matching; Classification} - G06V 30/413: Classification of content, e.g. text, photographs or tables - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An ink parsing system receives ink strokes at an inking device input and render the received ink strokes into an image in a pixel space. Writing strokes are detected in the image and labeled. Pixels corresponding to the labeled writing strokes are removed from the image. Drawing strokes in the image having the removed pixels are detected using and labeled. Writing objects and drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes are output. A digital ink parsing pipeline is thereby provided having accurate ink stroke detection and segmentation. ## Claims ### Claim 1 An ink parsing system comprising: a memory associated with a computing device, the memory including a writing detector component and a drawing detector component; and at least one processor that executes an ink parsing engine that uses the writing detector component and the drawing detector component to: receive ink strokes at an inking device input; render the received ink strokes into an image; detect one or more writing strokes in the image using the writing detector component and label the writing strokes; remove one or more pixels corresponding to the labeled writing strokes from the image; detect one or more drawing strokes in the image, after detecting the one or more writing strokes and removing the one or more pixels corresponding to the labeled writing strokes using the drawing detector component, and label the drawing strokes; and output one or more writing objects and one or more drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes. ### Claim 2 (depends on 1) The ink parsing system of claim 1, wherein the at least one processor executes the ink parsing engine to perform a semantic segmentation with the writing detector component, the semantic segmentation using pixel-wise classification to detect the writing strokes. ### Claim 3 (depends on 2) The ink parsing system of claim 2, wherein the at least one processor executes the ink parsing engine to use predicted writing masks to label the writing strokes. ### Claim 4 (depends on 2) The ink parsing system of claim 2, wherein the at least one processor executes the ink parsing engine to use a neural network having a plurality of layers, each with a fixed number of kernels, to perform the semantic segmentation. ### Claim 5 (depends on 1) The ink parsing system of claim 1, wherein the at least one processor executes the ink parsing engine to perform drawing stroke detection with the drawing detector component using a convolutional network. ### Claim 6 (depends on 5) The ink parsing system of claim 5, wherein the at least one processor executes the ink parsing engine to decode detected drawing strokes using a plurality of bounding boxes with corresponding predicted labels, and further uses a decision tree as a binary classifier to detect the drawing strokes. ### Claim 7 (depends on 1) The ink parsing system of claim 1, wherein (i) the ink parsing engine has a parsing pipeline configured to perform detection using the writing detector component before performing detection using the drawing detector component, (ii) the writing detector is configured to remove the one or more pixels to generate a modified inking input, (iii) the drawing detector component is configured to perform detection of the one or more drawing strokes on the modified inking input that, comprises a non-writing image, and (iv) the modified inking input, the one or more writing objects, and the one or more drawing objects are output to a next stage for processing. ### Claim 8 A computerized method for digital ink parsing, the computerized method comprising: receiving ink strokes at an inking device input; rendering the received ink strokes into an image; detecting one or more writing strokes in the image and label the writing strokes; removing one or more pixels corresponding to the labeled writing strokes from the image; detecting one or more drawing strokes in the image, after detecting the one or more writing strokes and removing the one or more pixels corresponding to the labeled writing strokes, and label the drawing strokes; and outputting one or more writing objects and one or more drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes. ### Claim 9 (depends on 8) The computerized method of claim 8, further comprising performing a semantic segmentation using pixel-wise classification to detect the writing strokes. ### Claim 10 (depends on 9) The computerized method of claim 9, further comprising using predicted writing masks to label the writing strokes. ### Claim 11 (depends on 9) The computerized method of claim 9, further comprising using a neural network having a plurality of layers, each with a fixed number of kernels, to perform the semantic segmentation. ### Claim 12 (depends on 8) The computerized method of claim 8, further comprising performing drawing stroke detection using a convolutional network. ### Claim 13 (depends on 12) The computerized method of claim 12, further comprising decoding detected drawing strokes using a plurality of bounding boxes with corresponding predicted labels, and using a decision tree as a binary classifier to detect the drawing strokes. ### Claim 14 (depends on 8) The computerized method of claim 8, further comprising using a parsing pipeline that is configured to detect writing strokes before detecting drawing strokes. ### Claim 15 One or more computer storage media having computer-executable instructions for digital ink parsing that, upon execution by at least one processor, cause the at least one processor to at least: receive ink strokes at an inking device input; render the received ink strokes into an image; detect one or more writing strokes in the image and label the writing strokes; remove one or more pixels corresponding to the labeled writing strokes from the image; detect one or more drawing strokes in the image, after detecting the one or more writing strokes and removing the one or more pixels corresponding to the labeled writing strokes, and label the drawing strokes; and output one or more writing objects and one or more drawing objects corresponding, respectively, to the labeled writing strokes and the labeled drawing strokes. ### Claim 16 (depends on 15) The one or more computer storage media of claim 15 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least perform a semantic segmentation using pixel-wise classification to detect the writing strokes. ### Claim 17 (depends on 16) The one or more computer storage media of claim 16 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least use predicted writing masks to label the writing strokes. ### Claim 18 (depends on 16) The one or more computer storage media of claim 16 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least use a neural network having a plurality of layers, each with a fixed number of kernels, to perform the semantic segmentation. ### Claim 19 (depends on 15) The one or more computer storage media of claim 15 having further computer-executable instructions that, upon execution by the at least one processor, cause the at least processor to at least perform drawing stroke detection using a convolutional network, decode detected drawing strokes using a plurality of bounding boxes with corresponding predicted labels, and use a decision tree as a binary classifier to detect the drawing strokes. ### Claim 20 (depends on 15) The one or more computer storage media of claim 15 having further computer-executable instructions that, upon execution by the at least processor, cause the at least processor to at least use a parsing pipeline that is configured to detect writing strokes before detecting drawing strokes.