# US 2015/0106699 A1 · Note recognition for overlapping physical notes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0106699 A1
- Document ID: US20150106699A1
- Status: Published application
- Kind: A1
- Application number: 14/514,473
- Filed: 2014-10-15
- Published: 2015-04-16
- Earliest claimed: 2013-10-16
- Assignees: 3M Innovative Properties Company
- Applicants: 3M Innovative Properties Company
- Inventors: Guruprasad Somasundaram; Ravishankar Sivalingam; David M. Mahli; Cristin E. Moran; Richard J. Moore; Pontus Axelsson; Michael Rotstein; Roland Persson
- Web page: https://patentgrep.com/patent/US20150106699A1

## CPC

- H04L 51/08: Annexed information, e.g. attachments
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/171: by use of digital ink
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- H04L 12/1827: {Network arrangements for conference optimisation or adaptation}
- H04L 51/10: Multimedia information
- H04N 1/04: Scanning arrangements {, i.e. arrangements for the displacement of active reading or reproducing elements relative to the original or reproducing medium, or vice versa}(H04N1/387 takes precedence)
- H04L 51/066: Format adaptation, e.g. format conversion or compression
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Techniques for creating and manipulating software notes representative of physical notes are described. A computing device includes a processor, an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes, and an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image. The image processing engine determines a boundary of each note in the plurality of overlapping physical notes in the input image, and generates a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

## Claims

### Claim 1

A method comprising:
receiving, by a computing device, an input image of an environment having a plurality of overlapping physical notes;
processing the input image with the computing device to identify each of the overlapping physical notes in the input image including determining a boundary of each of the overlapping physical notes in the input image; and
generating, by the computing device, a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining, by the computing device, whether the plurality of overlapping physical notes in the input image comprise a substantially similar background color;
upon determining that the plurality of overlapping physical notes includes one or more physical notes with different background colors, applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and
generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining, by the computing device, whether the plurality of overlapping physical notes in the input image comprise a substantially similar background color and a border color different from the background color;
upon determining that the plurality of overlapping physical notes includes one or more physical notes with a border color, applying a color segmenting process, by the computing device, to segment the overlapping physical notes; and
generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented overlapping physical notes.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
displaying, by the computing device, the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and
displaying, by the computing device, the plurality of segmented digital notes on a second portion of the graphical user interface.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining, by the computing device, whether the plurality of overlapping physical notes in the input image comprise a substantially similar background color;
upon determining that the plurality of overlapping physical notes includes physical notes with a substantially similar background color, determining, by the computing device, whether at least one of the overlapping physical notes in the input image includes one or more fiducial markers using a marker detection module;
based on the determination that at least one of the overlapping physical notes in the input image includes one or more fiducial markers, detecting one or more fiducial markers which indicate the size and shape of the note;
determining, by the computing device and based on the fiducial markers, the boundary of each of the physical notes in the plurality of overlapping physical notes;
segmenting, by the computing device and based on the determined boundaries, each of the physical notes in the plurality of overlapping physical notes;
generating, by the computing device, a plurality of segmented digital notes corresponding to the plurality of segmented physical notes;
displaying, by the computing device, the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and
displaying, by the computing device, the plurality of segmented digital notes on a second portion of the graphical user interface.

### Claim 6 (depends on 1)

The method of claim 1, wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the size and shape of each note in the plurality of overlapping physical notes.

### Claim 7 (depends on 1)

The method of claim 1, wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the plurality of physical notes according to color spaces associated with background colors of the plurality of notes.

### Claim 8 (depends on 1)

The method of claim 1, wherein identifying, by the computing device, the plurality of overlapping physical notes includes identifying the plurality of notes according to a pattern recognition algorithm.

### Claim 9 (depends on 8)

The method of claim 8, wherein the pattern recognition algorithm includes at least one of a group consisting of:
a support vector machine;
a cross-correlation; and
a template matching.

### Claim 10

A non-transitory computer-readable medium comprising instructions for causing a programmable processor to:
receive, by a computing device, an input image of an environment having a plurality of overlapping physical notes;
process the input image with the computing device to identify each of the overlapping physical notes in the input image including determining a boundary of each of the overlapping physical notes in the input image; and
generate, by the computing device, a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

### Claim 11

A computing device comprising:
a processor;
an image collection module executable by the processor and configured to receive an input image of an environment having a plurality of overlapping physical notes; and
an image processing engine executable by the processor and configured to process the input image with the computing device to identify the plurality of overlapping physical notes in the input image, determine a boundary of each note in the plurality of overlapping physical notes in the input image, and generate a plurality of digital notes corresponding to the determined boundary of each of the overlapping physical notes identified in the input image.

### Claim 12 (depends on 11)

The device of claim 11, wherein the image collection module includes an image capture device configured to capture the input image.

### Claim 13 (depends on 11)

The device of claim 11, wherein the image collection module includes a communication interface configured to receive the input image.

### Claim 14 (depends on 13)

The device of claim 13, wherein the communication interface includes a wireless communication device configured to receive the input image.

### Claim 15 (depends on 13)

The device of claim 13, wherein the communication interface includes a wired communication device configured to receive the input image.

### Claim 16 (depends on 11)

The device of claim 11, wherein the image processing engine is further configured to:
determine whether the plurality of overlapping physical notes in the input image comprise a substantially similar background color;
upon determining that the plurality of overlapping physical notes includes two or more physical notes with different background colors, applying a color segmenting process to segment each physical note in the plurality of overlapping physical notes; and
generate a plurality of segmented digital notes corresponding to the plurality of segmented notes.

### Claim 17 (depends on 16)

The device of claim 16, where the image processing engine further configured to:
display the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and
display the plurality of segmented digital notes on a second portion of the graphical user interface.

### Claim 18 (depends on 11)

The device of claim 11, wherein the image processing engine is further configured to:
determine whether the plurality of overlapping physical notes in the input image comprise a substantially similar background color and a different border color;
upon determining that the plurality of overlapping physical notes includes two or more physical notes with different border colors, applying a color segmenting process to segment each physical note in the plurality of overlapping physical notes; and
generate a plurality of segmented digital notes corresponding to the plurality of segmented notes.

### Claim 19 (depends on 18)

The device of claim 18, where the image processing engine further configured to:
display the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and
display the plurality of segmented digital notes on a second portion of the graphical user interface.

### Claim 20 (depends on 11)

The device of claim 11, wherein the image processing engine further configured to:
determine whether the plurality of overlapping physical notes in the input image comprise a substantially similar background color;
upon determining that the plurality of overlapping physical notes includes physical notes with a substantially similar background color, determining whether at least one of the plurality of physical notes in the input image includes one or more fiducial markers using a marker detection module,
based on the determination that at least one of the plurality of notes in the input image includes one or more fiducial markers, detecting one or more fiducial markers which indicate the size and shape of the note;
determine based on the fiducial markers, the boundary of each physical note in the plurality of overlapping physical notes;
segment based on the determined boundaries, each of the physical notes in the plurality of overlapping physical notes;
generate a plurality of segmented digital notes corresponding to the plurality of segmented physical notes;
display the input image including the plurality of segmented digital notes in place of the plurality of overlapping physical notes on a first portion of a graphical user interface; and
display the plurality of segmented digital notes on a second portion of the graphical user interface.

### Claim 21 (depends on 11)

The device of claim 11, wherein identifying the plurality of overlapping physical notes includes identifying the size and shape of each note in the plurality of overlapping physical notes.

### Claim 22 (depends on 11)

The device of claim 11, wherein identifying the plurality of overlapping physical notes includes identifying the plurality of notes according to color spaces associated with background colors of the plurality of notes.

### Claim 23 (depends on 11)

The device of claim 11, wherein identifying the plurality of overlapping physical notes includes identifying the plurality of notes according to a pattern recognition algorithm.

### Claim 24 (depends on 23)

The device of claim 23, wherein the pattern recognition algorithm includes at least one of a group consisting of:
a support vector machine;
a cross-correlation; and
a template matching.

### Claim 25

A method comprising:
receiving, by a computing device, an input image of an environment having a plurality of physical notes, wherein each of the physical notes comprises a background color and a border color different from the background color of the note;
applying a color segmenting process to the input image to identify boundaries for the physical notes in the input image by segmenting the physical notes based on the background color and border color of each of the physical notes; and
generating, by the computing device, a plurality of segmented digital notes corresponding to the segmented substantially adjacent physical notes.

### Claim 26 (depends on 25)

The method of claim 25, wherein two or more of the plurality of physical notes are substantially adjacent physical notes.

### Claim 27 (depends on 25)

The method of claim 25, wherein the boundaries for two or more of the plurality of physical notes are at least partially overlapping.
