# US 9,436,937 B1 · Highlight-based bill processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,436,937 B1
- Document ID: US9436937B1
- Status: Granted
- Kind: B1
- Application number: 14/845,075
- Filed: 2015-09-03
- Granted: 2016-09-06
- Earliest claimed: 2012-08-29
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Sunil H. Madhani; Anu Sreepathy; Samir R. Kakkar
- Web page: https://patentgrep.com/patent/US9436937B1

## CPC

- G06Q 20/102: {Bill distribution or payments}
- G06K 19/06037: {multi-dimensional coding}
- G06Q 20/3276: {using a pictured code, e.g. barcode or QR-code, being read by the M-device}
- G06V 10/235: {based on user input or interaction}
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Document processing utilizing extraneous highlight added to the document. Sections or fields of a document including data utilized for document processing are marked or highlighted for processing. For example, a paper or printed document may be manually highlighted by a highlighter pen, or the highlight may be computer generated. An image of the highlighted bill is acquired. Highlighted sections within the image are identified, and bill data within the highlighted fields is extracted and utilized for bill processing.

## Claims

### Claim 1

A computerized bill processing system, comprising:
a computerized acquisition element configured to generate an image of a printed bill or invoice of a payer that is a recipient of the printed bill or invoice, the printed bill or invoice reflecting an amount owed by the payer recipient to a payee; and
a bill processing application, in communication with the computerized acquisition element or a data store associated with the computerized acquisition element, and configured to:
receive the image comprising at least one section manually highlighted by the payer recipient and at least one unhighlighted section,
determine bill or invoice data within the at least one section that was manually highlighted by the payer recipient based at least in part upon identifying a pre-determined number of contiguous pixels in the image having a color that differs from a background color of the image; and
process the bill or invoice for electronic payment by the payer recipient to the payee based at least in part upon the determined bill or invoice data within the at least one section that was manually highlighted by the payer recipient.

### Claim 2 (depends on 1)

The computerized bill processing system of claim 1, the computerized acquisition element comprising:
a lens; and
a controller or processor configured to acquire the image utilizing the lens and to store the image to the data store.

### Claim 3 (depends on 1)

The computerized bill processing system of claim 1, the computerized acquisition element comprising a component of a mobile communication device of the payer recipient.

### Claim 4 (depends on 1)

The computerized bill processing system of claim 1, wherein the computerized acquisition element and the bill processing application are integrated within a single computing apparatus.

### Claim 5 (depends on 4)

The computerized bill processing system of claim 4, wherein the single computing apparatus is a computing apparatus of the payer recipient.

### Claim 6 (depends on 1)

The computerized bill processing system of claim 1, wherein the computerized acquisition element and the bill processing application are separate components of respective different computing devices.

### Claim 7 (depends on 1)

The computerized bill processing system of claim 1, the computerized acquisition element comprising a camera.

### Claim 8 (depends on 1)

The computerized bill processing system of claim 1, the computerized acquisition element comprising a video recording component, the image comprising a video frame generated by the video component.

### Claim 9 (depends on 1)

The computerized bill processing system of claim 1, the image comprising a screen shot of an image displayed by a computing apparatus executing the bill processing application.

### Claim 10 (depends on 1)

The computerized bill processing system of claim 1, the at least one manually highlighted section comprising a section manually marked with a transparent, fluorescent color.

### Claim 11 (depends on 10)

The computerized bill processing system of claim 10, the transparent, fluorescent color being selected from the group consisting of pink, blue, green, cyan, yellow, orange and purple.

### Claim 12 (depends on 1)

The computerized bill processing system of claim 1, the at least one manually highlighted section comprising a section manually marked by the payer recipient with a marking instrument comprising a highlighter pen.

### Claim 13 (depends on 1)

The computerized bill processing system of claim 1, the at least one manually highlighted section comprising a section manually marked with a marking instrument comprising a wax marking instrument.

### Claim 14 (depends on 1)

The computerized bill processing system of claim 1, the image further comprising at least one section manually highlighted by a source of the printed bill or invoice of the payer recipient.

### Claim 15 (depends on 1)

The computerized bill processing system of claim 1, the image further comprising at least one section manually highlighted by an intermediary that receives an unhighlighted printed bill or invoice from a source and sends the highlighted printed bill to the payer recipient.

### Claim 16 (depends on 1)

The computerized bill processing system of claim 1, the image comprising a plurality of manually highlighted sections, electronic payment of the printed bill or invoice being processed based at least in part upon respective bill or invoice data determined from the plurality of manually highlighted sections.

### Claim 17 (depends on 16)

The computerized bill processing system of claim 16, at least one of the plurality of manually highlighted sections being selected from the group consisting of the payee name, a payment amount owed by the payer recipient to the payee, and a payment due date.

### Claim 18 (depends on 1)

The computerized bill processing system of claim 1, the bill processing application being further configured to:
receive a request for sections of the printed bill or invoice to be manually highlighted,
determine which sections are to be manually highlighted for bill or invoice processing, and
send a reply in response to the request, the reply identifying the determined sections to be manually highlighted.

### Claim 19 (depends on 1)

The computerized bill processing system of claim 1, the bill processing application being configured to process the bill or invoice for electronic payment by being configured to communicate the determined bill or invoice data and electronic bill or invoice payment instructions from a computing apparatus executing the bill processing application through a network to a computer of a financial institution.

### Claim 20 (depends on 1)

The computerized bill processing system of claim 1, further comprising an optical character recognition component in communication with the bill processing application, wherein the optical character recognition component is configured to determine the bill or invoice data.

### Claim 21 (depends on 1)

The computerized bill processing system of claim 1, the bill processing application being configured to determine the bill or invoice data within the manually highlighted section by being configured to:
mask unhighlighted sections of the image;
identify at least one unmasked, manually highlighted section within the masked image; and
extract bill or invoice data from the at least one identified unmasked, manually highlighted section.

### Claim 22 (depends on 21)

The computerized bill processing system of claim 21, the bill processing application being configured to mask the unhighlighted sections by being configured to
determine a first color of a background of an unhighlighted section of the printed bill or invoice;
determine a second color of alpha-numeric data within the printed bill or invoice;
generate an electronic mask, a color of the electronic mask being based at least in part upon the first color and the second color,
wherein the electronic mask covers the unhighlighted sections while the manually highlighted sections remain unmasked.

### Claim 23 (depends on 22)

The computerized bill processing system of claim 22, the color of the electronic mask being a darker color of the first color and the second color.

### Claim 24 (depends on 22)

The computerized bill processing system of claim 22, the color of the electronic mask being a third color that is darker than both of the first color and the second color.

### Claim 25 (depends on 22)

The computerized bill processing system of claim 22, the first color of the background being determined based on respective color data of respective pixels at respective pre-determined locations within the image.

### Claim 26 (depends on 25)

The computerized bill processing system of claim 25, the pre-determined locations comprising respective corners of the image.

### Claim 27 (depends on 1)

The computerized bill processing system of claim 1, the bill processing application being configured to determine the bill or invoice data by being configured to:
determine a background color of an unhighlighted section of the printed bill or invoice,
read pixels in the image,
determine respective color data of the pixels,
identify at least one manually highlighted section based at least in part upon determining that a pre-determined minimum number of read pixels has a color different than the background color, and
extract the bill or invoice data from the at least one manually highlighted section that was identified.

### Claim 28 (depends on 1)

The computerized bill processing system of claim 1, wherein the bill processing application is configured to determine the bill or invoice data within the at least one manually highlighted section without reference to a database of bill or invoice templates.

### Claim 29 (depends on 1)

The computerized bill processing system of claim 1, wherein a field of the bill or invoice containing bill or invoice data is partially highlighted, the bill processing application being further configured to:
resolve the partially highlighted field to identify a complete field that should have been highlighted; and
determine bill or invoice data within the identified complete field,
wherein the printed bill or invoice is processed based at least in part upon the determined bill or invoice data within the identified complete field.

### Claim 30 (depends on 1)

The computerized bill processing system of claim 1, wherein the image is an image of a printed bill or invoice selected from the group consisting of a credit card bill, a utility bill, a phone bill, a loan bill, a home owners association bill, and an insurance bill.

### Claim 31

A computerized bill processing system, comprising:
a computerized acquisition element configured to generate an image of a printed bill or invoice of a payer that is a recipient of the printed bill or invoice, the printed bill or invoice reflecting an amount owed by the payer recipient to a payee; and
a bill processing application, in communication with the computerized acquisition element or a data store associated with the computerized acquisition element, and being configured to:
receive the image comprising:
at least one section manually highlighted by an intermediary that receives the printed bill or invoice unhighlighted from a source and sends the manually highlighted printed bill to the payer recipient, and
at least one unhighlighted section;
determine bill or invoice data within the at least one section that was manually highlighted by the payer recipient based at least in part upon identifying a pre-determined number of contiguous pixels in the image having a color that differs from a background color of the image; and
process the bill or invoice for electronic payment by the payer recipient to the payee based at least in part upon the determined bill or invoice data within the at least one section that was manually highlighted by the payer recipient.

### Claim 32

A computerized bill processing system, comprising:
a computerized acquisition element configured to generate an image of a printed bill or invoice of a payer that is a recipient of the printed bill or invoice, the printed bill or invoice reflecting an amount owed by the payer recipient to a payee; and
a bill processing application, in communication with the computerized acquisition element or a data store associated with the computerized acquisition element, and being configured to:
receive a request for sections of the printed bill or invoice to be manually highlighted,
determine which sections are to be manually highlighted for bill or invoice processing,
send a reply in response to the request, the reply identifying the determined sections to be manually highlighted,
receive the image of a printed bill or invoice of a payer that is a recipient of the printed bill or invoice, the printed bill or invoice reflecting an amount owed by the payer recipient to a payee, the received image comprising at least one section manually highlighted by the payer recipient and at least one unhighlighted section,
determine bill or invoice data within the at least one section that was manually highlighted by the payer recipient, and
process the bill or invoice for electronic payment by the payer recipient to the payee based at least in part upon the determined bill or invoice data within the at least one section that was manually highlighted by the payer recipient.

### Claim 33

A computerized bill processing system, comprising:
a computerized acquisition element configured to generate an image of a printed bill or invoice of a payer that is a recipient of the printed bill or invoice, the printed bill or invoice reflecting an amount owed by the payer recipient to a payee; and
a bill processing application, in communication with the computerized acquisition element or a data store associated with the computerized acquisition element, and being configured to:
receive the image of the printed bill or invoice, the received image comprising at least one section manually highlighted by the payer recipient and at least one unhighlighted section,
determine bill or invoice data within the section that was manually highlighted by the payer recipient by being configured to mask unhighlighted sections of the image by determining a first color of a background of an unhighlighted section of the printed bill or invoice, determining a second color of alpha-numeric data within the printed bill or invoice, and generating an electronic mask, a color of the electronic mask being based at least in part upon the first color and the second color, wherein the electronic mask covers the unhighlighted sections while the at least one manually highlighted section remains unmasked,
identify at least one unmasked, manually highlighted section within the masked image;
extract bill or invoice data from the at least one identified unmasked, manually highlighted section; and
process the bill or invoice for electronic payment by the payer recipient to the payee based at least in part upon the determined bill or invoice data within the section that was manually highlighted by the payer recipient.

### Claim 34

A computerized bill processing system, comprising:
a computerized acquisition element configured to generate an image of a printed bill or invoice of a payer that is a recipient of the printed bill or invoice, the printed bill or invoice reflecting an amount owed by the payer recipient to a payee; and
a bill processing application, in communication with the computerized acquisition element or a data store associated with the computerized acquisition element, and being configured to:
receive the image of the printed bill or invoice, the received image comprising at least one section manually highlighted by the payer recipient and at least one unhighlighted section, wherein a field of the bill or invoice containing bill or invoice data is partially highlighted,
resolve the partially highlighted field to identify a complete field that should have been highlighted;
determine bill or invoice data within the identified complete field, the printed bill or invoice being processed based at least in part upon the determined bill or invoice data within the identified complete field; and
process the bill or invoice for electronic payment by the payer recipient to the payee based at least in part upon the determined bill or invoice data within the at least one section that was manually highlighted by the payer recipient.