Back to results
US 12,657,943 B2Granted
Method and system for extracting information from documents via eye gaze tracking
AssigneeJPMorgan Chase Bank, N.A.
InventorsNancy Thomas, Daniel Borrajo
Life of this patent
Estimating the term…
Abstract
A method and system for using eye gaze tracking to extract information in textual form from documents is provided. The method includes: receiving an image that corresponds to a document; receiving, from an eye-tracking sensor configured to detect a sequence of eye-gaze positions on the document as a function of time, a sequence of measurements that correspond to a human reading of the document; determining, based on the received sequence of measurements, a region of the document that is being read by a human; and extracting the textual information that corresponds to the region.
Claims13
A method for extracting textual information from a document, the method being implemented by at least one processor, the method comprising:
- receiving, by the at least one processor, an image that corresponds to a document;
- receiving, by the at least one processor from an eye-tracking sensor, a sequence of measurements of eye-gaze positions that correspond to a reading of the document by a human, wherein the eye-gaze positions are detected by emitting an infrared light and detecting a reflection off a cornea of the human;
- identifying, by the at least one processor, at least one fixation and at least one saccade based on the sequence of measurements, wherein the at least one fixation is identified by calculating a distance between an eye-gaze position and a previous eye-gaze position divided by elapsed time between the eye-gaze position and the previous eye-gaze position;
- determining, by the at least one processor based on the received sequence of measurements and the identified at least one fixation and at least one saccade, at least one region of the document that is being read by the human, wherein the determining includes:
- drawing, for each textual unit included in the document, a respective box that surrounds a corresponding textual unit, in order to generate a plurality of boxes;
- determining, based on a predetermined set of criteria, whether at least a first box from among the plurality of boxes is to be merged with at least one additional box from among the plurality of boxes;
- and
- merging at least a subset of the plurality of boxes into a plurality of box groups based on a result of the determining with respect to the plurality of boxes, wherein the at least one region of the document corresponding to one or more of the plurality of box groups, and wherein the subset of the plurality of boxes are selected for the merging based on a number of eye-gaze points included in each box included in the subset;
- and
- extracting, by the at least one processor, the textual information that corresponds to the determined at least one region.
7 more elements
A computing apparatus for extracting textual information from a document, the computing apparatus comprising:
- a processor;
- a memory;
- and
- a communication interface coupled to each of the processor and the memory,
- wherein the processor is configured to:
- receive, via the communication interface, an image that corresponds to a document;
- receive, via the communication interface from an eye-tracking sensor, a sequence of measurements of eye-gaze positions that correspond to a reading of the document by a human, wherein the eye-gaze positions are detected by emitting an infrared light and detecting a reflection off a cornea of the human;
- identify at least one fixation and at least one saccade based on the sequence of measurements, wherein the at least one fixation is identified by calculating a distance between an eye-gaze position and a previous eye-gaze position divided by elapsed time between the eye-gaze position and the previous eye-gaze position;
- determine, based on the received sequence of measurements and the identified at least one fixation and at least one saccade, at least one region of the document that is being read by the human, wherein the at least one region of the document is determined by:
- drawing, for each textual unit included in the document, a respective box that surrounds a corresponding textual unit, in order to generate a plurality of boxes;
- determining, based on a predetermined set of criteria, whether at least a first box from among the plurality of boxes is to be merged with at least one additional box from among the plurality of boxes;
- and
- merging at least a subset of the plurality of boxes into a plurality of box groups based on a result of the determining with respect to the plurality of boxes, wherein the at least one region of the document corresponding to one or more of the plurality of box groups, and wherein the subset of the plurality of boxes are selected for the merging based on a number of eye-gaze points included in each box included in the subset;
- and
- extract the textual information that corresponds to the determined at least one region.
12 more elements
A non-transitory computer readable storage medium storing instructions for extracting textual information from a document, the storage medium comprising executable code which, when executed by a processor, causes the processor to:
- receive an image that corresponds to a document;
- receive, from an eye-tracking sensor, a sequence of measurements of eye-gaze positions that correspond to a reading of the document by a human, wherein the eye-gaze positions are detected by emitting an infrared light and detecting a reflection off a cornea of the human;
- identify at least one fixation and at least one saccade based on the sequence of measurements, wherein the at least one fixation is identified by calculating a distance between an eye-gaze position and a previous eye-gaze position divided by elapsed time between the eye-gaze position and the previous eye-gaze position;
- determine, based on the received sequence of measurements and the identified at least one fixation and at least one saccade, at least one region of the document that is being read by the human, wherein the at least one region of the document is determined by:
- drawing, for each textual unit included in the document, a respective box that surrounds the corresponding textual unit, in order to generate a plurality of boxes;
- determining, based on a predetermined set of criteria, whether at least a first box from among the plurality of boxes is to be merged with at least one additional box from among the plurality of boxes;
- and
- merging at least a subset of the plurality of boxes into a plurality of box groups based on a result of the determining with respect to the plurality of boxes, wherein the at least one region of the document corresponding to one or more of the plurality of box groups, and wherein the subset of the plurality of boxes are selected for the merging based on a number of eye-gaze points included in each box included in the subset;
- and
- extract the textual information that corresponds to the determined at least one region.
7 more elements
Dependencies are inferred from the claim text.