# US 12,112,563 B2 · Method of detecting, segmenting and extracting salient regions in documents using attention tracking sensors > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,563 B2 - Document ID: US12112563B2 - Status: Granted - Kind: B2 - Application number: 17/444,721 - Filed: 2021-08-09 - Published: 2023-02-09 - Granted: 2024-10-08 - Earliest claimed: 2021-08-09 - Assignees: JPMorgan Chase Bank, N.A. - Applicants: JPMorgan Chase Bank, N.A. - Inventors: Naftali Y Cohen; Sameena Shah; Natraj Raman; Zhen Zeng; Salwa Husam Alamir; Daniel Borrajo; Alec Louis Clemente Candidato - Web page: https://patentgrep.com/patent/US12112563B2 ## CPC - G06V 30/413: Classification of content, e.g. text, photographs or tables - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 16/345: {Summarisation for human users} - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 30/18143: {Extracting features based on salient regional features, e.g. scale invariant feature transform [SIFT] keypoints} - G06V 30/41: Analysis of document content (recognition of printed characters based on code marks G06V30/224) - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text - G06V 40/18: Eye characteristics, e.g. of the iris - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method and system for detecting, segmenting, and extracting salient regions in documents by using attention tracking sensors is provided. The method includes: receiving an image that corresponds to a document; receiving, from a sensor, a sequence of measurements that correspond to a human reading of the document; determining, based on the sequence of measurements, at least one region of the document as being a salient document region; demarcating the salient document region in an electronically displayable manner; and outputting a file that includes a displayable version of the document with the demarcated document region. The salient document region may include a title, a section header, and/or a table. The sensor may be an eye-tracking sensor that detects a sequence of eye-gaze positions on the document as a function of time. ## Claims ### Claim 1 A method for detecting, segmenting, and extracting salient regions in documents by using attention tracking sensors, the method being implemented by at least one processor, the method comprising: receiving, by the at least one processor, an image that corresponds to a document; receiving, by the at least one processor from a sensor, a sequence of measurements that correspond to a human reading of the document, wherein the sensor includes an eye-tracking sensor configured to detect a sequence of eye-gaze positions on the document as a function of time; determining, by the at least one processor based on the received sequence of measurements, an identification of at least one of a title, a section header, a graph, and a table, included in the document; outputting, by the at least one processor, the identification of the at least one of the title, the section header, the graph, and the table; determining, by the at least one processor based on the received sequence of measurements, at least one region of the document as being a salient document region; demarcating, by the at least one processor, the salient document region in an electronically displayable manner; and outputting, by the at least one processor, a file that includes a displayable version of the document with the demarcated salient document region. ### Claim 2 (depends on 1) The method of claim 1, wherein the salient document region includes at least one from among the title, the section header, the table, and the graph. ### Claim 3 (depends on 1) The method of claim 1, wherein the demarcating comprises drawing a box that surrounds the salient document region. ### Claim 4 (depends on 1) The method of claim 1, wherein the image that corresponds to the document comprises at least one from among a physical page that corresponds to the document and a digital file that corresponds to the document. ### Claim 5 (depends on 1) The method of claim 1, wherein the human reading of the document comprises the human focusing on a predetermined aspect of the document that corresponds to the salient region of the document. ### Claim 6 (depends on 1) The method of claim 1, wherein the file that includes the displayable version of the document with the demarcated document region comprises at least one from among an image file and a text file. ### Claim 7 (depends on 1) The method of claim 1, further comprising determining, by the at least one processor based on the received sequence of measurements, a text summarization of the document. ### Claim 8 (depends on 1) The method of claim 1, further comprising determining, by the at least one processor based on the received sequence of measurements, an identification of at least one key phrase included in the document. ### Claim 9 A computing apparatus for detecting, segmenting, and extracting salient regions in documents by using attention tracking sensors, the computing apparatus comprising: a processor; a memory; and a communication interface coupled to each of the processor and the memory, wherein the processor is configured to: receive, via the communication interface, an image that corresponds to a document; receive, from a sensor via the communication interface, a sequence of measurements that correspond to a human reading of the document, wherein the sensor includes an eye-tracking sensor configured to detect a sequence of eye-gaze positions on the document as a function of time; determine, based on the received sequence of measurements, an identification of at least one of a title, a section header, a graph, and a table, included in the document; output the identification of the at least one of the title, the section header, the graph, and the table; determine, based on the received sequence of measurements, at least one region of the document as being a salient document region; demarcate the salient document region in an electronically displayable manner; and output a file that includes a displayable version of the document with the demarcated salient document region. ### Claim 10 (depends on 9) The computing apparatus of claim 9, wherein the salient document region includes at least one from among the title, the section header, the table, and the graph. ### Claim 11 (depends on 9) The computing apparatus of claim 9, wherein the processor is further configured to perform the demarcating by drawing a box that surrounds the salient document region. ### Claim 12 (depends on 9) The computing apparatus of claim 9, wherein the image that corresponds to the document comprises at least one from among a physical page that corresponds to the document and a digital file that corresponds to the document. ### Claim 13 (depends on 9) The computing apparatus of claim 9, wherein the human reading of the document comprises the human focusing on a predetermined aspect of the document that corresponds to the salient region of the document. ### Claim 14 (depends on 9) The computing apparatus of claim 9, wherein the file that includes the displayable version of the document with the demarcated document region comprises at least one from among an image file and a text file. ### Claim 15 (depends on 9) The computing apparatus of claim 9, wherein the processor is further configured to determine, based on the received sequence of measurements, a text summarization of the document. ### Claim 16 (depends on 9) The computing apparatus of claim 9, wherein the processor is further configured to determine, based on the received sequence of measurements, an identification of at least one key phrase included in the document. ### Claim 17 A non-transitory computer readable storage medium storing instructions for detecting, segmenting, and extracting salient regions in documents by using attention tracking sensors, the storage medium comprising executable code which, when executed by a processor, causes the processor to: receive an image that corresponds to a document; receive, from a sensor, a sequence of measurements that correspond to a human reading of the document, wherein the sensor includes an eye-tracking sensor configured to detect a sequence of eye-gaze positions on the document as a function of time; determine, based on the received sequence of measurements, an identification of at least one of a title, a section header, a graph, and a table, included in the document; output the identification of the at least one of the title, the section header, the graph, and the table; determine, based on the received sequence of measurements, at least one region of the document as being a salient document region; demarcate the salient document region in an electronically displayable manner; and output a file that includes a displayable version of the document with the demarcated salient document region. ### Claim 18 (depends on 17) The storage medium of claim 17, wherein the salient document region includes at least one from among the title, the section header, the table, and the graph.