# US 2017/0286402 A1 · Content Availability for Natural Language Processing Tasks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2017/0286402 A1 - Document ID: US20170286402A1 - Status: Published application - Kind: A1 - Application number: 15/627,044 - Filed: 2017-06-19 - Published: 2017-10-05 - Earliest claimed: 2013-12-13 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Jeffrey N. Eisen - Web page: https://patentgrep.com/patent/US20170286402A1 ## CPC - G06F 40/30: Semantic analysis - G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191) - G06F 40/151: Transformation - G06F 40/166: Editing, e.g. inserting or deleting - G06F 40/169: Annotation, e.g. comment data or footnotes - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An approach is provided to make content available to natural language processing (NLP) tasks. In the approach, a screen view of a document section is provided as input to a screen reader application. The screen reader application converts information displayed on the screen into a natural language format. A NLP operation is then performed on the natural language format. ## Claims ### Claim 1 A method, in an information handling system comprising a processor and a memory, of making content available to natural language processing (NLP) tasks, the method comprising: in response to determining that a document section comprises non-textual image data, converting the document section into a natural language format, wherein the natural language format comprises textual data compatible with the NLP tasks, and wherein the converting comprises: providing a screen view of the document section to a screen reader application; in response to the providing, receiving, from the screen reader application, an audible speech output; and providing the audible speech output to a speech recognition application, wherein the speech recognition application converts the audible speech input into a natural language textual output that describes the non-textual image data; and performing a NLP operation on the natural language textual output. ### Claim 2 (depends on 1) The method of claim 1 wherein the NLP operation is carried out by a question and answer system. ### Claim 3 (depends on 1) The method of claim 1 further comprising: prior to the converting: receiving a document that includes the document section; and identifying that the document section is incompatible with one or more native converters used to convert document contents to the NLP format. ### Claim 4 (depends on 1) The method of claim 1 further comprising: receiving an input document that includes a plurality of document sections including the document section comprising non-textual image data; identifying a first set of document sections from the plurality of document sections that are compatible with one or more native converters used to convert document contents to the natural language format; converting the first set of document sections to the natural language format using the one or more native converters; identifying a second set of document sections from the plurality of document sections that are incompatible with the one or more native converters, wherein the second set of document sections includes the document section comprising non-textual image data; and converting the second set of document sections to the natural language format by providing one or more screen views corresponding to each of the second set of document sections to the screen reader application, receiving one or more audible speech outputs from the screen reader application, providing the one or more audible speech outputs to the speech recognition application, and receiving one or more natural language textual outputs that describe the non-textual image data. ### Claim 5 (depends on 4) The method of claim 4 wherein the identifying the second set of document sections further comprises: detecting a content type corresponding to one or more document sections included in the second set of document sections, wherein the detected content type is selected from a group consisting of a diagram, a table, a picture, and a non-textual document type. ### Claim 6 (depends on 4) The method of claim 4 further comprising: merging the converted first set of document sections and the converted second set of document sections to a natural language output file, wherein a merge order of the first and second set of converted document sections in the natural language output file is consistent with an original order of the plurality of document sections found in the input document. ### Claim 7 An information handling system comprising: one or more processors; a memory coupled to at least one of the processors; a display; and a set of instructions stored in the memory and executed by at least one of the processors to make content available to natural language processing (NLP) tasks, wherein the set of instructions perform actions of: in response to determining that a document section comprises non-textual image data, converting the document section into a natural language format, wherein the natural language format comprises textual data compatible with the NLP tasks, and wherein the converting comprises: providing a screen view of the document section to a screen reader application; in response to the providing, receiving, from the screen reader application, an audible speech output; and providing the audible speech output to a speech recognition application, wherein the speech recognition application converts the audible speech input into a natural language textual output that describes the non-textual image data; and performing a NLP operation on the natural language textual output. ### Claim 8 (depends on 7) The information handling system of claim 7 wherein the NLP operation is carried out by a question and answer system. ### Claim 9 (depends on 7) The information handling system of claim 7 wherein the actions further comprise: prior to the converting: receiving a document that includes the document section; identifying that the document section is incompatible with one or more native converters used to convert document contents to the NLP format. ### Claim 10 (depends on 7) The information handling system of claim 7 wherein the actions further comprise: receiving an input document that includes a plurality of document sections including the document section comprising non-textual image data; identifying a first set of document sections from the plurality of document sections that are compatible with one or more native converters used to convert document contents to the natural language format; converting the first set of document sections to the natural language format using the one or more native converters; identifying a second set of document sections from the plurality of document sections that are incompatible with the one or more native converters, wherein the second set of document sections includes the document section comprising non-textual image data; and converting the second set of document sections to the natural language format by providing one or more screen views corresponding to each of the second set of document sections to the screen reader application, receiving one or more audible speech outputs from the screen reader application, providing the one or more audible speech outputs to the speech recognition application, and receiving one or more natural language textual outputs that describe the non-textual image data. ### Claim 11 (depends on 10) The information handling system of claim 10 wherein the identifying the second set of document sections further comprises: detecting a content type corresponding to one or more document sections included in the second set of document sections, wherein the detected content type is selected from a group consisting of a diagram, a table, a picture, and a non-textual document type. ### Claim 12 (depends on 10) The information handling system of claim 10 wherein the actions further comprise: merging the converted first set of document sections and the converted second set of document sections to a natural language output file, wherein a merge order of the first and second set of converted document sections in the natural language output file is consistent with an original order of the plurality of document sections found in the input document. ### Claim 13 A computer program product stored in a computer readable storage medium, comprising computer instructions that, when executed by an information handling system, causes the information handling system to make content available to natural language processing (NLP) tasks by performing actions comprising: in response to determining that a document section comprises non-textual image data, converting the document section into a natural language format, wherein the natural language format comprises textual data compatible with the NLP tasks, and wherein the converting comprises: providing a screen view of the document section to a screen reader application; in response to the providing, receiving, from the screen reader Application, an audible speech output; and providing the audible speech output to a speech recognition application, wherein the speech recognition application converts the audible speech input into a natural language textual output that describes the non-textual image data; and performing a NLP operation on the natural language textual output. ### Claim 14 (depends on 13) The computer program product of claim 13 wherein the NLP operation is carried out by a question and answer system. ### Claim 15 (depends on 13) The computer program product of claim 13 wherein the actions further comprise: prior to the converting: receiving a document that includes the document section; identifying that the document section is incompatible with one or more native converters used to convert document contents to the NLP format. ### Claim 16 (depends on 13) The computer program product of claim 13 wherein the actions further comprise: receiving an input document that includes a plurality of document sections including the document section comprising non-textual image data; identifying a first set of document sections from the plurality of document sections that are compatible with one or more native converters used to convert document contents to the natural language format; converting the first set of document sections to the natural language format using the one or more native converters; identifying a second set of document sections from the plurality of document sections that are incompatible with the one or more native converters, wherein the second set of document sections includes the document section comprising non-textual image data; and converting the second set of document sections to the natural language format by providing one or more screen views corresponding to each of the second set of document sections to the screen reader application, receiving one or more audible speech outputs from the screen reader application, providing the one or more audible speech outputs to the speech recognition application, and receiving one or more natural language textual outputs that describe the non-textual image data. ### Claim 17 (depends on 16) The computer program product of claim 16 wherein the identifying the second set of document sections further comprises: detecting a content type corresponding to one or more document sections included in the second set of document sections, wherein the detected content type is selected from a group consisting of a diagram, a table, a picture, and a non-textual document type. ### Claim 18 (depends on 16) The computer program product of claim 16 wherein the actions further comprise: merging the converted first set of document sections and the converted second set of document sections to a natural language output file, wherein a merge order of the first and second set of converted document sections in the natural language output file is consistent with an original order of the plurality of document sections found in the input document.