# US 12,217,525 B1 · Multi-modal ensemble deep learning for start page classification of document image file including multiple different documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,525 B1 - Document ID: US12217525B1 - Status: Granted - Kind: B1 - Application number: 18/608,746 - Filed: 2024-03-18 - Granted: 2025-02-04 - Earliest claimed: 2023-04-18 - Assignees: First American Financial Corporation - Applicants: First American Financial Corporation - Inventors: Arun Rangarajan; Dan Thompson; Madhu Kolli; Ritaprava Dutta; Zheqi Tan - Web page: https://patentgrep.com/patent/US12217525B1 ## CPC - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 10/82: using neural networks - G06V 30/18057: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 30/19173: {Classification techniques} - G06V 30/1918: {Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion} - G06V 30/245: {Font recognition} - G06V 30/413: Classification of content, e.g. text, photographs or tables - G06V 30/422: Technical drawings; Geographical maps - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Multimodal techniques are described for classifying start pages and/or document types of an unstructured document image package. To that end, some implementations of the disclosure relate to a method, including: obtaining a document image file including multiple pages and multiple document types; generating, for each page of the multiple pages, using multiple independent trained models, multiple independent predictions, each of the multiple independent predictions indicating: whether or not the page is a first page of a document, or a document type of the multiple document types that the page corresponds to; and generating, for each page of the multiple pages, based on the multiple independent predictions, using a neural network, a final prediction output indicating whether or not the page is the first page of a document, or one of the multiple document types that the page corresponds to. ## Claims ### Claim 1 A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising: obtaining a document image file including multiple pages corresponding to multiple documents; generating, for each page of the multiple pages, using multiple independent trained models, multiple independent predictions, each of the multiple independent predictions indicating whether or not the page is a first page of one of the documents; generating, for each page of the multiple pages, based on the multiple independent predictions, using an additional trained model, a final prediction output indicating whether or not the page is the first page of one of the documents, wherein the multiple independent trained models comprise two or more models selected from the group consisting of: a first trained model configured to predict, for each page, based on first image features of a multi-page image associated with the page, whether or not the page is the first page of one of the documents; a second trained model configured to predict, for each page, based on second image features of the page, whether or not the page is the first page of one of the documents; a third trained model configured to predict, for each page, based on font feature changes between the page and an adjacent page, whether or not the page is the first page of one of the documents; a fourth trained model configured to predict, for each page, based on margin feature changes between the page and the adjacent page, whether or not the page is the first page of one of the documents; and a fifth trained model configured to predict, for each page, based on textual features obtained by performing optical character recognition on the page, whether or not the page is the first page of one of the documents. ### Claim 2 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the multiple independent trained models comprise the first trained model. ### Claim 3 (depends on 2) The non-transitory computer-readable medium of claim 2, wherein the operations further comprise obtaining multiple multi-page images by combining every two consecutive pages of the multiple pages, the multiple multi-page images including the multi-page image associated with each page. ### Claim 4 (depends on 3) The non-transitory computer-readable medium of claim 3, wherein obtaining the multiple multi-page images further comprises: forming one of the multi-page images by combining a blank page with one of the multiple pages. ### Claim 5 (depends on 3) The non-transitory computer-readable medium of claim 3, wherein combining every two consecutive pages of the multiple pages comprises concatenating every two consecutive pages of the multiple pages. ### Claim 6 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the operations further comprise generating the first trained model by: obtaining multiple page images associated with multiple document image files; obtaining multiple multi-page images by combining every two consecutive page images of the multiple page images; tagging the multiple multi-page images with multiple labels, each of the labels indicating whether a respective one of multi-page images corresponds to a first page; and training, based on the multiple multi-page images and the multiple labels, an image classification model as the first trained model. ### Claim 7 (depends on 3) The non-transitory computer-readable medium of claim 3, wherein: generating the first trained model further comprises: adding a blank page image to the multiple page images; and obtaining the multiple multi-page images comprises combining the blank page image with one of the multiple page images to form one of the multi-page images. ### Claim 8 (depends on 2) The non-transitory computer-readable medium of claim 2, wherein the multiple independent trained models comprise the third trained model. ### Claim 9 (depends on 2) The non-transitory computer-readable medium of claim 2, wherein the multiple independent trained models comprise the fourth trained model. ### Claim 10 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the multiple documents are associated with a real property, the multiple documents comprising: a map, a search result, or a recorded document. ### Claim 11 (depends on 1) The non-transitory computer-readable medium of claim 1, wherein the operations further comprise: splitting, based on the final prediction output generated for each page, the document image file into the multiple documents. ### Claim 12 A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising: obtaining a document image file including multiple pages corresponding to multiple documents; obtaining, for each page of the multiple pages, using multiple dependent models of a neural network, multiple feature embeddings, each of the feature embeddings corresponding to at least features of the page that are indicative of whether or not the page is a first page of one of the documents; combining, for each page of the multiple pages, the feature embeddings into combined feature embeddings; and generating, for each page of the multiple pages, based on the combined feature embeddings, using a neural network, a prediction output indicating whether or not the page is the first page of one of the documents, wherein the multiple feature embeddings of each comprise two or more feature embeddings selected from the group consisting of: first image features of a multi-page image including a combination of the page and an adjacent page; second image features of the page; textual features of the page; font features of the page and the adjacent page; and margin features of the page and the adjacent page, wherein a first feature embedding of the multiple feature embeddings comprises the first image features, and wherein the operations further comprise obtaining the first image features of each page by concatenating the page with the adjacent page to form the multi-page image. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein a second feature embedding of the multiple feature embeddings comprises an output of a last hidden layer of one of the multiple dependent models. ### Claim 14 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein a second feature embedding of the multiple feature embeddings comprises the font features of the page and the adjacent page. ### Claim 15 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein a second feature embedding of the multiple feature embeddings comprises the margin features of the page and the adjacent page. ### Claim 16 A non-transitory computer-readable medium having executable instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising: obtaining a document image file including multiple pages corresponding to multiple documents; obtaining first multiple page images corresponding to the multiple pages; adding a blank page image to the first multiple page images to obtain second multiple page images; obtaining multiple multi-page images by combining every two consecutive page images of the second multiple page images into a multi-page image associated with a page of the multiple pages; and generating, for each page of the multiple pages, based at least on the multi-page image associated with the page, using a trained image classification model, a first prediction indicating whether or not the page is a first page of one of the documents. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: splitting, based on the first prediction generated for each page, the document image file into the multiple documents. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: generating, for each page of the multiple pages, using a second trained model, a second prediction indicating whether or not the page is the first page of one of the documents; and generating, for each page of the multiple pages, based at least on the first prediction and the second prediction, using a third trained model, a final prediction output indicating whether or not the page is the first page of one of the documents.