# US 2014/0152849 A1 · Video capture of multi-faceted documents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0152849 A1 - Document ID: US20140152849A1 - Status: Published application - Kind: A1 - Application number: 13/705,771 - Filed: 2012-12-05 - Published: 2014-06-05 - Earliest claimed: 2012-12-05 - Assignees: Xerox Corporation - Applicants: Xerox Corporation - Inventors: Raja Bala; Hengzhou Ding; Jayant Kumar - Web page: https://patentgrep.com/patent/US20140152849A1 ## CPC - G06T 3/4092: Image resolution transcoding, e.g. by using client-server architectures - G06V 30/133: {Evaluation of quality of the acquired characters} - H04N 1/3872: {Repositioning or masking} - H04N 23/6811: {based on the image signal} - H04N 23/683: {performed by a processor, e.g. controlling the readout of an image memory} - H04N 23/80: Camera processing pipelines; Components thereof - G06V 30/10: Character recognition - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system captures or otherwise receives a video and uses the video to create an electronic file corresponding to a multi-faceted printed artifact, such as a multi-page document. When the system receives the video, it selects a set of some or all of the video's image frames, determines a frame quality for each frame in the set, and identifies a subset of the frames such that the frame quality of each frame in the subset satisfies one or more image quality criteria. The subset will include at least one frame for each facet of the multi-faceted printed artifact, such as a page of the document. The processor then automatically combines the subset of frames into a single electronic file. ## Claims ### Claim 1 A method of creating an electronic file corresponding to a multi-faceted printed artifact, the method comprising: by a processing device, receiving a video of a multi-faceted printed artifact, wherein the video comprises a series of image frames; by the processing device, selecting a set of the image frames; by the processing device, determining a frame quality for each frame in the set; by the processing device, identifying a subset of the frames so that the frame quality of each frame in the subset satisfies one or more image quality criteria, and the subset comprises at least one frame for each facet of the multi-faceted printed artifact; and by the processing device, automatically combining the subset of frames into a single electronic file. ### Claim 2 (depends on 1) The method of claim 1, wherein receiving the video comprises: by a video capture module of an image capture device, capturing the video while a user moves the artifact, the image capture device, or both so that the video includes images of a plurality of facets of the multi-faceted printed artifact; and wherein the processing device and the video capture module comprise elements of a mobile electronic device. ### Claim 3 (depends on 2) The method of claim 2, wherein determining the frame quality of each frame and identifying the subset comprises at least one of the following: determining whether the frame includes a presence of a user's hand on the artifact, and if so determining that the frame does not satisfy the one or more image quality criteria; determining whether the frame exhibits movement of a user's hand over the artifact, and if so determining that the frame does not satisfy the one or more image quality criteria; determining whether the frame includes each border of the artifact, and determining that the frame satisfies the one or more criteria only if the frame includes each border of the artifact; determining an image quality score for the frame, and determining that the frame satisfies the one or more criteria only if the image quality score satisfies a numeric threshold; and analyzing motion detector device data and time-correlating the motion detector device data to the frame, and determining that frame satisfies the one or more criteria only if the frame does not correspond to a threshold level of motion of the image capture device. ### Claim 4 (depends on 3) The method of claim 3, wherein: determining the frame quality of each frame comprises determining whether the frame includes a presence of a user's hand on the artifact; and determining whether the frame includes the presence of the user's hand on the artifact comprises, by the processor: dividing the frame into a set of N×N sub-blocks of pixels, where N equals an integer N>1, for each sub-block, determining a standard deviation of luminance values of the pixels within each sub-block, for each sub-block, comparing the standard deviation with a standard deviation of a corresponding sub-block in a reference frame for which there is known to be no hand presence to yield a difference, and if the difference is larger than a threshold then determining whether the frame includes the presence of the user's hand on the artifact. ### Claim 5 (depends on 3) The method of claim 3, wherein determining the frame quality of each frame comprises determining whether the frame exhibits movement of a user's hand over the artifact. ### Claim 6 (depends on 3) The method of claim 3, wherein determining the frame quality of each frame comprises determining whether the frame includes each border of the artifact by using a template matching technique and a constant color border identification technique. ### Claim 7 (depends on 3) The method of claim 3, wherein determining the frame quality of each frame comprises determining an image quality score for the frame, and determining the image quality score comprises, by the processor: extracting a set of patches from the frame; determining a feature vector for each of the patches from the frame; pooling the determined feature vectors; and classifying an image quality of the frame based on the feature vectors. ### Claim 8 (depends on 1) The method of claim 1 wherein selecting the set of image frames comprises subsampling the series of image frames to identify a representative set of image frames that will comprise the video to be processed. ### Claim 9 (depends on 1) The method of claim 1 wherein the determining and identifying occur in real time while the receiving occurs, and the method further comprises, before the combining: determining that a first cluster of the received frames corresponds to a first one of the facets; outputting a notification that one of the frames in the first cluster exhibits acceptable frame quality; determining that a second cluster of the received frames corresponds to a second one of the facets; and outputting a notification that one of the frames in the second cluster exhibits acceptable frame quality. ### Claim 10 (depends on 1) The method of claim 1, further comprising, after identifying the subset of frames and before automatically combining the subset of frames into a single electronic file, performing one or more of the following image processing operations on at least one of the frames in the subset: de-skew, de-rotation, perspective correction, automatic cropping, image enhancement, or image binarization. ### Claim 11 A system, comprising: a processor; and a computer-readable memory containing programming instructions that, when executed, are operable to instruct the processor to: receive a video of a multi-faceted printed artifact, wherein the video comprises a series of image frames; select a set of the image frames; determine a frame quality for each frame in the set; identify a subset of the frames so that the frame quality of each frame in the subset satisfies one or more image quality criteria, and the subset comprises at least one frame for each facet of the multi-faceted printed artifact; and automatically combine the subset of frames into a single electronic file. ### Claim 12 (depends on 11) The system of claim 11, further comprising: a video capture module, wherein the processing device, the video capture module, and the computer-readable memory comprise elements of a mobile electronic device; and additional programming instructions that, when executed, are operable to cause the video capture module to capture the video while a user moves the artifact, the image capture device, or both so that the video includes images of a plurality of facets of the multi-faceted artifact. ### Claim 13 (depends on 11) The system of claim 11, wherein the instructions that, when executed, are operable to instruct the processor to determine the frame quality of each frame and identifying the subset comprise instructions to: determine whether the frame includes a presence of a user's hand on the artifact, and if so determining that the frame does not satisfy the one or more image quality criteria; determine whether the frame exhibits movement of a user's hand over the artifact, and if so determine that the frame does not satisfy the one or more image quality criteria; determine whether the frame includes each border of the artifact, and determine that the frame satisfies the one or more criteria only if the frame includes each border of the artifact; determine an image quality score for the frame, and determine that the frame satisfies the one or more criteria only if the image quality score satisfies a numeric threshold; and analyze motion detector device data and time-correlate the motion detector device data to the frame, and determine that frame satisfies the one or more criteria only of the frame does not correspond to a threshold level of motion of the image capture device. ### Claim 14 (depends on 13) The system of claim 13, wherein: the instructions to determine the frame quality of each frame comprise instructions to determine whether the frame includes a presence of a user's hand on the artifact; and the instructions to determine whether the frame includes the presence of the user's hand on the artifact comprise instructions to: divide the frame into a set of N×N sub-blocks of pixels, where N equals an integer N>1, for each sub-block, determine a standard deviation of luminance values of the pixels within each sub-block, for each sub-block, compare the standard deviation with a standard deviation of a corresponding sub-block in a reference frame for which there is known to be no hand presence to yield a difference, and if the difference is larger than a threshold then determine whether the frame includes the presence of the user's hand on the artifact. ### Claim 15 (depends on 13) The system of claim 13, wherein the instructions to determine the frame quality of each frame comprise instructions to determine whether the frame exhibits movement of a user's hand over the artifact. ### Claim 16 (depends on 13) The system of claim 13, wherein the instructions to determine the frame quality of each frame comprise instructions to determine whether the frame includes each border of the artifact by using a template matching technique and a constant color border identification technique. ### Claim 17 (depends on 13) The system of claim 13, wherein the instructions to determine the frame quality of each frame comprise instructions to determining an image quality score for the frame by: extracting a set of patches from the frame; determining a feature vector for each of the patches from the frame; pooling the determined feature vectors; and classifying an image quality of the frame based on the feature vectors. ### Claim 18 (depends on 11) The system of claim 11, wherein the instructions to select the set of image frames comprise instructions to subsample the series of image frames to identify a representative set of image frames that will comprise the video to be processed. ### Claim 19 (depends on 11) The system of claim 11 wherein the instructions, when executed, are also operable to instruct the processor to perform the following before automatically combining the subset of frames into a single electronic file: determining that a first cluster of the received frames corresponds to a first one of the facets; outputting a notification that one of the frames in the first cluster exhibits acceptable frame quality; determining that a second cluster of the received frames corresponds to a second one of the facets; and outputting a notification that one of the frames in the second cluster exhibits acceptable frame quality. ### Claim 20 A method of creating an electronic document file for a multi-page printed document, the method comprising: by a processing device, receiving a video of a multi-page printed document, wherein the video comprises a series of image frames; by the processing device, selecting a set of the image frames; by the processing device, determining a frame quality for each frame in the set; by the processing device, identifying a subset of the frames so that the frame quality of each frame in the subset satisfies one or more image quality criteria, and the subset comprises at least one frame for each page of the multi-page printed document; and by the processing device, automatically combining the subset of frames into a single electronic file so that each frame of the electronic file represents a single page of the multi-page printed document. ### Claim 21 (depends on 20) The method of claim 20, wherein receiving the video comprises: by a video capture module of an image capture device, capturing the video while a user moves the document, the image capture device, or both so that the video includes images of a plurality of pages of the multi-page printed document; and wherein the processing device and the video capture module comprise elements of a mobile electronic device. ### Claim 22 (depends on 21) The method of claim 21, wherein selecting the set of image frames comprises subsampling the video to reduce the number of image frames and yield of a representative image frame set. ### Claim 23 (depends on 22) The method of claim 22, wherein determining the frame quality of each frame and identifying the subset comprises at least one of the following: determining whether the frame includes a presence of a user's hand on the document, and if so determining that the frame does not satisfy the one or more image quality criteria; determining whether the frame exhibits movement of a user's hand over the document, and if so determining that the frame does not satisfy the one or more image quality criteria; determining whether the frame includes each border of the document, and determining that the frame satisfies the one or more criteria only if the frame includes each border of the document; determining an image quality score for the frame, and determining that the frame satisfies the one or more criteria only if the image quality score satisfies a numeric threshold; and analyzing motion detector device data and time-correlating the motion detector device data to the frame, and determining that frame satisfies the one or more criteria only if the frame does not correspond to a threshold level of motion of the image capture device.