# US 11,307,875 B2 · Method and system for accessing table content in a digital image of the table > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,307,875 B2 - Document ID: US11307875B2 - Status: Granted - Kind: B2 - Application number: 16/899,930 - Filed: 2020-06-12 - Published: 2020-10-01 - Granted: 2022-04-19 - Earliest claimed: 2017-01-26 - Assignees: Nice Ltd. - Applicants: Nice Ltd. - Inventors: Alexander Vaindiner; Vitaly Shelest; Semyon Kleinerman - Web page: https://patentgrep.com/patent/US11307875B2 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables - G06F 8/10: Requirements analysis; Specification techniques - G06F 8/38: for implementing user interfaces - G06T 5/20: using local operators - G06T 5/73: Deblurring; Sharpening - G06T 7/13: Edge detection - G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and method for accessing elements of a table in a digital image of the table, including: obtaining the digital image of the table; finding table elements in the digital image based on digital table properties, wherein the table elements define table cells; calculating coordinates of the table cells in the digital image based on the table elements; and accessing content of a selected table cell in the digital image using the coordinates of the selected table element. ## Claims ### Claim 1 A computer implemented method for accessing elements of a table in a digital image-of the table, the method comprising: obtaining the digital image of the table; finding table elements in the digital image based on digital table properties by processing the digital image using imaging processing techniques, wherein the table elements define table cells; calculating coordinates of the table cells in the digital image based on the table elements; and accessing content of a selected table cell in the digital image using the coordinates of the selected table element. ### Claim 2 (depends on 1) The method of claim 1, wherein the table elements comprises at least one of gridlines and table text blocks. ### Claim 3 (depends on 1) The method of claim 1, wherein the digital table properties comprise at least one of: has row separators, has column separators, has row header, has row header separators, has column header, has column header separators, table direction, single line, selection mode and a fragmentation method. ### Claim 4 (depends on 3) The method of claim 3, comprising obtaining at least one of the digital table properties in a design phase prior to runtime. ### Claim 5 (depends on 1) The method of claim 1, wherein finding table elements in the digital image is performed using a fragmentation method that is selected from the list consisting of: an orthogonality filter method, a gridlines detection method, a morphology detection method, a table skeleton method a cartoonizing method and Fast line detector method. ### Claim 6 (depends on 5) The method of claim 5, wherein the orthogonality filter method comprises: converting the digital image to a grayscale image; applying a horizontal discrete differentiation operator to the grayscale image to calculate a horizontal derivative and vertical discrete differentiation operator to calculate a vertical derivative for pixels of the grayscale image, wherein the horizontal derivative and the vertical derivative of a pixel form a two-dimensional (2D) vector of the pixel; calculating magnitudes and angles of the 2D vectors of the pixels of the grayscale image; marking a pixel (i,j) as pertaining to a gridline if the pixel (i,j) and its (i+1,j), (i,j+1), (i+1,j+1) neighbors are orthogonal, wherein i and j are pixel indexes; and extracting coordinates of vertical and horizontal gridlines. ### Claim 7 (depends on 6) The method of claim 6, wherein the cartoonizing method comprises: downsampling to the digital image to obtain a down sampled image; performing edge detection on the down sampled image to obtain a processed image; applying a bilateral filter to the processed image to obtain a filtered image; upsampling the filtered image to obtain a zoomed-out image; smoothing the zoomed-out image by applying a median filter with n×n aperture, wherein n is an odd number, to obtain a of smoothed image; converting the smoothed image to a smoothed grayscale image; converting the smoothed grayscale image to a binary image with adaptive threshold; resizing the binary image to the image size of the digital image to obtain a resized binary image; and applying the orthogonality filter method to the resized binary image. ### Claim 8 (depends on 5) The method of claim 5, wherein the gridlines detection method comprises: building grid mask by: converting digital image of the table to a grayscale image; blurring the grayscale image by applying a gaussian blur function to obtain a blurred image; thresholding the blurred image to obtain a binary image; scanning the binary image to obtain a pixel that is connected to a maximal number of pixels with a given color of the binary image; and filling the connected pixels with a first color and the rest of the pixels with a second color to generate a mask; and extracting coordinates of horizontal and vertical gridlines from the mask. ### Claim 9 (depends on 5) The method of claim 5, wherein the morphology detection method comprises: thresholding the digital image to obtain a binary image; blurring the binary image to obtain a blurred image; performing edge detection on the blurred image to obtain a processed image; obtaining a structuring element; performing morphological image processing on the processed image using the structuring element, to find geometry shapes; identifying contours of the geometry shapes using a contour approximation method, wherein the contours represent text blocks in the table image; and generating vertical and horizontal lines between the table blocks, wherein the vertical and horizontal lines form the gridlines of the digital image. ### Claim 10 (depends on 5) The method of claim 5, wherein the table skeleton method comprises: applying a Laplacian filter to the digital image to obtain a filtered image; normalizing the filtered image to obtain a color image; thresholding the color image to obtain a binary image; blurring the binary image to obtain a blurred image; calculating contours of a first color in the blurred image, wherein the contours represent text blocks in the digital image; and generating vertical and horizontal lines between the table blocks, wherein the vertical and horizontal lines form the gridlines of the digital image. ### Claim 11 (depends on 1) The method of claim 1, wherein accessing a selected table cell in the digital image comprises performing at least one of: get rows count, get columns count, get cell text, set cell text, click cell, get row cells, get column, get selected row index, get selected column index, get cell image, paste cell, copy cell, get header cell, header click and generate an event if table has changed. ### Claim 12 (depends on 1) The method of claim 1, comprising presenting the digital image on a client device and wherein the computer is remote from the client device. ### Claim 13 A system for accessing elements of a table in a digital image-of the table, the system comprising: a memory; and a processor configured to: obtain the digital image of the table; find gridlines in the digital image based on digital table properties by processing the digital image using imaging processing techniques, wherein gridlines define table cells; calculate coordinates of the table cells in the digital image based on the table elements; and access content of a selected table cell in the digital image using the coordinates of the selected table element. ### Claim 14 (depends on 13) The system of claim 13, wherein the digital table properties comprise at least one of: has row separators, has column separators, has row header, has row header separators, has column header, has column header separators, table direction, single line, selection mode and a fragmentation method. ### Claim 15 (depends on 13) The system of claim 13, wherein finding table elements in the digital image is performed using a fragmentation method that is selected from the list consisting of: an orthogonality filter method, a gridlines detection method, a morphology detection method, a table skeleton method a cartoonizing method and Fast line detector method. ### Claim 16 (depends on 15) The system of claim 15, wherein the orthogonality filter method comprises: converting the digital image to a grayscale image; applying a horizontal discrete differentiation operator to the grayscale image to calculate a horizontal derivative and vertical discrete differentiation operator to calculate a vertical derivative for pixels of the grayscale image, wherein the horizontal derivative and the vertical derivative of a pixel form a two-dimensional (2D) vector of the pixel; calculating magnitudes and angles of the 2D vectors of the pixels of the grayscale image; marking a pixel (i,j) as pertaining to a gridline if the pixel (i,j) and its (i+1,j), (i,j+1), (i+1,j+1) neighbors are orthogonal, wherein i and j are pixel indexes; and extracting coordinates of vertical and horizontal gridlines. ### Claim 17 (depends on 16) The system of claim 16, wherein the cartoonizing method comprises: downsampling to the digital image to obtain a down sampled image; performing edge detection on the down sampled image to obtain a processed image; applying a bilateral filter to the processed image to obtain a filtered image; upsampling the filtered image to obtain a zoomed-out image; smoothing the zoomed-out image by applying a median filter with n×n aperture, wherein n is an odd number, to obtain a of smoothed image; converting the smoothed image to a smoothed grayscale image; converting the smoothed grayscale image to a binary image with adaptive threshold; resizing the binary image to the image size of the digital image to obtain a resized binary image; and applying the orthogonality filter method to the resized binary image. ### Claim 18 (depends on 15) The system of claim 15, wherein the gridlines detection method comprises: building grid mask by: converting digital image of the table to a grayscale image; blurring the grayscale image by applying a gaussian blur function to obtain a blurred image; thresholding the blurred image to obtain a binary image; scanning the binary image to obtain a pixel that is connected to a maximal number of pixels with a given color of the binary image; and filling the connected pixels with a first color and the rest of the pixels with a second color to generate a mask; and extracting coordinates of horizontal and vertical gridlines from the mask. ### Claim 19 (depends on 15) The system of claim 15, wherein the morphology detection method comprises: thresholding the digital image to obtain a binary image; blurring the binary image to obtain a blurred image; performing edge detection on the blurred image to obtain a processed image; obtaining a structuring element; performing morphological image processing on the processed image using the structuring element, to find geometry shapes; identifying contours of the geometry shapes using a contour approximation method, wherein the contours represent text blocks in the table image; and generating vertical and horizontal lines between the table blocks, wherein the vertical and horizontal lines form the gridlines of the digital image. ### Claim 20 (depends on 15) The system of claim 15, wherein the table skeleton method comprises: applying a Laplacian filter to the digital image to obtain a filtered image; normalizing the filtered image to obtain a color image; thresholding the color image to obtain a binary image; blurring the binary image to obtain a blurred image; calculating contours of a first color in the blurred image, wherein the contours represent text blocks in the digital image; and generating vertical and horizontal lines between the table blocks, wherein the vertical and horizontal lines form the gridlines of the digital image.