# US 2019/0171876 A1 · System and Method for Detecting Objects in an Image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171876 A1
- Document ID: US20190171876A1
- Status: Published application
- Kind: A1
- Application number: 16/258,707
- Filed: 2019-01-28
- Published: 2019-06-06
- Earliest claimed: 2015-06-22
- Assignees: Photomyne Ltd.
- Applicants: Photomyne Ltd.
- Inventors: Yair SEGALOVITZ; Omer SHOOR; Yaron LIPMAN; Nir TZEMAH; Natalie VERTER
- Web page: https://patentgrep.com/patent/US20190171876A1

## CPC

- G06V 30/40: Document-oriented image-based pattern recognition
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 5/92: based on global image properties
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 30/142: using hand-held instruments; Constructional details of the instruments
- G06V 30/413: Classification of content, e.g. text, photographs or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/10024: Color image
- G06T 2207/20132: Image cropping
- G06T 2207/20164: Salient point detection; Corner detection
- G06T 2207/30176: Document
- G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for cropping photos images captured by a user from an image of a page of a photo album is described. Corners in the page image are detected using corner detection algorithm or by detecting intersections of line-segments (and their extensions) in the image using edge, corner, or line detection techniques. Pairs of the detected corners are used to define all potential quads, which are then are qualified according to various criteria. A correlation matrix is generated for each potential pair of the qualified quads, and candidate quads are selected based on the Eigenvector of the correlation matrix. The content of the selected quads is checked using a salience map that may be based on a trained neuron network, and the resulting photos images are extracted as individual files for further handling or manipulation by the user.

## Claims

### Claim 1

A method for detecting, by a portable or a hand-held device, one or more photographs from a background in an image captured by a digital camera, the method by the device comprising:
analyzing the captured image using a neural network for detecting the photographs;
cropping or extracting from the captured image each of the detected photographs into a respective file; and
transmitting one or more of the files over a Wireless Local Area Network (WLAN) using a WLAN transmitter, wherein the neural network is further trained to recognize or classify the photographs in the captured image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the neural network is further trained to recognize or classify the photographs in the captured image and having multiple stages or layers.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the analyzing of the content of each of the photographs uses an output of an intermediate stage or layer in the neural network.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the neural network is ImageNet having 26 stages or layers, the intermediate stage or layer is an eighth stage or layer, the output includes 256 saliency maps.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the analyzing of the content comprises generating an output map calculated by a weighted average of the 256 saliency maps.

### Claim 6 (depends on 1)

The method according to claim 1, further for use with one or more rectangular-shaped object regions in an image captured by a digital camera, the method by the device further comprising:
analyzing the captured image using a deep convolutional neural network for detecting the rectangular-shaped object regions;
cropping or extracting from the captured image each of the detected regions into a respective file; and
transmitting one or more of the files over a Wireless Local Area Network (WLAN) using a WLAN transmitter, wherein the neural network is further trained to recognize or classify the rectangular-shaped object regions in the captured image.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the objects are receipts, business cards, sticky notes, printed newspapers, or stamps.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the background comprises a pattern and the captured image comprises an entire or part of a page of a photo album.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the captured image is a single image file that is in a compressed format that is according to, based on, or consists of, Portable Network Graphics (PNG), Graphics Interchange Format (GIF), Joint Photographic Experts Group (JPEG), Exchangeable image file format (Exif), Tagged Image File Format (TIFF).

### Claim 10 (depends on 1)

The method according to claim 1, wherein the captured image is using a color image that is using a color model that is according to, or based on, the RGB color space.

### Claim 11 (depends on 1)

The method according to claim 1, further comprising capturing a first image by the digital camera, and forming the captured image by converting the first image from the color model to a grayscale format, wherein the converting of the captured image to a grayscale format is according to, based on, or using, linearly encoding grayscale intensity from linear RGB.

### Claim 12 (depends on 1)

The method according to claim 1, further comprising capturing a first image by the digital camera, and downscaling the first image to form the captured image by downscaling the first image using a downscaling algorithm, wherein the captured image is having less than 10%, 7%, 5%, 3%, or 1% pixels of the first image or wherein the captured image is having less than 10,000, 5,000, 2,000, or 1,000 pixels.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the downscaling algorithm is according to, is based on, or uses, an adaptive or non-adaptive image interpolation algorithm.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the non-adaptive image interpolation algorithm consists of, comprises, or is part of, nearest-neighbor replacement, bilinear interpolation, bi-cubic interpolation, Spline interpolation, Lanczos interpolation, or a digital filtering technique.

### Claim 15 (depends on 1)

The method according to claim 1, further comprising enhancing the image in each of the detected photographs.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the enhancing the region image comprises generating a color balanced image by correcting the color balance of the image.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the region image color space is using RGB whereby each pixel is defined by (r, g, b), and the correcting of the color balance comprises:
obtaining a gray reference pixel values (rref, gref, bref);
calculating average pixel values of the region image (ravg, gavg, bavg);
calculating a color shift (rsft, gsft, bsft) of the region image according to, or based on, (rsft, gsft, bsft)=(ravg, gavg, bavg)−(rref, gref, bref); and
calculating the color balanced image having pixels values (rc, gc, bc), where each pixel value is calculated (rc, gc, bc)=(r, g, b)−(rsft, gsft, bsft).

### Claim 18 (depends on 17)

The method according to claim 17, wherein the obtaining of a gray reference pixel values is based on, or equal to, an average of pixel values in multiple images.

### Claim 19 (depends on 1)

The method according to claim 1, wherein the analyzing of the captured image further comprises producing a list of quads in the captured image.

### Claim 20 (depends on 19)

The method according to claim 19, for use with plurality of detected corners in the captured image, wherein each of the quads is a quadrilateral having two or more vertices that are selected from the detected corners.

### Claim 21 (depends on 20)

The method according to claim 20, further comprising detecting the corners in the captured image, wherein each corner is defined by a point location and two directions from the point.

### Claim 22 (depends on 21)

The method according to claim 21, wherein the detecting of the corners is according to, is based on, or consists of, a corner detection algorithm.

### Claim 23 (depends on 22)

The method according to claim 22, wherein the detecting of the corners comprises detecting straight-line segments in the captured image according to, or based on, a pattern recognition algorithm.

### Claim 24 (depends on 1)

The method according to claim 1, wherein the analyzing of the captured image comprises generating a saliency map identifying salient pixels or saliency region in the captured image.

### Claim 25 (depends on 1)

The method according to claim 1, further comprising training the neural network to detect the photographs in the captured image.

### Claim 26 (depends on 1)

The method according to claim 1, wherein the neural network consists of, or comprises, a deep convolutional neural network.

### Claim 27 (depends on 1)

The method according to claim 1, wherein the WLAN is according to, or based on, IEEE 802.11-2012, IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, or IEEE 802.11ac.

### Claim 28 (depends on 1)

The method according to claim 1, further comprising obtaining the captured image by the digital camera.

### Claim 29 (depends on 1)

The method according to claim 1, wherein the device is housed in a single enclosure and comprising in the single enclosure the digital camera, and wherein the device is battery-operated and consists of, comprises, or is part of, a notebook, a laptop computer, a media player, a cellular phone, a tablet, a Personal Digital Assistant (PDA), or an image-processing device.

### Claim 30 (depends on 1)

A non-transitory tangible computer readable storage media comprising code to perform the steps of the method of claim 1.