# US 2020/0005075 A1 · Computer Vision Systems and Methods for Automatically Detecting, Classifying, and Pricing Objects Captured in Images or Videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0005075 A1
- Document ID: US20200005075A1
- Status: Published application
- Kind: A1
- Application number: 16/458,827
- Filed: 2019-07-01
- Published: 2020-01-02
- Earliest claimed: 2018-06-29
- Assignees: Geomni, Inc.
- Applicants: Geomni, Inc.
- Inventors: Bryce Zachary Porter; Dean Lebaron
- Web page: https://patentgrep.com/patent/US20200005075A1

## CPC

- G06Q 30/0278: {Product appraisal}
- G06F 18/24: Classification techniques
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method for automatically detecting, classifying, and processing objects captured in an image. The system receives an image from the image source and detects one or more objects in the image. The system then performs a high-level classification of each of the one or more objects in the image and extracts each of the one or more objects from the image. The system then performs a specific classification of each of the one or more objects and determines a price of each of the one or more objects. Finally, the system generates a pricing report comprising a price of each of the one or more objects.

## Claims

### Claim 1

A system for automatically detecting, classifying, and processing objects captured in an image, comprising:
a processor in communication with an image source; and
computer system code executed by the processor, the computer system code causing the processor to:
receive an image from the image source;
detect one or more objects in the image;
perform a high-level classification of each of the one or more objects in the image;
extract each of the one or more objects from the image;
perform a specific classification of each of the one or more objects;
determine a price of each of the one or more objects; and
generate a report comprising a price of each of the one or more objects.

### Claim 2 (depends on 1)

The system of claim 1, wherein the image comprises a photograph or a video frame.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processor performs the steps of detecting the at least one object in the image and performing the high-level classification of the at least one object in the image using a convolutional neural network (“CNN”).

### Claim 4 (depends on 3)

The system of claim 3, wherein the CNN generates one or more bounding boxes on the image.

### Claim 5 (depends on 4)

The system of claim 4, wherein each of the one or more bounding boxes are assigned a confidence score.

### Claim 6 (depends on 5)

The system of claim 5, wherein the system retains each of the one or more bounding boxes with a confidence score higher than a predetermined threshold and discards each of the one or more bounding boxes with a confidence score lower than the predetermined threshold.

### Claim 7 (depends on 6)

The system of claim 6, wherein the system selects a single bounding box when there are more than one bounding boxes, using a non-maximal suppression method.

### Claim 8 (depends on 4)

The system of claim 4, wherein the system transforms the one or more bounding boxes into an original image space using scaling parameters.

### Claim 9 (depends on 4)

The system of claim 4, wherein the processor performs the step of extracting the at least object from the image by cropping out the one or more bounding boxes.

### Claim 10 (depends on 1)

The system of claim 1, wherein the system tracks each of the one or more objects in further images using a tracking algorithm.

### Claim 11 (depends on 1)

The system of claim 1, wherein the tracking algorithm comprises a Multiple Instance Learning algorithm or a Kernelized Correlation Filters algorithm.

### Claim 12 (depends on 10)

The system of claim 10, wherein when a new object is detected in the further images, the system performs a high-level classification of the new object.

### Claim 13 (depends on 1)

The system of claim 1, wherein step of performing the specific classification comprises comparing an object from the one or more objects to images of items stored in a database and generating a score between the object and each of the images of items.

### Claim 14 (depends on 13)

The system of claim 13, wherein the system generates the score using a key point matching algorithm.

### Claim 15 (depends on 14)

The system of claim 14, wherein the key point matching algorithm generates key point descriptors at locations on the object and at locations on each of the images of items and compares the descriptors to identify matching points.

### Claim 16 (depends on 15)

The system of claim 15, wherein the descriptors comprise at least one of scale-invariant feature transform (“SIFT”) descriptors, histogram of oriented gradients (“HoG”) descriptors, or KAZE descriptors.

### Claim 17 (depends on 1)

The system of claim 1, wherein, prior to the step of detecting the at least one object in the image, the system preprocesses the image using a normalization process or a channel value centering process.

### Claim 18

A method for automatically detecting, classifying, and processing objects captured in an image, comprising steps of:
receiving an image;
detecting one or more objects in the image;
performing a high-level classification of each of the one or more objects in the image;
extracting each of the one or more objects from the image;
performing a specific classification of each of the one or more objects;
determining a price of each of the one or more objects; and
generating a pricing report comprising a price of each of the one or more objects.

### Claim 19 (depends on 18)

The method of claim 18, wherein the image comprises a photograph or a video frame.

### Claim 20 (depends on 18)

The method of claim 18, wherein the steps of detecting the at least one object in the image and performing the high-level classification of the at least one object in the image are performed using a convolutional neural network (“CNN”).

### Claim 21 (depends on 20)

The method of claim 20, wherein the CNN generates one or more bounding boxes on the image.

### Claim 22 (depends on 21)

The method of claim 21, wherein each of the one or more bounding boxes are assigned a confidence score.

### Claim 23 (depends on 22)

The method of claim 22, further comprising retaining each of the one or more bounding boxes with a confidence score higher than a predetermined threshold and discarding each of the one or more bounding boxes with a confidence score lower than the predetermined threshold.

### Claim 24 (depends on 23)

The method of claim 23, further comprising selecting a single bounding box when there are more than one bounding boxes, using a non-maximal suppression method.

### Claim 25 (depends on 21)

The method of claim 21, further comprising transforming the one or more bounding boxes into an original image space using scaling parameters.

### Claim 26 (depends on 21)

The method of claim 21, further comprising performing step of extracting the at least object from the image by cropping out the one or more bounding boxes.

### Claim 27 (depends on 18)

The method of claim 18, further comprising tracking each of the one or more objects in further images using a tracking algorithm.

### Claim 28 (depends on 18)

The method of claim 18, wherein the tracking algorithm comprises a Multiple Instance Learning algorithm or a Kernelized Correlation Filters algorithm.

### Claim 29 (depends on 27)

The method of claim 27, further comprising performing a high-level classification of a new object when the new object is detected in the further images.

### Claim 30 (depends on 18)

The method of claim 18, wherein the step of performing the specific classification comprises comparing an object from the one or more objects to images of items stored in a database and generating a score between the object and each of the images of items.

### Claim 31 (depends on 30)

The method of claim 30, further comprising generating the score with a key point matching algorithm.

### Claim 32 (depends on 31)

The method of claim 31, wherein the key point matching algorithm generates key point descriptors at locations on the object and at locations on each of the images of items and compares the descriptors to identify matching points.

### Claim 33 (depends on 32)

The method of claim 32, wherein the descriptors comprise at least one of scale-invariant feature transform (“SIFT”) descriptors, histogram of oriented gradients (“HoG”) descriptors, or KAZE descriptors.

### Claim 34 (depends on 18)

The method of claim 18, further comprising, prior to the step of detecting the at least one object in the image, preprocessing the image using a normalization process or a channel value centering process.
