# US 12,002,252 B2 · Image matching system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,252 B2 - Document ID: US12002252B2 - Status: Granted - Kind: B2 - Application number: 17/318,984 - Filed: 2021-05-12 - Published: 2022-11-17 - Granted: 2024-06-04 - Earliest claimed: 2021-05-12 - Assignees: Rakuten Group, Inc. - Applicants: Rakuten Group, Inc. - Inventors: Enyuan Wu; Luis Bernardo - Web page: https://patentgrep.com/patent/US12002252B2 ## CPC - G06V 10/443: {by matching or filtering} - G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56) - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 3/60: Rotation of whole images or parts thereof - G06T 7/11: Region-based segmentation - G06T 7/13: Edge detection - G06T 7/194: involving foreground-background segmentation - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/32: Normalisation of the pattern dimensions - G06V 10/56: relating to colour - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06T 2207/10024: Color image - G06T 2207/20132: Image cropping - G06T 2207/20212: Image combination - G06T 2210/12: Bounding box - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An image matching system includes a non-transitory computer-readable medium and a processor. The non-transitory computer-readable medium is configured to store information of a plurality of images. The processor is configured to identify an object area in an original image that illustrates an object. The processor is configured to normalize the object area, resulting in a normalized image. The processor is configured to calculate a shape vector and a color vector from the normalized image. The processor is configured to calculate a match score using the shape vector and the color vector. The processor is configured to determine if the non-transitory computer-readable medium stores an identical match for the original image based on the match score. ## Claims ### Claim 1 An image matching system, comprising: a non-transitory computer-readable medium configured to store information of a plurality of images; and a processor configured to: identify an object area in an original image that illustrates an object; normalize the object area, resulting in a normalized image; calculate a shape vector and a color vector from the normalized image; calculate a match score using the shape vector and the color vector; and determine if the non-transitory computer-readable medium stores an identical match for the original image based on the match score, wherein the processor is configured to: generate a sketch image from the original image; identify a bounding rectangle in the sketch image; crop the sketch image with the bounding rectangle, resize and pad the cropped sketch image, resulting in the normalized image; and calculate the shape vector from the normalized image. ### Claim 2 (depends on 1) The image matching system of claim 1, wherein the processor is configured to: pre-rotate the sketch image based on a principal axis of the object; and generate the normalized image from the pre-rotated sketch image. ### Claim 3 (depends on 2) The image matching system of claim 2, wherein the processor is configured to: generate one or more normalized images by further rotating the pre-rotated sketch image; and calculate the shape vector from each of the normalized images. ### Claim 4 (depends on 1) The image matching system of claim 1, wherein the processor is configured to: generate a silhouette image from the original image; resize the silhouette image, resulting in the normalized image; and calculate the color vector from color and the normalized image. ### Claim 5 (depends on 1) The image matching system of claim 1, wherein the processor is configured to: determine if the identical match is falsely identified by performing false identical match check. ### Claim 6 (depends on 1) The image matching system of claim 1, wherein the processor is configured to generate the sketch image from the original image by using edge detection. ### Claim 7 (depends on 6) The image matching system of claim 6, wherein the processor is configured to use edge detection with three different sets of threshold values, dilate with different iterations for high-resolution, medium-resolution and low-resolution edge images, and combine the three edge images with a logical OR operation. ### Claim 8 (depends on 4) The image matching system of claim 4, wherein the processor is configured to generate the silhouette image from the original image by segmenting main objects from a background using image segmentation. ### Claim 9 (depends on 4) The image matching system of claim 4, wherein the processor is configured to: generate from the original image a sketch image, based on which the shape vector is calculated; and perform a logical OR operation between the silhouette image and the sketch image. ### Claim 10 (depends on 1) The image matching system of claim 1, wherein the processor is configured to crop the sketch image with the bounding rectangle so that the object is enclosed in the bounding rectangle and touches either both top and bottom edges or both left and right edges of the bounding rectangle. ### Claim 11 (depends on 1) The image matching system of claim 1, wherein the processor is configured to resize the cropped sketch image to a standard size. ### Claim 12 (depends on 1) The image matching system of claim 1, wherein the processor is configured to pad the resized sketch image to be square. ### Claim 13 (depends on 1) The image matching system of claim 1, wherein the processor is configured to calculate the shape vector by dividing the normalized sketch image into a plurality of cells. ### Claim 14 (depends on 13) The image matching system of claim 13, wherein the processor is configured to count bright pixels per cell. ### Claim 15 (depends on 1) The image matching system of claim 1, wherein the processor is configured to calculate the color vector by looping through each foreground pixel and assign it to one of 64 buckets. ### Claim 16 (depends on 1) The image matching system of claim 1, wherein the processor is configured to normalize the color vector. ### Claim 17 (depends on 1) The image matching system of claim 1, wherein the processor is configured to calculate four locality sensitive hashing values for both the shape vector and the color vector. ### Claim 18 An image matching method, comprising: identifying, by a processor, an object area in an original image that illustrates an object; normalizing, by the processor, the object area, resulting in a normalized image; calculating, by the processor, a shape vector and a color vector from the normalized image; calculating, by the processor, a match score using the shape vector and the color vector; and determining, by the processor, if a non-transitory computer-readable medium stores an identical match for the original image based on the match score, wherein the non-transitory computer-readable medium stores information of a plurality of images, wherein the processor is configured to: generate a sketch image from the original image; identify a bounding rectangle in the sketch image; crop the sketch image with the bounding rectangle, resize and pad the cropped sketch image, resulting in the normalized image; and calculate the shape vector from the normalized image. ### Claim 19 A computer program product for image matching, the computer program product comprising a computer-readable storage medium containing computer program code for: identifying, by a processor, an object area in an original image that illustrates an object; normalizing, by the processor, the object area, resulting in a normalized image; calculating, by the processor, a shape vector and a color vector from the normalized image; calculating, by the processor, a match score using the shape vector and the color vector; and determining, by the processor, if a non-transitory computer-readable medium stores an identical match for the original image based on the match score, wherein the non-transitory computer-readable medium stores information of a plurality of images, wherein the processor is configured to: generate a sketch image from the original image; identify a bounding rectangle in the sketch image; crop the sketch image with the bounding rectangle, resize and pad the cropped sketch image, resulting in the normalized image; and calculate the shape vector from the normalized image.