# US 12,112,513 B2 · System and method for identifying non-standard user interface object

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,513 B2
- Document ID: US12112513B2
- Status: Granted
- Kind: B2
- Application number: 17/511,711
- Filed: 2021-10-27
- Published: 2022-04-28
- Granted: 2024-10-08
- Earliest claimed: 2020-10-27
- Assignees: Samsung Sds Co., Ltd.
- Applicants: Samsung Sds Co., Ltd.
- Inventors: Hyo Young Kim; Koo Hyun Park; Keun Taek Park
- Web page: https://patentgrep.com/patent/US12112513B2

## CPC

- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06F 16/2282: {Tablespace storage structures; Management thereof}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 30/1912: {Selecting the most significant subset of features (G06V30/19127 takes precedence)}
- G06V 30/19127: {Extracting features by transforming the feature space, e.g. multidimensional scaling; Mappings, e.g. subspace methods}
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06V 30/10: Character recognition
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A non-standard user interface object identification system includes an object candidate extractior that extracts one or more objects from an image, a first similarity analyzer that determines object type candidates of the one or more objects in accordance with similarities between the one or more objects and a standard user interface (UI) element, a second similarity analyzer that selects object type-specific weight values in accordance with layout characteristics of the one or more objects and determines object types of the one or more objects using the object type candidates and the object type-specific weight values, and an object identifier that receives type and characteristic information of a search target object and identifies the search target object in accordance with characteristic information and the object types of the one or more objects.

## Claims

### Claim 1

A non-standard user interface object identification system comprising:
an object candidate extractior extracting one or more objects from an image;
a first similarity analyzer determining object type candidates of the one or more objects in accordance with similarities between the one or more objects and a standard user interface (UI) element;
a second similarity analyzer selecting object type-specific weight values in accordance with layout characteristics of the one or more objects and determining object types of the one or more objects using the object type candidates and the object type-specific weight values; and
an object identifier receiving type and characteristic information of a search target object and identifying the search target object in accordance with characteristic information and the object types of the one or more objects,
wherein the second similarity analyzer generates one or more groups from the one or more objects in accordance with similarities of position information of the one or more objects and selects one among a plurality of object type-specific weight values in consideration of positions of the one or more groups in the image.

### Claim 2 (depends on 1)

The non-standard user interface object identification system according to claim 1, wherein the object candidate extractior extracts bounding rectangles of the one or more objects from the image and generates an object list comprising information regarding the bounding rectangles.

### Claim 3 (depends on 2)

The non-standard user interface object identification system according to claim 2, wherein the object list comprises position information and size information of the objects.

### Claim 4 (depends on 1)

The non-standard user interface object identification system according to claim 1, wherein the first similarity analyzer calculates standard UI characteristic scores of the objects and calculates similarities of the objects to standard UI types using the standard UI characteristic scores and predetermined characteristic values of the standard UI types.

### Claim 5 (depends on 4)

The non-standard user interface object identification system according to claim 4, wherein the first similarity analyzer determines a standard UI type among the standard UI types, the similarity of which is equal to or greater than a threshold valve, to be an object type candidate of a corresponding object among the one or more objects.

### Claim 6 (depends on 1)

The non-standard user interface object identification system according to claim 1, wherein the second similarity analyzer determines the object types of the one or more objects by applying the object type-specific weight values to the similarities of the object type candidates.

### Claim 7 (depends on 1)

The non-standard user interface object identification system according to claim 1, wherein the object identifier first identifies at least one object having the same object type as the search target object, among the one or more objects, and identifies the search target object by comparing characteristic information of each of the first-identified at least one object with the characteristic information of the search target object.

### Claim 8

A non-standard user interface object identification method comprising:
extracting one or more objects from an image;
determining object type candidates of the one or more objects in accordance with similarities between the one or more objects and a standard user interface (UI) element;
selecting object type-specific weight values in accordance with layout characteristics of the one or more objects and determining object types of the one or more objects using the object type candidates and the object type-specific weight values; and
receiving type and characteristic information of a search target object and identifying the search target object in accordance with characteristic information and the object types of the one or more objects,
wherein the determination of the object types comprises generating one or more groups from the one or more objects in accordance with similarities of position information of the one or more objects and selecting one among a plurality of object type-specific weight values in consideration of positions of the one or more groups in the image.

### Claim 9 (depends on 8)

The non-standard user interface object identification method according to claim 8, wherein the extraction of the one or more objects comprises extracting bounding rectangles of the one or more objects from the image and generating an object list comprising information regarding the bounding rectangles.

### Claim 10 (depends on 9)

The non-standard user interface object identification method according to claim 9, wherein the determination of the object type candidates comprises position information and size information of the objects.

### Claim 11 (depends on 8)

The non-standard user interface object identification method according to claim 8, wherein the determination of the object type candidates comprises calculating standard UI characteristic scores of the objects and calculating similarities of the objects to standard UI types using the standard UI characteristic scores and predetermined characteristic values of the standard UI types.

### Claim 12 (depends on 11)

The non-standard user interface object identification method according to claim 11, wherein the determination of the object type candidates determines a standard UI type among the standard UI types, the similarity of which is equal to or greater than a threshold valve, to be an object type candidate of a corresponding object among the one or more objects.

### Claim 13 (depends on 8)

The non-standard user interface object identification method according to claim 8, wherein the determination of the object types determines the object types of the one or more objects by applying the object type-specific weight values to the similarities of the object type candidates.

### Claim 14 (depends on 8)

The non-standard user interface object identification method according to claim 8, wherein the identification of the search target object comprises first identifying at least one object having the same object type as the search target object, among the one or more objects, and identifying the search target object by comparing characteristic information of each of the first-identified at least one object with the characteristic information of the search target object.