# US 11,893,766 B2 · Neural network system and operating method thereof > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,893,766 B2 - Document ID: US11893766B2 - Status: Granted - Kind: B2 - Application number: 17/031,039 - Filed: 2020-09-24 - Published: 2021-06-10 - Granted: 2024-02-06 - Earliest claimed: 2019-12-09 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Seungsoo Yang - Web page: https://patentgrep.com/patent/US11893766B2 ## CPC - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06F 18/22: Matching criteria, e.g. proximity measures - G06T 7/11: Region-based segmentation - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 2201/10: Recognition assisted with metadata - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A neural network system, includes: a processor configured to detect a plurality of object candidates included in a first image, generate metadata corresponding to the plurality of object candidates based on the first image, and set data processing orders of the plurality of object candidates based on the metadata; and at least one resource configured to perform data processing with respect to the plurality of object candidates. The processor is configured to sequentially provide pieces of information related to data processing of the plurality of object candidates to the at least one resource according to the set data processing orders, and the at least one resource is configured to sequentially perform data processing with respect to the plurality of object candidates according to an order in which a piece of information related to data processing of each of the plurality of object candidates is received. ## Claims ### Claim 1 A neural network system, comprising: a processor configured to detect a plurality of object candidates included in a first image, generate metadata corresponding to the plurality of object candidates based on the first image, and set data processing orders of the plurality of object candidates based on the metadata; and at least one resource configured to perform data processing with respect to the plurality of object candidates, wherein the processor is further configured to sequentially provide pieces of information related to data processing of the plurality of object candidates to the at least one resource according to the set data processing orders of the plurality of object candidates, wherein the at least one resource is further configured to sequentially perform data processing with respect to the plurality of object candidates according to an order in which a piece of information related to data processing of each of the plurality of object candidates is received, wherein the at least one resource is further configured to perform first object recognition with respect to a first object candidate of a first order, and execute a first task corresponding to a result of the first object recognition, the first task being related to an operation of generating a control command for controlling a device to which the neural network system is applied, and wherein the processor is further configured to, based on completion of the first task, provide a piece of information related to data processing of a second object candidate of a next order to the first order, among the plurality of object candidates, to the at least one resource. ### Claim 2 (depends on 1) The neural network system of claim 1, wherein the at least one resource is further configured to: based on receiving the piece of information related to data processing of the second object candidate, perform second object recognition with respect to the second object candidate, and execute a second task corresponding to a result of the second object recognition. ### Claim 3 (depends on 2) The neural network system of claim 2, wherein the processor is further configured to determine a plurality of regions of interest (RoIs) of the first image and detect the plurality of RoIs as the plurality of object candidates. ### Claim 4 (depends on 1) The neural network system of claim 1, wherein the processor is further configured to obtain scores of the plurality of object candidates according to one or more criteria based on the metadata, and set the data processing orders of the plurality of object candidates by using the obtained scores. ### Claim 5 (depends on 4) The neural network system of claim 4, wherein the processor is further configured to set the data processing orders of the plurality of object candidates according to magnitudes of score values of the plurality of object candidates. ### Claim 6 (depends on 4) The neural network system of claim 4, wherein the processor is further configured to identify an object candidate having a score value greater then a first threshold and immediately provide information related to data processing of the object candidate to the at least one resource. ### Claim 7 (depends on 4) The neural network system of claim 4, wherein the processor is further configured to identify an object candidate having a score value less than a second threshold and not set a data processing order of the identified object candidate. ### Claim 8 (depends on 1) The neural network system of claim 1, wherein the metadata comprises information of at least one of a size, a position in the first image, a top left coordinate, a bottom right coordinate, a distance from a preset coordinate, or a depth of each of the plurality of object candidates. ### Claim 9 (depends on 1) The neural network system of claim 1, wherein the processor is further configured to set the data processing orders of the plurality of object candidates further based on metadata of the plurality of object candidates included in a second image that is a previous frame of the first image. ### Claim 10 (depends on 1) The neural network system of claim 1, wherein the at least one resource comprises a plurality of resources, and wherein the processor is further configured to match the plurality of object candidates to the plurality of resources, based on the metadata corresponding to the plurality of object candidates, and respectively provide the pieces of information related to data processing of the plurality of object candidates to the plurality of resources, based on a matching result. ### Claim 11 (depends on 10) The neural network system of claim 10, wherein the processor is further configured to identify operation states of the plurality of resources and match the plurality of object candidates to the plurality of resources further based on the identified operation states. ### Claim 12 (depends on 1) The neural network system of claim 1, wherein the at least one resource comprises a plurality of task processing modules, and wherein the at least one resource is further configured to match the plurality of task processing modules to the plurality of object candidates, based on the metadata corresponding to the plurality of object candidates, and respectively provide the pieces of information related to data processing of the plurality of object candidates to the plurality of task processing modules, based on a matching result. ### Claim 13 An operating method of a neural network system, the operating method comprising: detecting a plurality of object candidates included in a first image; performing first object recognition with respect to a first object candidate of a first order among the plurality of object candidates; executing a first task corresponding to a result of the first object recognition, the first task being related to an operation of generating a control command for controlling a device to which the neural network system is applied; performing second object recognition with respect to a second object candidate of a next order to the first order, among the plurality of object candidates, upon completion of the first task; and executing a second task corresponding to a result of the second object recognition. ### Claim 14 (depends on 13) The operating method of claim 13, further comprising: generating metadata corresponding to the plurality of object candidates, based on the first image; selecting the first object candidate among the plurality of object candidates, based on the metadata; and selecting the second object candidate, among remaining object candidates excluding the first object candidate, based on the metadata. ### Claim 15 (depends on 14) The operating method of claim 14, wherein the selecting the first object candidate based on the metadata comprises: obtaining scores of the plurality of object candidates according to one or more criteria, based on the metadata; and selecting the first object candidate based on the obtained scores. ### Claim 16 (depends on 15) The operating method of claim 15, wherein the selecting the first object candidate based on the metadata comprises selecting an object candidate having a score value greater than a threshold among the obtained scores, as the first object candidate. ### Claim 17 (depends on 15) The operating method of claim 15, wherein the selecting the first object candidate based on the metadata comprises selecting an object candidate having a largest score value among the plurality of object candidates as the first object candidate. ### Claim 18 (depends on 15) The operating method of claim 15, wherein the selecting the second object candidate comprises selecting an object candidate having a largest score value, among the remaining object candidates excluding the first object candidate, as the second object candidate. ### Claim 19 An electronic device, comprising: a sensor configured to obtain data about a vicinity of the electronic device and output a first image based on the obtained data; at least one resource configured to perform object recognition with respect to the first image; a memory configured to store programs; and a processor configured to read the programs and operate as instructed by the programs, to detect a plurality of object candidates in the first image, generate metadata of the plurality of object candidates based on the first image, and provide, to the at least one resource, information about a first object candidate, that is selected based on the metadata among the plurality of object candidates, wherein the at least one resource is further configured to perform first object recognition with respect to the first object candidate of a first order and execute a first task corresponding to a result of the first object recognition, the first task being related to an operation of generating a control command for controlling the electronic device, and wherein the processor is further configured to, based on completion of the first task, provide information about a second object candidate of a next order to the first order, among the plurality of object candidates, to the at least one resource. ### Claim 20 (depends on 19) The electronic device of claim 19, wherein the at least one resource is further configured to, based on receiving the information about the second object candidate, perform second object recognition with respect to the second object candidate and execute a second task corresponding to a result of the second object recognition.