# US 2023/0056672 A1 · Scalable road sign knowledge graph construction with machine learning and human in the loop > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0056672 A1 - Document ID: US20230056672A1 - Status: Published application - Kind: A1 - Application number: 17/404,295 - Filed: 2021-08-17 - Published: 2023-02-23 - Earliest claimed: 2021-08-17 - Assignees: Robert Bosch GmbH - Applicants: Robert Bosch GmbH - Inventors: Ji Eun KIM; Kevin H. HUANG; Mohammad Sadegh NOROUZZADEH; Shashank SHEKHAR - Web page: https://patentgrep.com/patent/US20230056672A1 ## CPC - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/582: {of traffic signs} - G06F 18/2178: {based on feedback of a supervisor} - G06N 5/02: Knowledge representation; Symbolic representation - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 20/00: Machine learning - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for constructing and managing a unique road sign knowledge graph across various countries and regions is disclosed. The system utilizes machine learning methods to assist humans when comparing a new sign template with a plurality of stored sign templates to reduce or eliminate redundancy in the road sign knowledge graph. Such a machine learning method and system is also used in providing visual attributes of road signs such as sign shapes, colors, symbols, and the like. If the machine learning determines that the input road sign template is not found in the road sign knowledge graph, the input sign template can be added to the road sign knowledge graph. The road sign knowledge graph can be maintained to add signs templates that are not already in the knowledge graph but are found in real-world by integrating human annotator's feedback during ground truth generation for machine learning. ## Claims ### Claim 1 A system of constructing a road sign knowledge graph for use in machine-learning road sign recognition, including labeling tools to train machine learning models, the system comprising: a user interface; a storage configured to maintain (i) a road sign knowledge graph including a database of stored road sign templates accompanied with a plurality of stored visual attributes of the stored road sign templates, and (ii) image data including video having a sequence of frames, and annotations of the frames that indicate visual attributes of road signs identified in respective frames; and a processor in communication with the storage and the user interface, the processor programmed to: receive an input road sign template from a road sign template source, determine a plurality of visual attributes of the input road sign template utilizing a visual attribute recognition machine learning model, query the road sign knowledge graph for one of the stored road sign templates having stored visual attributes that correspond with the plurality of visual attributes of the input road sign template, in response to the query yielding one of the stored road sign templates with stored visual attributes that correspond with the plurality of visual attributes of the input road sign template, not update the road sign knowledge graph to include the input road sign template, and in response to the query failing to yield one of the stored road sign templates with stored visual attributes that correspond with the plurality of visual attributes of the input road sign template, receive, via the user interface, annotations corresponding to the visual attributes of the input road sign template, and update the road sign knowledge graph to include the input road sign template. ### Claim 2 (depends on 1) The system of claim 1, wherein the plurality of visual attributes of the input road sign template includes at least one of a background color, a presence of text, and a symbol. ### Claim 3 (depends on 1) The system of claim 1, wherein the annotations are received from a crowdsourcing platform such that a plurality of crowdsourcing annotators provide the annotations. ### Claim 4 (depends on 1) The system of claim 1, wherein the processor is programmed to, based on the query failing to yield one of the stored road sign templates with stored visual attributes that correspond with the plurality of visual attributes of the input road sign template, receive domain-specific information regarding the input road sign template and update the road sign knowledge graph to include the domain-specific information. ### Claim 5 (depends on 1) The system of claim 1, wherein the processor is programmed to, based on the query yielding one of the stored road sign templates with stored visual attributes that correspond with the plurality of visual attributes of the input road sign template, receive human-in-loop validation via the user interface that the input road sign template corresponds to the one of the stored road sign templates in the road sign knowledge graph. ### Claim 6 (depends on 1) The system of claim 1, wherein the processor is programmed to: receive annotations from a video annotator regarding incorrect visual attributes for the input road sign template stored in the road sign knowledge graph, and update the road sign knowledge graph with corrected visual attributes via the user interface. ### Claim 7 A method of constructing and maintaining a road sign knowledge graph for use in a machine-learning road sign recognition system, the method comprising: maintaining a road sign knowledge graph including a database of stored road sign templates accompanied with a plurality of stored visual attributes corresponding to the stored road sign templates; maintaining image data including video having a sequence of frames, and annotations of one or more of the frames that indicate visual attributes of road signs identified in the respective frames; receiving an input road sign template from a road sign template source; determining a plurality of visual attributes of the input road sign template utilizing a visual attribute recognition machine learning model; querying a road sign knowledge graph for one of the stored road sign templates having stored visual attributes that correspond with the plurality of visual attributes of the input road sign template; confirming the input road sign template is already stored in the road sign knowledge graph based on a comparison between the visual attributes of the input road sign template and the one of the stored road sign templates; and preventing the input road sign template form being added to the road sign knowledge graph based on the confirming to prevent redundancy within the road sign knowledge graph. ### Claim 8 (depends on 7) The method of claim 7, further comprising: confirming the input road sign template is not already stored in the road sign knowledge graph based on a comparison between the visual attributes of the input road sign template and the stored visual attributes of the stored road sign templates; and updating the road sign knowledge graph to include the input road sign template. ### Claim 9 (depends on 8) The method of claim 8, further comprising: receiving domain-specific information regarding the input road sign template and update the road sign knowledge graph to include the domain-specific information. ### Claim 10 (depends on 8) The method of claim 8, further comprising: receiving annotations corresponding to the visual attributes of the input road sign template, and updating the road sign knowledge graph to include the visual attributes. ### Claim 11 (depends on 8) The method of claim 8, wherein the annotations are received via a crowdsourcing platform of a plurality of annotators. ### Claim 12 (depends on 8) The method of claim 8, further comprising: receiving annotations from a video annotator regarding incorrect visual attributes for the input road sign in the road sign knowledge graph based on a visual check of a video feed of a real-world road sign corresponding to the input road sign. ### Claim 13 (depends on 7) The method of claim 7, wherein the utilizing the visual attribute recognition machine learning model includes determining probabilities of predefined visual attributes presented in the input road sign template. ### Claim 14 (depends on 7) The method of claim 7, further comprising: receiving human-in-loop validation via a user interface that the input road sign template corresponds to the one of the stored road sign templates in the road sign knowledge graph. ### Claim 15 A system comprising: a user interface; and a processor in communication with the display and programmed to: receive an input road sign template; execute a visual attribute prediction model to determine probabilities of predefined visual attributes present in the input road sign template; when the determined probabilities of predefined visual attributes correspond with stored visual attributes of one or more stored road sign templates within a road sign knowledge graph, prevent the input road sign template from being stored in the road sign knowledge graph; and when the determined probabilities of predefined visual attributes do not correspond with stored visual attributes of the one or more stored road sign templates, receive annotations corresponding to the visual attributes present in the input road sign template, and update the road sign knowledge graph to include the input road sign template. ### Claim 16 (depends on 15) The system of claim 15, wherein the input road sign template is received from a road sign template source containing a plurality of official road sign templates. ### Claim 17 (depends on 15) The system of claim 15, wherein the input road sign template is received from a video annotator viewing video feed, and the input road sign template includes annotations that indicate visual attributes of the input road sign as identified by the video annotator. ### Claim 18 (depends on 15) The system of claim 15, wherein the processor is further programmed to: when the determined probabilities of predefined visual attributes do not correspond with stored visual attributes of the one or more stored road sign templates, receive domain-specific information regarding the input road sign template and update the road sign knowledge graph to include the domain-specific information. ### Claim 19 (depends on 15) The system of claim 15, wherein the processor is further programmed to: when the determined probabilities of predefined visual attributes correspond with the stored visual attributes of the one or more stored road sign templates, receive human-in-loop validation via a user interface that the input road sign template corresponds to the one or more stored road sign templates in the road sign knowledge graph. ### Claim 20 (depends on 15) The system of claim 15, wherein the processor is further programmed to: receive annotations from a video annotator regarding incorrect visual attributes for the input road sign template stored in the road sign knowledge graph, and update the road sign knowledge graph with corrected visual attributes via a user interface.