# US 12,009,993 B2 · Systems and methods for selecting a machine learning model and training images for the machine learning model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,009,993 B2 - Document ID: US12009993B2 - Status: Granted - Kind: B2 - Application number: 17/664,949 - Filed: 2022-05-25 - Published: 2023-12-21 - Granted: 2024-06-11 - Earliest claimed: 2022-05-25 - Assignees: Verizon Patent and Licensing Inc. - Applicants: Verizon Patent and Licensing Inc. - Inventors: Karthik Ramaswamy; Stephen C. Opferman; Felipe Castro; Sivanaga Ravi Kumar Chunduru Venkata - Web page: https://patentgrep.com/patent/US12009993B2 ## CPC - H04L 41/16: using machine learning or artificial intelligence - H04L 43/0829: Packet loss - H04L 43/0847: {Transmission error} - H04L 67/101: based on network conditions - H04W 24/08: Testing, {supervising or monitoring} using real traffic - H04L 43/16: Threshold monitoring - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract A device may receive network data identifying at least one of a signal-to-interference-plus-noise ratio (SINR), a bit error rate (BER), a packet loss, or a frame loss associated with a radio access network (RAN), and may receive model data associated with a plurality of machine learning models. The device may receive inference confidence scores associated with the plurality of machine learning models, and may process the network data, the model data, and the inference confidence scores, with a model, to select a machine learning model from the plurality of machine learning models. The device may cause the selected machine learning model to be implemented in connection with processing images. ## Claims ### Claim 1 A method, comprising: receiving, by a device, network data identifying at least one of a signal-to-interference-plus-noise ratio (SINR), a bit error rate (BER), a packet loss, or a frame loss associated with a radio access network (RAN); receiving, by the device, model data associated with a plurality of machine learning models; receiving, by the device, inference confidence scores associated with the plurality of machine learning models, wherein the inference confidence scores are predicted based on processing the network data with historical data associated with the plurality of machine learning models, and wherein the inference confidence scores are indicative of an accuracy of results generated by the plurality of machine learning models based on the plurality of machine learning models utilizing the network data to calculate the results; determining one or more machine learning models, of the plurality of machine learning models, to be considered for selection, the one or more machine learning models being determined based on values of the network data satisfying a first network threshold and a second network threshold for each of the one or more machine learning models; selecting a machine learning model of the one or more machine learning models based on comparing a previously selected machine learning model with the selected machine learning model; causing, by the device, the selected machine learning model to be implemented in connection with processing images; processing the network data, image data identifying images to be processed by the selected machine learning model, and an inference confidence score associated with a quality of the image data, with another model, to select images, from the image data, for training the selected machine learning model; and utilizing the selected images to train the selected machine learning model. ### Claim 2 (depends on 1) The method of claim 1, wherein the model data includes one or more of: historical latency data associated with the plurality of machine learning models, historical computing resource usage data associated with the plurality of machine learning models, or historical storage usage data associated with the plurality of machine learning models. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the one or more machine learning models comprises one or more of: determining, for each of the plurality of machine learning models, whether the SINR associated with the RAN is greater than a first SINR threshold and less than a second SINR threshold; determining, for each of the plurality of machine learning models, whether the BER associated with the RAN is greater than a first BER threshold and less than a second BER threshold; determining, for each of the plurality of machine learning models, whether the packet loss associated with the RAN is greater than a first packet loss threshold and less than a second packet loss threshold; determining, for each of the plurality of machine learning models, whether the frame loss associated with the RAN is greater than a first frame loss threshold and less than a second frame loss threshold; or determining whether the inference confidence scores are less than a confidence threshold. ### Claim 4 (depends on 1) The method of claim 1, wherein selecting the selected machine learning model comprises one or more of: selecting, as the selected machine learning model, one of the one or more machine learning models with a least historical latency, a least historical computing resource usage, or a least historical storage usage. ### Claim 5 (depends on 1) The method of claim 1, wherein each of the plurality of machine learning models is a computer vision machine learning model. ### Claim 6 (depends on 1) The method of claim 1, wherein causing the selected machine learning model to be implemented comprises: causing the selected machine learning model to be implemented by a multi-access edge computing (MEC) device associated with the RAN. ### Claim 7 (depends on 6) The method of claim 6, wherein the MEC device utilizes the selected machine learning model to process images. ### Claim 8 A device, comprising: one or more processors configured to: receive network data identifying at least one of a signal-to-interference-plus-noise ratio (SINR), a bit error rate (BER), a packet loss, or a frame loss associated with a radio access network (RAN); receive model data associated with a plurality of machine learning models; receive inference confidence scores associated with the plurality of machine learning models, wherein the inference confidence scores are predicted based on processing the network data with historical data associated with the plurality of machine learning models, and wherein the inference confidence scores are indicative of an accuracy of results generated by the plurality of machine learning models based on the plurality of machine learning models utilizing the network data to calculate the results; determine one or more machine learning models, of the plurality of machine learning models, to be considered for selection, the one or more machine learning models being determined based on values of the network data satisfying a first network threshold and a second network threshold for each of the one or more machine learning models; select a machine learning model of the one or more machine learning models based on comparing a previously selected machine learning model with the selected machine learning model; cause the selected machine learning model to be implemented by a multi-access edge computing device associated with the RAN; process the network data, image data identifying images to be processed by the selected machine learning model, and an inference confidence score associated with a quality of the image data, with another model, to select images, from the image data, for training the selected machine learning model; and utilize the selected images to train the selected machine learning model. ### Claim 9 (depends on 8) The device of claim 8, wherein the one or more processors are further configured to: receive the image data and the inference confidence score, wherein processing the network data, the image data, and the inference confidence score is based on receiving the image data and the inference confidence score. ### Claim 10 (depends on 8) The device of claim 8, wherein the one or more processors, to utilize the selected images to train the selected machine learning model, are configured to: provide the selected images to the selected machine learning model for training. ### Claim 11 (depends on 8) The device of claim 8, wherein the one or more processors are further configured to: provide the selected images to the selected machine learning model for development of one or more new machine learning models. ### Claim 12 (depends on 8) The device of claim 8, wherein the one or more processors, to process the network data, the image data, and the inference confidence score, with the other model, to select the selected images, are configured to: determine whether the SINR associated with the RAN is greater than an SINR threshold; determine whether the BER associated with the RAN is greater than a BER threshold; determine whether the packet loss associated with the RAN is greater than a packet loss threshold; determine whether the frame loss associated with the RAN is greater than a frame loss threshold; or determine whether the inference confidence score is less than a confidence threshold. ### Claim 13 (depends on 8) The device of claim 8, wherein the one or more processors are further configured to: receive updated network data and updated inference confidence scores; process the updated network data, the model data, and the updated inference confidence scores to select another machine learning model from the plurality of machine learning models; and cause the selected other machine learning model to be implemented. ### Claim 14 A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to: receive network data identifying at least one of a signal-to-interference-plus-noise ratio (SINR), a bit error rate (BER), a packet loss, or a frame loss associated with a radio access network (RAN); receive model data associated with a plurality of machine learning models; receive inference confidence scores associated with the plurality of machine learning models, wherein the inference confidence scores are predicted based on processing the network data with historical data associated with the plurality of machine learning models, and wherein the inference confidence scores are indicative of an accuracy of results generated by the plurality of machine learning models based on the plurality of machine learning models utilizing the network data to calculate the results; determine one or more machine learning models, of the plurality of machine learning models, to be considered for selection, the one or more machine learning models being determined based on values of the network data satisfying a first network threshold and a second network threshold for each of the one or more machine learning models; select a machine learning model of the one or more machine learning models based on comparing a previously selected machine learning model with the selected machine learning model; cause the selected machine learning model to be implemented; receive image data identifying images to be processed by the selected machine learning model; receive an inference confidence score associated with a quality of the image data; process the network data, the image data, and the inference confidence score, with another model, to select images, from the image data, for training the selected machine learning model; and utilize the selected images to train the selected machine learning model. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the model data includes one or more of: historical latency data associated with the plurality of machine learning models, historical computing resource usage data associated with the plurality of machine learning models, or historical storage usage data associated with the plurality of machine learning models. ### Claim 16 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, that cause the device to determine the one or more machine learning models, cause the device to: determine, for each of the plurality of machine learning models, whether the SINR associated with the RAN is greater than a first SINR threshold and less than a second SINR threshold; determine, for each of the plurality of machine learning models, whether the BER associated with the RAN is greater than a first BER threshold and less than a second BER threshold; determine, for each of the plurality of machine learning models, whether the packet loss associated with the RAN is greater than a first packet loss threshold and less than a second packet loss threshold; determine, for each of the plurality of machine learning models, whether the frame loss associated with the RAN is greater than a first frame loss threshold and less than a second frame loss threshold; or determine whether the inference confidence scores are less than a confidence threshold. ### Claim 17 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions further cause the device to: provide the selected images to the selected machine learning model for training. ### Claim 18 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the selected machine learning model is implemented by a multi-access edge computing device associated with the RAN. ### Claim 19 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, that cause the device to process the network data, the image data, and the inference confidence score, with the other model, to select the selected images, cause the device to: determine whether the SINR associated with the RAN is greater than an SINR threshold; determine whether the BER associated with the RAN is greater than a BER threshold; determine whether the packet loss associated with the RAN is greater than a packet loss threshold; determine whether the frame loss associated with the RAN is greater than a frame loss threshold; or determine whether the inference confidence score are less than a confidence threshold. ### Claim 20 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein each of the plurality of machine learning models is a computer vision machine learning model.