Model-aware synthetic image generation

#Software for vision tasks utilizes machine learning. Machine learning models for vision related tasks such as image recognition are trained using labeled images. In other words, large amounts of images with labels identifying what is in the images are used to create the models. Once trained, the models are used for the vision tasks.

#It can be very time consuming to obtain and label the large number of images that are needed for obtaining accurate models. Images that are created using computer software can be used to supplement the real images. Such computer created images are referred to as synthetic images. A problem with synthetic images is that there is a domain gap between real images and synthetic images such that the use of synthetic images to train a model can actually hurt the performance of the resulting model.

#Synthetics images can differ from real images is because the are somehow “cleaner” than real images, since by definition they are the result of a simulation and the simulation cannot capture reality in all its complexity but only the most dominant aspects of it. The sampling accuracy of synthetic images is limited by the available compute power, there is a budget for how many sampling rays that can be sent into a scene for tracking and how many dispersion rays can be calculated. This can lead to inaccurately capturing high frequency details in the scene compared to real images where the sampling density and dispersion rate of photon particles is much higher. When the synthetic images are simulation are not close enough to reality, performance of the ML model will be impaired because the model will not be able to transfer what was learned from the synthetic images to reality. The model will not generalize.

#FIG. 1 is a block diagram illustrating an end-to-end synthetic image data generation framework according to an example embodiment.

#FIG. 2 is a flowchart illustrating an end-to-end computer implemented method of jointly optimizing DNN models for image related tasks according to an example embodiment.

#FIG. 3 is a flowchart illustrating a computer implemented method of optimizing knob parameters for creation of synthetic images that have characteristics suitable for use in training DNNs for vision related tasks according to an example embodiment.

#In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present invention. The following description of example embodiments is, therefore, not to be taken in a limited sense, and the scope of the present invention is defined by the appended claims.

#The functions or algorithms described herein may be implemented in software in one embodiment. The software may consist of computer executable instructions stored on computer readable media or computer readable storage device such as one or more non-transitory memories or other type of hardware-correspond to modules, which may be software, hardware, firmware or any combination thereof. Multiple functions may be performed in one or more modules as desired, and the embodiments described are merely examples. The software may be executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a computer system, such as a personal computer, server or other computer system, turning such computer system into a specifically programmed machine.

#The functionality can be configured to perform an operation using, for instance, software, hardware, firmware, or the like. For example, the phrase “configured to” can refer to a logic circuit structure of a hardware element that is to implement the associated functionality. The phrase “configured to” can also refer to a logic circuit structure of a hardware element that is to implement the coding design of associated functionality of firmware or software. The term “module” refers to a structural element that can be implemented using any suitable hardware (e.g., a processor, among others), software (e.g., an application, among others), firmware, or any combination of hardware, software, and firmware. The term, “logic” encompasses any functionality for performing a task. For instance, each operation illustrated in the flowcharts corresponds to logic for performing that operation. An operation can be performed using, software, hardware, firmware, or the like. The terms, “component,” “system,” and the like may refer to computer-related entities, hardware, and software in execution, firmware, or combination thereof. A component may be a process running on a processor, an object, an executable, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term, “processor,” may refer to a hardware component, such as a processing unit of a computer system.