# US 11,443,414 B2 · Image signal processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,414 B2 - Document ID: US11443414B2 - Status: Granted - Kind: B2 - Application number: 17/075,299 - Filed: 2020-10-20 - Published: 2021-04-29 - Granted: 2022-09-13 - Earliest claimed: 2019-10-23 - Assignees: Arm Limited - Applicants: Apical Limited - Inventors: Maxim Novikov; James Stuart Imber; Yury Khrustalev; David Hanwell - Web page: https://patentgrep.com/patent/US11443414B2 ## CPC - H04N 1/6033: {using test pattern analysis (H04N1/6055 takes precedence)} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/24: Classification techniques - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T 7/0002: {Inspection of images, e.g. flaw detection} - G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)} - G06V 10/993: {Evaluation of the quality of the acquired pattern} - H04N 25/611: Correction of chromatic aberration - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30168: Image quality inspection - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of optimising an image signal processor (ISP), which is to be used to process sensor image data generating output image data. The method may include obtaining sensor image data; processing the sensor image data according to one or more ISP settings to produce output image data; producing quality metric data associated with the output image data and optimising the one or more ISP settings based on the quality metric data. ## Claims ### Claim 1 A method of optimising an image signal processor (ISP), which is to be used to process sensor image data generating output image data, the method comprising: receiving test image data, processing the test image data using a simulator to generate simulated sensor image data, wherein the simulator simulates a physical capture process of a scene by a sensor; processing the simulated sensor image data according to one or more ISP settings to produce output image data; producing quality metric data associated with the output image data by performing a comparison between the output image data and the test image data; and optimising the one or more ISP settings based on the quality metric data. ### Claim 2 (depends on 1) The method of claim 1, wherein the method comprises: inputting the output image data into a computer vision algorithm; processing the output image data using the computer vision algorithm to generate one or more output labels; and comparing test labels associated with the test image data to the output labels. ### Claim 3 (depends on 2) The method of claim 2, wherein the method comprises processing the output image data using a classification neural network. ### Claim 4 (depends on 1) The method of claim 1, wherein the method comprises optimising a plurality of the ISP settings using at least one of: a single setting at a time; and a group of related settings at a time. ### Claim 5 (depends on 1) The method of claim 1, wherein the method comprises optimising one or more ISP parameters using at least one of: a line search; and a Bayesian optimisation method. ### Claim 6 A method of optimising a simulator, which is to be used to process test image data to generate simulated sensor image data, the method comprising: receiving test image data consistent with a scene; processing the test image data according to one or more simulator settings to produce simulated sensor image data, wherein the simulator simulates a physical capture process of a scene by a sensor; capturing the scene using the sensor to produce sensor image data; performing a comparison between the simulated sensor image data and sensor image data; producing quality metric data associated with the comparison; and optimising one or more simulator settings based on the quality metric data. ### Claim 7 (depends on 6) The method of claim 6, wherein the method comprises optimising a plurality of the simulator settings using at least one of: a single setting at a time; and a group of related settings at a time. ### Claim 8 (depends on 6) The method of claim 6, wherein the method comprises optimising the one or more simulator settings using at least one of: a line search method; and a Bayesian optimisation method. ### Claim 9 (depends on 6) The method of claim 6, wherein the method comprises at least one of: performing an alignment between the simulated sensor image data and sensor image data; or performing the comparison using a cost function, and determining a cost associated with the cost function. ### Claim 10 An image processing system comprising: memory circuitry arranged to store test image data, one or more settings associated with an ISP and output image data; and processing circuitry configured to: receive test image data; process the test image data using a simulator to generate simulated sensor image data, wherein the simulator simulates a physical capture process of a scene by a sensor; process the simulated sensor image data according to one or more ISP settings to produce output image data; produce a quality metric associated with the output image data by performing a comparison between the output image data and the test image data; and optimise one or more settings associated with the ISP based on the quality metric. ### Claim 11 (depends on 10) The system of claim 10, wherein the processing circuitry is arranged to: input the output image data into a computer vision algorithm; process the output image data using the computer vision algorithm to generate one or more output labels; and compare test labels associated with the test image data to the output labels. ### Claim 12 An image processing system comprising: memory circuitry arranged to store test image data, one or more settings associated with a simulator, simulated sensor image data and sensor image data; and processing circuitry configured to: receive test image data consistent with a scene; process the test image data according to one or more simulator settings to produce simulated sensor image data, wherein the simulator simulates a physical capture process of a scene by a sensor; capture the scene using the sensor to produce sensor image data; perform a comparison between the simulated sensor image data and sensor image data; produce quality metric data associated with the comparison; and optimise one or more simulator settings based on the quality metric data. ### Claim 13 (depends on 12) The system of claim 12, wherein the processing circuitry is arranged to optimise at least one of: a plurality of the simulator settings using at least one of a single setting at a time, or a group of related settings at a time; or the one or more simulator settings using at least one of a line search method, or a Bayesian optimisation method. ### Claim 14 (depends on 12) The system of claim 12, wherein the processing circuitry is arranged to at least one of: perform an alignment between the simulated sensor image data and sensor image data; or perform the comparison using a cost function, and determine a cost associated with the cost function. ### Claim 15 A non-transitory computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to: receive test image data, process the test image data using a simulator to generate simulated sensor image data, wherein the simulator simulates a physical capture process of a scene by a sensor; process the simulated sensor image data according to one or more ISP settings to produce output image data; produce quality metric data associated with the output image data by performing a comparison between the output image data and the test image data; and optimise the one or more ISP settings based on the quality metric data. ### Claim 16 A non-transitory computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to: receive test image data consistent with a scene; process the test image data according to one or more simulator settings to produce simulated sensor image data, wherein the simulator simulates a physical capture process of a scene by a sensor; capture the scene using the sensor to produce sensor image data; perform a comparison between the simulated sensor image data and sensor image data; produce quality metric data associated with the comparison; and optimise one or more simulator settings based on the quality metric data.