# US 12,002,226 B2 · Using machine learning to selectively overlay image content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,226 B2 - Document ID: US12002226B2 - Status: Granted - Kind: B2 - Application number: 17/666,747 - Filed: 2022-02-08 - Published: 2022-05-26 - Granted: 2024-06-04 - Earliest claimed: 2019-11-26 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Raymond Kirk Price; Michael Bleyer; Christopher Douglas Edmonds - Web page: https://patentgrep.com/patent/US12002226B2 ## CPC - G06T 7/33: using feature-based methods - G06T 7/593: from stereo images - H04N 13/133: Equalising the characteristics of different image components, e.g. their average brightness or colour balance - H04N 13/156: Mixing image signals - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor - G06T 2207/10012: Stereo images - G06T 2207/10048: Infrared image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20212: Image combination - H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Modifications are performed to cause a style of an image to match a different style. A first image is accessed, where the first image has the first style. A second image is also accessed, where the second image has a second style. Subsequent to a deep neural network (DNN) learning these styles, a copy of the first image is fed as input to the DNN. The DNN modifies the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style. As a consequence, a modified style of the transitioned first image copy bilaterally matches the second style. ## Claims ### Claim 1 A computer system comprising: one or more processors; and one or more computer-readable hardware storage devices having stored thereon computer-executable instructions that are executable by the one or more processors to cause the computer system to at least: access a first image generated by a first camera that generates images having a first style such that the first image has the first style, wherein the first camera is one of a low light camera or a visible light camera, and wherein the first style is one of a low light style or a visible light style; access a second image generated by a second camera that generates images having a second style such that the second image has the second style, wherein the second camera is a thermal imaging camera, and wherein the second style is a thermal data style; subsequent to a deep neural network (DNN) learning the first style and the second style, feed a copy of the first image as input to the DNN; and cause the DNN to modify the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style such that a modified style of the transitioned first image copy bilaterally matches the second style. ### Claim 2 (depends on 1) The computer system of claim 1, wherein execution of the instructions further causes the computer system to: overlay selected portions of the transitioned first image copy onto the first image to generate a composite image; and display the composite image. ### Claim 3 (depends on 1) The computer system of claim 1, wherein the first camera is the visible light camera, and wherein the first style is the visible light style. ### Claim 4 (depends on 1) The computer system of claim 1, wherein the first camera is the low light camera, and wherein the first style is the low light style. ### Claim 5 (depends on 1) The computer system of claim 1, wherein the computer system includes a plurality of visible light cameras, and wherein the computer system includes a single low light camera, which is said low light camera. ### Claim 6 (depends on 1) The computer system of claim 1, wherein causing the DNN to modify the first image copy includes causing the DNN to modify one or more of a geometry, a texture, an outline, a content, one or more feature points, or an editing of the first image copy. ### Claim 7 (depends on 1) The computer system of claim 1, wherein the modified style of the transitioned first image copy bilaterally matching the second style occurs by the modified style being within a threshold level of similarity relative to the second style. ### Claim 8 (depends on 1) The computer system of claim 1, wherein a thermal imager analyzes the second image to identify thermal image data. ### Claim 9 (depends on 8) The computer system of claim 8, wherein the thermal image data includes hot or cold areas included in the second image. ### Claim 10 (depends on 1) The computer system of claim 1, wherein the first camera is the low light camera, and wherein the low light camera detects light spanning a range of illuminance between about 1 milli-lux and about 10 lux. ### Claim 11 A method performed by a computer system to modify a style of an image so the style subsequently corresponds to a different style, said method comprising: accessing a first image generated by a first camera that generates images having a first style such that the first image is of the first style, wherein the first camera is one of a low light camera or a visible light camera, and wherein the first style is one of a low light style or a visible light style; accessing a second image generated by a second camera that generates images having a second style such that the second image is of the second style, wherein the second camera is a thermal imaging camera, and wherein the second style is a thermal data style; subsequent to a deep neural network (DNN) learning the first style and the second style, feeding a copy of the first image as input to the DNN; causing the DNN to modify the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style such that a modified style of the transitioned first image copy bilaterally matches the second style; and displaying one or more portions of the first image or the transitioned first image copy on a display. ### Claim 12 (depends on 11) The method of claim 11, wherein the method further includes: overlaying selected portions of the transitioned first image copy onto the first image to generate a composite image; and displaying the composite image. ### Claim 13 (depends on 12) The method of claim 12, wherein the first style is the visible light style, and wherein the transitioned first image copy includes thermal data such that at least some of the thermal data is overlaid onto the first image and such that the composite image includes visible light data and the at least some of the thermal data. ### Claim 14 (depends on 11) The method of claim 11, wherein the first camera is the low light camera such that the first image is a low light image and such that the first style is the low light style. ### Claim 15 (depends on 14) The method of claim 14, wherein the first image includes low light data, and wherein, as a result of the transition, the transitioned first image copy includes thermal data. ### Claim 16 (depends on 15) The method of claim 15, wherein at least some of the thermal data is overlaid onto the first image to generate a composite image comprising the low light data and the at least some of the thermal data. ### Claim 17 A method performed by a computer system to modify a style of an image so the style subsequently corresponds to a different style, said method comprising: accessing a first image generated by a first camera that generates images having a first style such that the first image is of the first style, wherein the first camera is one of a low light camera or a visible light camera, and wherein the first style is one of a low light style or a visible light style; accessing a second image generated by a second camera that generates images having a second style such that the second image is of the second style, wherein the second camera is a thermal imaging camera, and wherein the second style is a thermal data style; subsequent to a deep neural network (DNN) learning the first style and the second style, feeding a copy of the first image as input to the DNN; causing the DNN to modify the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style such that a modified style of the transitioned first image copy bilaterally matches the second style; overlaying selected portions of the transitioned first image copy onto the first image to generate a composite image; and displaying the composite image. ### Claim 18 (depends on 17) The method of claim 17, wherein the first camera is the visible light camera, and wherein the visible light camera detects light spanning a range of illuminance from about 10 lux to about 100,000 lux. ### Claim 19 (depends on 17) The method of claim 17, wherein the DNN tunes an initial set of learning data. ### Claim 20 (depends on 17) The method of claim 17, wherein corresponding feature points are identified in the transitioned first image copy and the first image.