# US 2019/0171882 A1 · Image processing apparatus, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171882 A1 - Document ID: US20190171882A1 - Status: Published application - Kind: A1 - Application number: 16/083,887 - Filed: 2017-01-06 - Published: 2019-06-06 - Earliest claimed: 2016-03-17 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: HIDEKI OYAIZU; MAKIBI NAKAMURA; ATSUSHI ITO; YUKI ASUKABE; MASAYUKI YOKOYAMA - Web page: https://patentgrep.com/patent/US20190171882A1 ## CPC - G10H 1/0091: {Means for obtaining special acoustic effects (combined with modulation G10H1/043)} - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 20/20: in augmented reality scenes - G10H 1/00: Details of electrophonic musical instruments - G10L 25/51: for comparison or discrimination - H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability} - H04N 9/643: {Hue control means, e.g. flesh tone control} - H04N 9/74: for obtaining special effects (H04N9/65 - H04N9/73 take precedence) - H04N 23/11: for generating image signals from visible and infrared light wavelengths - H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images - H04N 23/60: Control of cameras or camera modules - G10H 2220/455: Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data - G10L 25/57: for processing of video signals - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract State information corresponding to a subject on a visible light image, such as temperature information and sound field information, is acquired from state information indicating a subject state not indicated on the visible light image by an information acquisition unit 21. An effect processing unit 22 performs an effect process on the visible light image on the basis of the state information acquired by the information acquisition unit 21. In the effect process, an effect component image is superimposed or a subject image is modified based on the type and temperature of the subject and an effect component image is superimposed or a subject image is modified based on the type of a sound source and the volume of a sound. The effect process according to the state information is performed on the visible light image, such that the reality can be improved. ## Claims ### Claim 1 An image processing apparatus comprising: an information acquisition unit that acquires state information corresponding to a subject on a visible light image from state information indicating a subject state not indicated on the visible light image; and an effect processing unit that performs an effect process on the visible light image on the basis of the state information acquired by the information acquisition unit. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the state information is temperature information, and the effect processing unit performs an effect process according to temperature. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the information acquisition unit acquires type information on the subject by performing a recognition process for the subject, and the effect processing unit superimposes an effect component image according to a temperature and a type of the subject onto an image of the subject on the basis of the state information and the type information. ### Claim 4 (depends on 3) The image processing apparatus according to claim 3, wherein the effect processing unit uses, as the effect component image, an image schematically representing a visually observable phenomenon that occurs in accordance with temperature. ### Claim 5 (depends on 3) The image processing apparatus according to claim 3, wherein the effect processing unit controls an attribute of the effect component image in accordance with a difference in temperature of the subject. ### Claim 6 (depends on 2) The image processing apparatus according to claim 2, wherein the effect processing unit modifies a subject image on the visible light image into an image in which a physical state of the subject is changed in accordance with a temperature of the subject. ### Claim 7 (depends on 6) The image processing apparatus according to claim 6, wherein the effect processing unit regards a change between a solid phase and a liquid phase as the change in the physical state. ### Claim 8 (depends on 6) The image processing apparatus according to claim 6, wherein the effect processing unit controls a modification level in accordance with a difference in the temperature. ### Claim 9 (depends on 2) The image processing apparatus according to claim 2, wherein the effect processing unit performs color adjustment on the visible light image in accordance with a temperature of the subject. ### Claim 10 (depends on 9) The image processing apparatus according to claim 9, wherein the effect processing unit performs color adjustment to a warm color system when a temperature of the subject is high and performs color adjustment to a cool color system in a case where a temperature of the subject is low. ### Claim 11 (depends on 1) The image processing apparatus according to claim 1, wherein the state information is sound field information, and the effect processing unit performs an effect process according to the sound field information. ### Claim 12 (depends on 11) The image processing apparatus according to claim 11, wherein the effect processing unit superimposes an effect component image onto a subject on the visible light image corresponding to a position of a sound source based on the sound field information. ### Claim 13 (depends on 12) The image processing apparatus according to claim 12, wherein the effect processing unit superimposes an effect component image associated with a sound output from the sound source onto a region of the visible light image. ### Claim 14 (depends on 11) The image processing apparatus according to claim 11, wherein the effect processing unit adjusts an effect amount for the visible light image in accordance with intensity of a sound output from a sound source based on the sound field information. ### Claim 15 An image processing method comprising: acquiring, by an information acquisition unit, state information corresponding to a subject on a visible light image from state information indicating a subject state not indicated on the visible light image; and performing, by an effect processing unit, an effect process on the visible light image on the basis of the state information acquired by the information acquisition unit. ### Claim 16 A program that causes a computer to implement: a function of acquiring state information corresponding to a subject on a visible light image from state information indicating a subject state not indicated on the visible light image; and a function of performing an effect process on the visible light image on the basis of the state information corresponding to the subject on the visible light image.