# US 2012/0249836 A1 · Method and apparatus for performing user inspired visual effects rendering on an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0249836 A1
- Document ID: US20120249836A1
- Status: Published application
- Kind: A1
- Application number: 13/073,772
- Filed: 2011-03-28
- Published: 2012-10-04
- Earliest claimed: 2011-03-28
- Assignees: Sony Corporation
- Inventors: GAZI ALI; ALEXANDER BERESTOV; CHUEN-CHIEN LEE
- Web page: https://patentgrep.com/patent/US20120249836A1

## CPC

- H04N 5/2621: {Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability}
- G06T 5/73: Deblurring; Sharpening
- H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Method and apparatus for performing visual effects rendering on an image are described. User specifications are processed for at least one visual effect for the image. Scene analysis information associated with the image may be accessed, wherein the scene analysis information is based on the user specifications and indicates at least one pixel of the image for coupling with the at least one visual effect. The at least one pixel is accentuated using depth map information, wherein dimensions for the at least one pixel are determined in response to corresponding depth values of the at least one pixel. An output image is generated having at least one visual effect.

## Claims

### Claim 1

A method of performing visual effect rendering on an image, comprising:
processing user specifications for at least one visual effect for the image;
accentuating the at least one pixel using depth map information, wherein characteristic for the at least one pixel are determined in response to corresponding depth values of the at least one pixel; and
generating an output image having at least one visual effect.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the output image further comprising:
creating a mask for a user defined aperture shape; and
creating a point spread function for the at least one pixel based on the binary mask.

### Claim 3 (depends on 2)

The method of claim 2, wherein a size of the mask depends on the corresponding depth values of the at least one pixel.

### Claim 4 (depends on 1)

The method of claim 1, wherein the at least one pixel forms at least one region of interest.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the output image further comprises creating a filter having at least one convolution window size that is related to the corresponding depth values of the at least one pixel.

### Claim 6 (depends on 1)

The method of claim 1, wherein accentuating the at least one pixel using the depth map information further comprises applying at least one image enhancement technique to the at least one pixel.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
identifying a region of interest within the image based on color information;
creating a sparse representation of the region of interest; and
selecting at least one area of the sparse representation of the region of the interest using the depth map information.

### Claim 8 (depends on 7)

The method of claim 7 further comprising:
superimposing a user defined shape on the at least one selected area to produce the output image; and
blurring the output image using the depth map information.

### Claim 9 (depends on 7)

The method of claim 7, wherein processing the filter for the particular visual effect further comprises creating a hollow mask based on a user defined shape for convolving with the region of interest;

### Claim 10 (depends on 9)

The method of claim 9, wherein creating the hollow mask further comprising:
applying a solid mask on the region of interest to detect at least one edge; and
dilating the masked region of interest with respect at least one detected edge.

### Claim 11 (depends on 7)

The method of claim 7 further comprising:
generating a color scheme based on the scene analysis information;
selecting color patterns for successive video frames; and
applying the selected color patterns to the region of interest.

### Claim 12 (depends on 1)

The method of claim 1, wherein the scene analysis information is based on the user specifications and indicates at least one pixel of the image for coupling with the at least one visual effect;

### Claim 13 (depends on 7)

The method of claim 7 further comprising:
generating a color scheme based on the scene analysis information;
replacing the region of interest with a virtual object; and
blending the virtual object into the image.

### Claim 14 (depends on 13)

The method of claim 13, wherein blending the virtual object further comprises modifying a size of the virtual object based on the depth map information.

### Claim 15 (depends on 1)

The method of claim 1 further comprising:
creating a lighting model based on the depth map information;
determining at least one lighting direction from at least one light source based on the scene analysis information; and
relighting the image using the lighting model and the at least one lighting direction.

### Claim 16 (depends on 15)

The method of claim 15, wherein relighting the image further comprises modifying the at least one pixel using the lighting model and the at least one lighting direction.

### Claim 17

An apparatus for performing visual effect rendering on an image, comprising:
means for processing user specifications for at least one visual effect for the image;
means for accentuating the at least one pixel using depth map information, wherein dimensions for the at least one pixel are determined in response to corresponding depth values of the at least one pixel; and
means for generating an output image having at least one visual effect.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the means for generating the output image further comprises:
means for creating a mask for a user defined aperture shape; and
means for creating a point spread function for the at least one pixel based on the binary mask.

### Claim 19 (depends on 17)

The apparatus of claim 17, further comprising:
means for identifying a region of interest within the image based on color information;
means for creating a sparse representation of the region of interest; and means for selecting at least one area of the sparse representation of the region of the interest using the depth map information;
wherein the scene analysis information is based on the user specifications and indicates at least one pixel of the image for coupling with the at least one visual effect.

### Claim 20

An apparatus, comprising:
a lens module configured to obtain an image; and
a depth map generator configured to:
process user specifications for at least one visual effect for the image;
accentuate the at least one pixel using depth map information, wherein characteristic for the at least one pixel are determined in response to corresponding depth values of the at least one pixel; and
generate an output image having at least one visual effect.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein the depth map generator is further configured to:
identify a region of interest within the image based on color information;
create a sparse representation of the region of interest; and
select at least one area of the sparse representation of the region of the interest using the depth map information.
