# US 2019/0172238 A1 · Intelligent camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172238 A1
- Document ID: US20190172238A1
- Status: Published application
- Kind: A1
- Application number: 16/265,871
- Filed: 2019-02-01
- Published: 2019-06-06
- Earliest claimed: 2017-11-01
- Assignees: Essential Products, Inc.
- Applicants: Essential Products, Inc.
- Inventors: Xiaoyu Miao; Andrew E. Rubin
- Web page: https://patentgrep.com/patent/US20190172238A1

## CPC

- G06V 10/82: using neural networks
- G06F 18/24: Classification techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 11/60: Creating or editing images; Combining images with text
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Presented here is technology to efficiently process camera images to generate lead part artistic images and videos using an artificial intelligence module receiving inputs from multiple sensors. Multiple sensors can include a depth sensor, a conventional camera, and a motion tracker providing inputs to the artificial intelligence module. Based on the inputs, the artificial intelligence module can segment the received image and/or video into a foreground image and a background image to produce portrait imagery by blurring the background image and/or video. The artificial intelligence module can select the most aesthetically pleasing image from a video. In addition, the artificial intelligence module can adjust lighting in an image or video to create artistic lighting effects. All the processing can be done in real time due to efficient combination of artificial intelligence modules, traditional image processing techniques, and use of specialized hardware.

## Claims

### Claim 1

A method comprising:
receiving an image from a camera and a depth information associated with the image;
identifying an entity in the image and a shadow associated with the entity in the image;
determining a quality of a lighting of the image based on the entity and the shadow associated with the entity in the image;
when the quality of the lighting of the image is low, adjusting the shadow associated with the entity in the image, said adjusting comprising:
constructing a depth representation of the image having a depth based on the depth information;
based on the entity obtaining a surfacing software associated with the entity, wherein the surfacing software describes interaction of light with a surface of the entity;
creating a light source illuminating the depth representation of the image and producing light interacting with the surfacing software; and
producing a new image based on the depth representation of the image, the surfacing software and the light source interacting with the depth representation of the image and the surfacing software.

### Claim 2

A method comprising:
receiving an image from an imaging device;
identifying an entity in the image and a shadow associated with the entity in the image;
determining a quality of a lighting of the image based on the entity and the shadow associated with the entity in the image; and
when the quality of the lighting of the image is low, adjusting the shadow associated with the entity in the image.

### Claim 3 (depends on 2)

The method of claim 2, said determining the quality of the image comprising:
determining whether the shadow associated with the entity is sharp; and
when the shadow associated with the entity is sharp, creating a transition region between a non-shadow region and the shadow associated with the entity.

### Claim 4 (depends on 3)

The method of claim 3, said determining whether the shadow associated with the entity is sharp comprising:
determining a size of a darkest region of the shadow and a size of a lighter region of the shadow;
when the size of the lighter region of the shadow compared to the size of the darkest region of the shadow is below a predetermined threshold, determining that the shadow associated with the entity is sharp.

### Claim 5 (depends on 2)

The method of claim 2, said determining the quality of the image comprising:
determining whether the shadow associated with the entity indicates a shape of the entity; and
when the shadow associated with the entity does not indicate that the shape of the entity, adjusting the shadow associated with the image.

### Claim 6 (depends on 2)

The method of claim 2, said adjusting the shadow associated with the entity in the image comprising:
obtaining a depth information associated with the image;
constructing a depth representation of the image having a depth based on the depth information;
based on the entity obtaining a surfacing software associated with the entity, wherein the surfacing software describes interaction of light with a surface of the entity;
creating a light source illuminating the depth representation of the image and producing light interacting with the surfacing software; and
producing a new image based on the depth representation of the image, the surfacing software and the light source interacting with the depth representation of the image and the surfacing software.

### Claim 7 (depends on 6)

The method of claim 6, said creating the light source comprising:
determining a second entity within the image proximate to a visual focus of the image;
creating a plurality of lights surrounding the second entity.

### Claim 8 (depends on 6)

The method of claim 6, said creating the light source comprising:
obtaining a type of lighting to apply to the image;
obtaining the light source associated with the type of lighting and a position of the light source; and
creating the light source at the obtained position.

### Claim 9 (depends on 8)

The method of claim 8, said obtaining the type of lighting comprising:
automatically determining the type of lighting based on the entity contained in the image, a location of the imaging device, or a time of day when the image was recorded.

### Claim 10 (depends on 2)

The method of claim 2, wherein the image comprises a time-dependent visual representation, the method comprising:
performing a low-pass filter of time varying attributes of the time-dependent visual representation.

### Claim 11

A system comprising:
an imaging device to record a visual representation of a scene;
a processor configured to:
receive the visual representation from the imaging device;
identify an entity in the visual representation and a shadow associated with the entity in the visual representation;
determine whether a quality of a lighting of the visual representation is satisfactory based on the entity and the shadow associated with the entity in the visual representation; and
when the quality of the lighting of the visual representation is not satisfactory, adjust the shadow associated with the entity in the visual representation.

### Claim 12 (depends on 11)

The system of claim 11, the processor configured to:
determine whether the shadow associated with the entity is sharp; and
when the shadow associated with the entity is sharp, create a transition region between a non-shadow region and the shadow associated with the entity.

### Claim 13 (depends on 12)

The system of claim 12, the processor configured to determine whether the shadow associated with the entity is sharp comprising the processor configured to:
determine a size of a darkest region of the shadow and a size of a lighter region of the shadow;
when the size of the lighter region of the shadow compared to the size of the darkest region of the shadow is below a predetermined threshold, determine that the shadow associated with the entity is sharp.

### Claim 14 (depends on 11)

The system of claim 11, the processor configured to determine the quality of the visual representation comprising the processor configured to:
determine whether the shadow associated with the entity indicates a shape of the entity; and
adjust the shadow associated with the visual representation.

### Claim 15 (depends on 11)

The system of claim 11, the processor configured to adjust the shadow associated with the entity in the visual representation comprising the processor configured to:
obtain a depth information associated with the visual representation;
construct a depth representation of the visual representation having a depth based on the depth information;
based on the entity obtain a surfacing software associated with the entity, wherein the surfacing software describes interaction of light with a surface of the entity;
create a light source illuminating the depth representation of the visual representation and producing light interacting with the surfacing software; and
produce a new visual representation based on the depth representation of the visual representation, the surfacing software and the light source interacting with the depth representation of the visual representation and the surfacing software.

### Claim 16 (depends on 15)

The system of claim 15, the processor configured to create the light source comprising the processor configured to:
determine a second entity within the visual representation proximate to a visual focus of the visual representation; and
create a plurality of lights surrounding the second entity.

### Claim 17 (depends on 15)

The system of claim 15, the processor configured to create the light source comprising the processor configured to:
obtain a type of lighting to apply to the visual representation;
obtain the light source associated with the type of lighting and a position of the light source; and
create the light source at the obtained position.

### Claim 18 (depends on 17)

The system of claim 17, the processor configured to obtain the type of lighting comprising the processor configured to:
automatically determine the type of lighting based on the entity contained in the visual representation, a location of the imaging device, or a time of day when the visual representation was recorded.

### Claim 19 (depends on 11)

The system of claim 11, comprising the processor configured to:
receive a depth information associated with the visual representation;
based on the depth information in the visual representation, identify the entity in the visual representation.

### Claim 20 (depends on 11)

The system of claim 11, wherein the visual representation comprises a time-dependent visual representation, the processor configured to:
perform a low-pass filter of a time varying attribute of the time-dependent visual representation.
