# US 10,872,395 B2 · Image processing device, imaging system provided therewith, and calibration method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,395 B2
- Document ID: US10872395B2
- Status: Granted
- Kind: B2
- Application number: 16/088,697
- Filed: 2017-04-06
- Published: 2019-04-18
- Granted: 2020-12-22
- Earliest claimed: 2016-04-25
- Assignees: Panasonic Intellectual Property Management Co., Ltd.
- Applicants: Panasonic Intellectual Property Management Co., Ltd.
- Inventors: Ryoji Ogino; Junya Kuwada
- Web page: https://patentgrep.com/patent/US10872395B2

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G03B 15/00: Special procedures for taking photographs; Apparatus therefor
- G03B 37/04: with cameras or projectors providing touching or overlapping fields of view
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0412: {Digitisers structurally integrated in a display}
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06T 7/11: Region-based segmentation
- G06V 10/235: {based on user input or interaction}
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- H04N 5/2625: {for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect (sequence generated by event triggered capturing H04N7/188)}
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 13/128: Adjusting depth or disparity
- H04N 13/246: Calibration of cameras
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- H04N 23/60: Control of cameras or camera modules
- H04N 23/62: Control of parameters via user interfaces
- H04N 23/63: by using electronic viewfinders
- H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters}
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06V 10/16: {using multiple overlapping images; Image stitching}
- H04N 23/632: {for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters}
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing device includes a stitching processing unit that composites composition source images generated from captured images under a preset processing condition to generate a composite image, a screen generation unit that generates a screen and outputs the composite image to a display input device, a touch operation determination unit that determines a camera image region to be adjusted based on a detection result of a touch operation on the screen and determines an adjustment item according to an operation mode of the touch operation, and a processing condition setting unit that sets a temporary processing condition related to the camera image region to be adjusted. The stitching processing unit generates the composition source image from the captured image of the camera corresponding to the camera image region to be adjusted under the temporary processing condition and updates the composite image by compositing the composition source images.

## Claims

### Claim 1

An image processing device that outputs a composite image by performing an image composition process on an image for each of a plurality of cameras, the image processing device comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations, the operations including:
generating a plurality of composition source images from the image for each of the plurality of cameras based on a preset processing condition;
performing the image composition process on the plurality of composition source images to generate a composite image for preview, the composite image including a plurality of camera image regions, each of the plurality of camera image regions corresponding to one of the plurality of composition source images and one of the plurality of cameras;
generating a preview display screen on which the composite image is displayed and outputting the preview display screen to a display;
detecting a first touch operation that a user performs on the preview display screen displayed on the display and determining a first adjustment item according to the first touch operation, the first touch operation being performed with one or two fingers and in relation to a first camera image region of the plurality of camera image regions, the first camera image region corresponding to a first composition source image of the plurality of composition source images and a first camera of the plurality of cameras;
first adjusting at least one of the first composition source image or the first camera according to the first adjustment item and updating the composite image;
detecting a second touch operation that the user performs on the preview display screen displayed on the display and determining a second adjustment item, the second touch operation being performed with at least three fingers; and
second adjusting at least one of all composition source images of the plurality of composition source images or all cameras of the plurality of cameras according to the second adjustment item and updating the composite image.

### Claim 2 (depends on 1)

The image processing device of claim 1,
wherein the operations further include:
detecting a third touch operation that the user performs on the preview display screen displayed on the display and determining a third adjustment item, the third touch operation being performed with two fingers and in relation to two camera image regions; and
in a case where the third touch operation is performed, performing an image adjustment between the two camera image regions according to the third adjustment item.

### Claim 3 (depends on 1)

The image processing device of claim 1,
wherein the operations further include:
in a case where the first touch operation is performed, setting the first camera image region on which the first touch operation is performed as an adjustment target and determining the first adjustment item according to the first touch operation.

### Claim 4 (depends on 3)

The image processing device of claim 3,
wherein the first adjusting includes:
performing a pan adjustment in a case where the first touch operation is a swipe operation in a horizontal direction with one finger, performing a tilt adjustment in a case where the first touch operation is a swipe operation in a vertical direction with one finger, performing a rolling adjustment in a case where the first touch operation is a rotation operation with two fingers, and performing a zoom adjustment in a case where the first touch operation is a pinch operation with two fingers.

### Claim 5 (depends on 1)

The image processing device of claim 1,
wherein the operations further include:
performing image processing of changing a tone of at least one of the plurality of camera image regions not to be adjusted in the composite image from an initial tone.

### Claim 6 (depends on 5)

The image processing device of claim 5,
wherein the operations further include:
performing image processing of changing at least luminance on the at least one of the plurality of camera image regions not to be adjusted.

### Claim 7 (depends on 1)

The image processing device of claim 1,
wherein the operations further include:
in a case where the second touch operation is performed with at least three fingers, setting all of the plurality of camera image regions on which the second touch operation is performed as adjustment targets and determining the second adjustment item according to the second touch operation.

### Claim 8 (depends on 1)

The image processing device of claim 1,
wherein the second adjusting includes:
performing a pan adjustment in a case where the second touch operation is a swipe operation in a horizontal direction with at least three fingers, performing a tilt adjustment in a case where the second touch operation is a swipe operation in a vertical direction with at least three fingers, performing a rolling adjustment in a case where the second touch operation is a rotation operation with at least three fingers, performing a zoom adjustment in a case where the second touch operation is a pinch operation with at least three fingers.

### Claim 9 (depends on 1)

The image processing device of claim 1,
wherein the operations further include:
detecting a just fit state in which consistency of a stitching portion between the first camera image region to be adjusted and a second camera image region adjacent to the first camera image region to be adjusted in the composite image is high, and
in a case of detecting that the just fit state is high, outputting the composite image within a predetermined range at a timing at which the just fit state is detected.

### Claim 10 (depends on 1)

The image processing device of claim 1, further
wherein the operations further include:
detecting a third touch operation that the user performs on the preview display screen displayed on the display and determining a third adjustment item, the third touch operation being performed with two fingers and in simultaneous relationship to two camera image regions; and
in a case where the third touch operation is performed, performing an image adjustment so that at least either tones or brightness in the two camera image regions is matched.

### Claim 11 (depends on 1)

The image processing device of claim 1,
wherein the operations further include:
performing a disparity correction process on the plurality of composition source images before the image composition process; and
after the preview display screen is displayed, the disparity correction process is not performed on the plurality of composition source images.

### Claim 12 (depends on 1)

An imaging system comprising:
the image processing device according to claim 1;
the plurality of cameras; and
the display.

### Claim 13

A calibration method of securing consistency between camera image regions in a composite image generated by performing an image composition process on an image for each of a plurality of cameras, the calibration method comprising:
generating a plurality of composition source images from the image for each of the plurality of cameras based on a preset processing condition and performing the image composition process on the plurality of composition source images to generate a composite image for preview, the composite image including a plurality of camera image regions, each of the plurality of camera image regions corresponding to one of the plurality of composition source images and one of the plurality of cameras;
generating a preview display screen on which the composite image is displayed and outputting the preview display screen to a display;
detecting a first touch operation which a user performs on the preview display screen displayed on the display and determining a first adjustment item according to the first touch operation, the first touch operation being performed with one or two fingers and in relation to a first camera image region of the plurality of camera image regions, the first camera image region corresponding to a first composition source image of the plurality of composition source images and a first camera of the plurality of cameras;
adjusting at least one of the first composition source image or the first camera according to the first adjustment item and updating the composite image;
detecting a second touch operation that the user performs on the preview display screen displayed on the display and determining a second adjustment item, the second touch operation being performed with at least three fingers; and
adjusting at least one of all composition source images of the plurality of composition source images or all cameras of the plurality of cameras according to the second adjustment item and updating the composite image.

### Claim 14 (depends on 1)

The image processing device of claim 1, wherein, in the second adjusting, all cameras of the plurality of cameras are adjusted according to the second adjustment item.