# US 2018/0027222 A1 · Image pickup apparatus, information processing system, mat, and image generation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0027222 A1
- Document ID: US20180027222A1
- Status: Published application
- Kind: A1
- Application number: 15/547,707
- Filed: 2016-01-29
- Published: 2018-01-25
- Earliest claimed: 2015-02-06
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Hidehiko Ogasawara; Norio Nagatsuka; Naoki Numaguchi; Hiroyuki Segawa; Akio Ohba
- Web page: https://patentgrep.com/patent/US20180027222A1

## CPC

- A63F 13/25: Output arrangements for video game devices
- A63F 13/32: using local area network [LAN] connections
- A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
- A63F 13/65: automatically by game devices or servers from real world data, e.g. measurement in live racing competition
- A63H 33/22: Optical, colour, or shadow toys ({puzzles with optical elements A63F9/0613}; kaleidoscopes G02B27/08)
- G03B 15/00: Special procedures for taking photographs; Apparatus therefor
- G06V 10/56: relating to colour
- G06V 20/64: Three-dimensional [3D] objects
- H04N 1/4076: {dependent on references outside the picture}
- H04N 1/4078: {using gradational references, e.g. grey-scale test pattern analysis}
- H04N 9/77: Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase (circuits for matrixing H04N9/67)
- H04N 17/002: {for television cameras}
- H04N 17/02: for colour television signals
- H04N 23/10: for generating image signals from different wavelengths
- H04N 23/12: with one sensor only
- H04N 23/70: Circuitry for compensating brightness variation in the scene
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04N 23/83: specially adapted for colour signals
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63H 33/04: Building blocks, strips, or similar building parts
- A63H 2200/00: Computerized interactive toys, e.g. dolls
- H04N 23/958: for extended depth of field imaging
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63H: TOYS, e.g. TOPS, DOLLS, HOOPS OR BUILDING BLOCKS
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image of a block or the like entering a space above a play mat is picked up by an image pickup apparatus, and an information processing apparatus detects and recognizes the block on the basis of a picture of the block in a picked up image and performs information processing. The play mat includes a play field that defines a detection space for the block, and a calibration chart including at least a plurality of regions of colors of different luminances. The image pickup apparatus adjusts an exposure time period, a gain value for each color component and a correction rule upon gamma correction on the basis of the picture of the calibration chart in the picked up image.

## Claims

### Claim 1

An image pickup apparatus that outputs data of a picked up image to be used for recognition of a real object, comprising:
an output image production unit configured to produce data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element;
an adjustment processing unit configured to evaluate the picked up image and adjust at least one of an image pickup condition in the image pickup element and a processing condition in the output image production unit; and
a data sending out unit configured to output the data of the picked up image to an apparatus that recognizes the real object, wherein
the adjustment processing unit adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other.

### Claim 2 (depends on 1)

The image pickup apparatus according to claim 1, wherein the adjustment processing unit evaluates the luminance of a picture of the color regions of the calibration chart and adjusts the processing condition so as to expand the difference between a plurality of luminances corresponding to the luminance of the color of the real object.

### Claim 3 (depends on 2)

The image pickup apparatus according to claim 2, wherein the adjustment processing unit determines a luminance range within which the difference is to be expanded based on a luminance distribution of the picture of the real object in the picked up image.

### Claim 4 (depends on 1)

The image pickup apparatus according to claim 1, wherein the adjustment processing unit evaluates a color of a picture of a color region having a color same as a color of the real object in the calibration chart and adjusts the processing condition so as to expand a distance between a plurality of colors of different pictures in a color space.

### Claim 5 (depends on 2)

The image pickup apparatus according to claim 2, wherein the adjustment processing unit adjusts, as the adjustment of the processing condition, a correction rule upon gamma correction.

### Claim 6 (depends on 1)

The image pickup apparatus according to claim 1, wherein the adjustment processing unit adjusts a gain value of each color component such that a picture of a color region in the calibration chart, which has a color same as one of chromatic colors of the real object, has an achromatic color.

### Claim 7 (depends on 1)

The image pickup apparatus according to claim 1, wherein the adjustment processing unit evaluates a luminance of a picture of a color region in the calibration chart, which has a given luminance, and adjusts the image pickup condition and the processing condition in a direction in which the luminance is adjusted to a target luminance determined in advance.

### Claim 8 (depends on 1)

The image pickup apparatus according to claim 1, wherein the adjustment processing unit adjusts a white balance to a picture of a color region of an achromatic color in the calibration chart.

### Claim 9 (depends on 1)

The image pickup apparatus according to claim 1, wherein the data sending out unit parallelly outputs data of a picked up image produced in accordance with a processing condition adjusted by the adjustment processing unit and data of a picked up image produced in accordance with a processing condition that omits at least part of the adjustment by the adjustment processing unit.

### Claim 10 (depends on 1)

The image pickup apparatus according to claim 1, wherein the data sending out unit switches and outputs data of a picked up image produced in accordance with a processing condition adjusted by the adjustment processing unit and data of a picked up image produced in accordance with a processing condition that omits at least part of the adjustment by the adjustment processing unit in accordance with a request from an apparatus that recognizes the real object.

### Claim 11

An information processing system including an image pickup apparatus that outputs data of a picked up image obtained by image pickup of a real object and an information processing apparatus that recognizes the real object using the picked up image, wherein
the image pickup apparatus includes
an output image production unit configured to produce data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element,
an adjustment processing unit configured to evaluate the picked up image and adjust at least one of an image pickup condition in the image pickup element and a processing condition in the output image production unit, and
a data sending out unit configured to output the data of the picked up image to the information processing apparatus, and
the adjustment processing unit adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other, and
the information processing apparatus
recognizes a block by comparison between color information of the picture of the real object in the picked up image and color information of real objects registered already with each other and performs a process set for the block.

### Claim 12

A mat that is placed in an image pickup space in an information processing system that recognizes a real object based on color information of a picture in a picked up image, comprising:
a play field that defines a bottom region of a space in which, when the real object is placed into the space, the real object is determined as a recognition target; and
a calibration chart configured from a plurality of color regions having different luminances from each other and used by an image pickup apparatus to adjust, based on color information of a picture in the picked up image, at least one of an image pickup condition and a processing condition when data of the picked up image to be outputted is produced.

### Claim 13 (depends on 12)

The mat according to claim 12, wherein the calibration chart includes a color region having a luminance corresponding to a supposed color of the real object of the recognition target.

### Claim 14 (depends on 12)

The mat according to claim 12, wherein
the mat includes a first mat that includes the play field and a second mat that includes the calibration chart, and
the second mat is placed and removably mounted on the first mat.

### Claim 15

An image production method, by an image pickup apparatus, of producing data of a picked up image to be used for recognition of a real object, comprising:
producing data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element;
evaluating the picked up image and adjusting at least one of an image pickup condition in the image pickup element and a processing condition in the producing; and
outputting the data of the picked up image to an apparatus that recognizes the real object, wherein
the adjusting adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other.

### Claim 16

A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to carry out actions to implement a function for producing data of a picked up image to be used for recognition of a real object, the computer program causing the computer to carry out the actions of:
producing data of a picked up image to be outputted from data of a non-processed image picked up using an image pickup element;
evaluating the picked up image and adjusting at least one of an image pickup condition in the image pickup element and a processing condition in the production function; and
outputting the data of the picked up image to an apparatus that recognizes the real object, wherein
the adjusting adjusts the image pickup condition and the processing condition based on color information of a picture of a calibration chart that is provided in a space of a recognition target and is configured from a plurality of color regions having luminances different from each other.

### Claim 17

(canceled)
