# US 12,003,899 B2 · Information processing apparatus and information processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,003,899 B2
- Document ID: US12003899B2
- Status: Granted
- Kind: B2
- Application number: 18/001,119
- Filed: 2021-05-07
- Published: 2023-07-06
- Granted: 2024-06-04
- Earliest claimed: 2020-06-17
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Yu Aoki; Takuya Ikeda; Honoka Ozaki
- Web page: https://patentgrep.com/patent/US12003899B2

## CPC

- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- H04N 9/3188: {Scale or resolution adjustment (scaling in general G06T3/40; resolution modifying circuits for displays in general G09G5/391)}
- G06F 3/0325: {using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image}
- G06F 3/03542: {Light pens for emitting or receiving light}
- G06F 3/03545: {Pens or stylus}
- G06F 3/0386: {for light pen}
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G09G 5/00: Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T)
- G09G 5/02: characterised by the way in which colour is displayed {(details of colour display specific for CRTs G09G1/28; specific for flat matrix panels other than liquid crystal displays G09G3/2003; specific for liquid crystal displays G09G3/3607)}
- G09G 5/10: Intensity circuits
- G09G 5/377: for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence)
- H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)}
- H04N 9/3194: {including sensor feedback}
- G06T 2207/10012: Stereo images
- G06T 2207/10048: Infrared image
- G06T 2207/30204: Marker
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

Provided is an information processing apparatus that includes a transformation matrix calculation section, a scale calculation section, and an input display generation section. The transformation matrix calculation section calculates a transformation matrix that converts the coordinates of an input position of an input device within a projection region projected from a display apparatus from the coordinates of an image coordinate system of a sensor apparatus having detected the input position to the coordinates of a screen coordinate system of the display apparatus. The scale calculation section calculates a scale correction value for a display mode of the locus of the input position in reference to distance information regarding the distance between the display apparatus and the input position. The input display generation section generates an input image depicting the locus of the input position through the use of the transformation matrix and the scale correction value.

## Claims

### Claim 1

An information processing apparatus, comprising:
a transformation matrix calculation section configured to calculate a transformation matrix for converting coordinates of an input position of an input device within a projection region projected from a display apparatus from coordinates of an image coordinate system of a sensor apparatus having detected the input position to coordinates of a screen coordinate system of the display apparatus;
a scale calculation section configured to calculate a scale correction value for a display mode of a locus of the input position in reference to distance information regarding a distance between the display apparatus and the input position; and
an input display generation section configured to generate an input image depicting the locus of the input position through use of the transformation matrix and the scale correction value.

### Claim 2 (depends on 1)

The information processing apparatus according to claim 1, further comprising:
a content display generation section configured to generate a content image to be displayed in the projection region together with the input image by the display apparatus; and
a control section configured to combine the input image with the content image.

### Claim 3 (depends on 1)

The information processing apparatus according to claim 1, wherein the scale correction value is a correction value for thickness, lightness, or saturation of the locus or for contrast of a contour.

### Claim 4 (depends on 3)

The information processing apparatus according to claim 3, wherein a lower-limit value of the thickness is preset according to the distance.

### Claim 5 (depends on 1)

The information processing apparatus according to claim 1, further comprising a reference point display generation section configured to generate a reference point image depicting at least four reference points displayed in the projection region by the display apparatus, wherein
the input device is a pen device that displays the input position with infrared light,
the sensor apparatus is an infrared stereo camera that is mounted on the display apparatus to detect the input position and the distance,
the transformation matrix calculation section is further configured to calculate the transformation matrix according to the reference point image and an infrared image that is detected by the infrared stereo camera and depicted by the infrared light whose reference points are outputted from the pen device, and
the scale calculation section is further configured to calculate the scale correction value for each of the reference points.

### Claim 6 (depends on 5)

The information processing apparatus according to claim 5, wherein the scale calculation section is further configured to:
perform linear interpolation to calculate a distance from the display apparatus to a pixel between the reference points within the reference point image; and
calculate the scale correction value for the pixel according to the calculated distance.

### Claim 7 (depends on 5)

The information processing apparatus according to claim 5, further comprising an input coordinate detection section configured to:
extract a drawing region of the infrared light from the infrared image; and
detect a centroid of the drawing region as position coordinates of the reference points.

### Claim 8 (depends on 1)

The information processing apparatus according to claim 1, wherein, after each change in a projection direction of the display apparatus,
the transformation matrix calculation section is further configured to
calculate the transformation matrix, and the scale calculation section is further configured to calculate the scale correction value.

### Claim 9

An information processing method, comprising:
calculating a transformation matrix that converts coordinates of an input position of an input device within a projection region projected from a display apparatus from coordinates of an image coordinate system of a sensor apparatus having detected the input position to coordinates of a screen coordinate system of the display apparatus;
calculating a scale correction value for a display mode of a locus of the input position in reference to distance information regarding a distance between the display apparatus and the input position; and
generating an input image depicting the locus of the input position through use of the transformation matrix and the scale correction value.

### Claim 10

A non-transitory computer-readable medium having stored thereon, computer-readable instructions which, when executed by a computer communicatively connected to each of an input device, a display apparatus, and a sensor apparatus, cause the computer to execute operations, the operations comprising:
calculating a transformation matrix that converts coordinates of an input position of the input device within a projection region projected from the display apparatus from coordinates of an image coordinate system of the sensor apparatus having detected the input position to coordinates of a screen coordinate system of the display apparatus;
calculating a scale correction value for a display mode of a locus of the input position in reference to distance information regarding a distance between the display apparatus and the input position; and
generating an input image depicting the locus of the input position through use of the transformation matrix and the scale correction value.