# US 2025/0085744 A1 · Wearable image pickup apparatus, portable device and calibrator that communicate with image pickup apparatus, control methods therefor, and storage media storing control programs therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0085744 A1
- Document ID: US20250085744A1
- Status: Published application
- Kind: A1
- Application number: 18/957,533
- Filed: 2024-11-22
- Published: 2025-03-13
- Earliest claimed: 2021-03-12
- Assignees: Canon Kabushiki Kaisha
- Applicants: Canon Kabushiki Kaisha
- Inventors: Nobuyuki Furukawa; Takuya Imaizumi; Koichi Ito; Akari Hayashi; Tohru Ishibashi; Emi Kondo; Takumi Ishikawa; Takashi Kawakami; Mineo Uchida; Takuma Yamazaki
- Web page: https://patentgrep.com/patent/US20250085744A1

## CPC

- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 1/1686: {the I/O peripheral being an integrated camera}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- H04N 5/33: Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20)
- H04N 17/002: {for television cameras}
- H04N 23/20: for generating image signals from infrared radiation only
- H04N 23/51: Housings
- H04N 23/60: Control of cameras or camera modules
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- H04N 23/6811: {based on the image signal}
- H04N 23/6812: {based on additional sensors, e.g. acceleration sensors}
- H04N 23/69: Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/50: Constructional details
- H04N 23/56: provided with illuminating means
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A wearable image pickup apparatus that eliminates manual change of an image pickup direction during picking up an image and can easily obtain an image that records experience while focusing attention on the experience. An image pickup apparatus includes an observation direction detection unit that is worn on a body other than a head of a user, an image pickup unit that is worn on the body of the user, a memory device that stores a set of instructions, and at least one processor that executes the set of instructions to: detect an observation direction of the user by the observation direction detection unit, pick up an image by the image pickup unit, and output an output image corresponding to the observation direction based on the image picked up by the image pickup unit.

## Claims

### Claim 1

(canceled)

### Claim 2

An image pickup apparatus that is wearable on a user, the image pickup apparatus comprising:
an observation direction detector;
an image pickup unit;
a memory device that stores a set of instructions; and
at least one processor that executes the set of instructions to:
detect an observation direction of the user by the observation direction detector;
pick up an image by the image pickup unit;
output an output image based on the image picked up by the image pickup unit;
generate metadata including in-image position information that shows a position and a size of the output image with respect to the image picked by the image pickup unit; and
generate a video file in which the metadata and the output image are encoded.

### Claim 3 (depends on 2)

The image pickup apparatus according to claim 2, wherein the output image corresponds to the observation direction.

### Claim 4 (depends on 2)

The image pickup apparatus according to claim 2, wherein the observation direction detector is worn on a body other than a head of a user.

### Claim 5 (depends on 2)

The image pickup apparatus according to claim 2, wherein the image pickup unit is worn on the body of the user.

### Claim 6 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to detect the observation direction of the user as a three-dimensional observation direction.

### Claim 7 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to:
output an observation direction of the user in a lateral direction as an angle in a first detection direction, and
output an observation direction of the user in a vertical direction as an angle in a second detection direction that is perpendicular to the first detection direction.

### Claim 8 (depends on 2)

The image pickup apparatus according to claim 2, wherein the observation direction detector comprises:
an infrared irradiator that irradiates an infrared irradiation surface of the user with infrared light; and
an infrared detector that detects reflected light of the infrared light reflected by the infrared irradiation surface.

### Claim 9 (depends on 8)

The image pickup apparatus according to claim 8, wherein the at least one processor executes instructions in the memory device to:
obtain distance information about each of distance areas of the infrared irradiation surface from the reflected light of the infrared light detected by the infrared detector, and
detect the observation direction based on the distance information.

### Claim 10 (depends on 9)

The image pickup apparatus according to claim 9, wherein the at least one processor executes instructions in the memory device to:
detect a head rotation center and a chin position of the user based on the distance information, and
detect the observation direction from the head rotation center and the chin position.

### Claim 11 (depends on 10)

The image pickup apparatus according to claim 10, wherein the at least one processor executes instructions in the memory device to:
set the head rotation center at a position that is nearest to the infrared detector and that is located at a center in a lateral direction in a distance area of which a relative distance to the infrared detector is shortest among the distance areas.

### Claim 12 (depends on 10)

The image pickup apparatus according to claim 10, wherein the at least one processor executes instructions in the memory device to:
set the chin position at a position that is farthest from the head rotation center and that is near a distance area of which a relative distance to the infrared detector is relatively large among the distance areas.

### Claim 13 (depends on 12)

The image pickup apparatus according to claim 12, wherein the at least one processor executes instructions in the memory device to:
output an observation direction of the user in a lateral direction as an angle in a first detection direction and outputs an observation direction of the user in a vertical direction as an angle in a second detection direction that is perpendicular to the first detection direction, and
calculate a moving angle of the chin position around the head rotation center as the angle in the first detecting direction.

### Claim 14 (depends on 12)

The image pickup apparatus according to claim 12, wherein the at least one processor executes instructions in the memory device to:
output an observation direction of the user in a lateral direction as an angle in a first detection direction,
output an observation direction of the user in a vertical direction as an angle in a second detection direction that is perpendicular to the first detection direction, and
calculate the angle in the second detecting direction based on intensity of the reflected light of the chin position.

### Claim 15 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to:
extract an image corresponding to the observation direction from the image picked up by the image pickup unit, and
output an extracted image.

### Claim 16 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to:
perform calibration of the observation direction detector using a calibrator capable of wirelessly connecting to the image pickup apparatus,
wherein the calibrator is provided with a face detector that projects infrared light and detects a face of the user, and
wherein the observation direction is not detected during a period in which the face detector is projecting the infrared light.

### Claim 17 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to:
obtain an optical correction value corresponding to an optical design of an image pickup lens in the image pickup unit,
wherein the optical correction value is included in the metadata.

### Claim 18 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to:
detect movement of the image pickup apparatus, and
obtain a moving amount,
wherein the moving amount is included in the metadata.

### Claim 19 (depends on 18)

The image pickup apparatus according to claim 18, wherein the moving amount is detected by one of an acceleration sensor that detects acceleration, an angular speed sensor that measures an angular speed, and a magnetometric sensor that measures a direction of a magnetic field.

### Claim 20 (depends on 2)

The image pickup apparatus according to claim 2, wherein the observation direction detector comprises a face direction detector that detects a face direction of the user,
wherein the at least one processor executes instructions in the memory device to:
obtain an image including a positioning index from the image pickup unit that picks up the positioning index;
detect the face direction by the face direction detector during calibration;
calculate a position of the positioning index in the image picked up during the calibration from a shape of the positioning index included in the image obtained;
generate information showing relationship between the face direction detected and the position of the positioning index calculated; and
perform calibration of a center position of a target visual field corresponding to the face direction detected by the face direction detector based on the information generated.

### Claim 21 (depends on 2)

The image pickup apparatus according to claim 2, wherein the observation direction detector comprises a face direction detector that detects a face direction of the user,
wherein the at least one processor executes instructions in the memory device to:
calculate an angular speed of the face of the user based on the face direction detected by the face direction detector;
determine a recording direction of the output image based on the observation direction; and
change the recording direction to a delayed direction that is delayed from movement of the detected face direction in a case where it is determined, as a result of the calculation, that the face of the user moves at the angular speed beyond a predetermined angular speed beyond a first predetermined period.

### Claim 22 (depends on 21)

The image pickup apparatus according to claim 21, wherein the at least one processor executes instructions in the memory device to output an entire image picked up by the image pickup unit during an image pickup operation of a moving object even if it is determined, as a result of the calculation, that the face of the user moves at the angular speed beyond the predetermined angular speed beyond the first predetermined period.

### Claim 23 (depends on 22)

The image pickup apparatus according to claim 22, wherein the at least one processor executes instructions in the memory device to stop changing the recording direction to the delayed direction in a case where a period elapsed after changing the recording direction to the delayed direction exceeds a second predetermined period.

### Claim 24 (depends on 21)

The image pickup apparatus according to claim 21, wherein the at least one processor executes instructions in the memory device not to change the recording direction in a case where it is determined, as a result of the calculation, that a period while the face of the user moves at the angular speed beyond a predetermined angular speed is less than the first predetermined period.

### Claim 25 (depends on 21)

The image pickup apparatus according to claim 21, wherein the at least one processor executes instructions in the memory device to add an image effect when the output image is switched from the image of the delayed direction to the image corresponding to the observation direction detected by the observation direction detector.

### Claim 26 (depends on 2)

The image pickup apparatus according to claim 2, wherein the at least one processor executes instructions in the memory device to:
detect a moving amount of the image pickup apparatus during a video image pickup operation by the image pickup unit;
delay the moving amount of the observation direction in a case where it is determined, based on a detection of the moving amount, that the image pickup apparatus is accelerating; and
accelerate the moving amount of the observation direction to cover a delayed amount in a case where it is determined, based on the detection of the moving amount, that the image pickup apparatus is slowing down.

### Claim 27 (depends on 26)

The image pickup apparatus according to claim 26, wherein the moving amount is detected by comparing images of a plurality of frames obtained by the image pickup unit during a video image pickup operation.

### Claim 28 (depends on 26)

The image pickup apparatus according to claim 26, wherein the at least one processor executes instructions in the memory device to output a part of the image that is extracted according to the observation direction.

### Claim 29 (depends on 26)

The image pickup apparatus according to claim 26, further comprising a driver that drives an image pickup direction of the image pickup unit in a yaw direction and a pitch direction,
wherein the at least one processor executes instructions in the memory device to control the driver so as to change the image pickup direction of the image pickup unit based on the observation direction.

### Claim 30 (depends on 26)

The image pickup apparatus according to claim 26, wherein the at least one processor executes instructions in the memory device to correct the moving amount of the observation direction so that the moving speed of the observation direction will become approximately constant in an output video image.

### Claim 31 (depends on 2)

The image pickup apparatus according to claim 2, further comprising a distance measurement sensor that measures a distance from the image pickup unit to an image pickup target area,
wherein the observation direction detector and the image pickup unit are integrally constituted,
wherein the at least one processor executes instructions in the memory device to:
create distance map information about the image pickup target area from the measurement result by the distance measurement sensor; and
calculate a direction of an observation object of the user seen from the image pickup unit based on the observation direction, the distance map information, and a vertical distance between the image pickup unit and an eye position of the user.

### Claim 32 (depends on 31)

The image pickup apparatus according to claim 31, further comprising a posture detector that detects a horizontal axis of the image pickup unit,
wherein the at least one processor executes instructions in the memory device to:
calculate an angle formed between the horizontal axis detected and a direction of an external positioning index seen from the image pickup unit;
calculate the vertical distance based on the angle calculated and a distance between the image pickup unit and the positioning index measured by the distance measurement sensor.

### Claim 33 (depends on 32)

The image pickup apparatus according to claim 32, wherein calibration of the observation direction detected by the observation direction detector is performed based on the vertical distance calculated, an observation direction detected by the observation direction detector in a case where the positioning index is located in each designated position, and the distance between the image pickup unit and the positioning index measured by the distance measurement sensor.

### Claim 34 (depends on 21)

The image pickup apparatus according to claim 21, wherein the at least one processor executes instructions in the memory device to:
switch an immediately preceding mode to a first image pickup mode, in which the recording direction is determined based on the observation direction during the video image pickup operation, in a case where the face direction detector can detect the face direction during the video image pickup operation; and
switch an immediately preceding mode to one of other image pickup modes, in which the recording direction is determined based on a factor other than the observation direction during the video image pickup operation, in a case where the face direction detector cannot detect the face direction.

### Claim 35 (depends on 34)

The image pickup apparatus according to claim 34, wherein the at least one processor executes instructions in the memory device to:
recognize an object from an image of the recording direction of a frame of a video image picked up by the image pickup unit;
set, in a case where the observation direction detector cannot detect the observation direction and an identical object is recognized in a past predetermined period, the factor other than the observation direction to a direction that tracks the identical object; and
switch an immediately preceding mode to a second image pickup mode that is one of the other image pickup modes.

### Claim 36 (depends on 35)

The image pickup apparatus according to claim 35, wherein the at least one processor executes instructions in the memory device to:
beforehand register an object to be detected;
set, in a case where the observation direction detector cannot detect the observation direction and the object registered beforehand is detected from a newest pickup image, the factor other than the observation direction to a direction that tracks the object registered beforehand; and
switch an immediately preceding mode to a third image pickup mode that is one of the other image pickup modes.

### Claim 37 (depends on 36)

The image pickup apparatus according to claim 36, wherein the at least one processor executes instructions in the memory device to:
set, in a case where the observation direction detector cannot detect the observation direction and neither an identical object nor the object registered beforehand can be detected, the factor other than the observation direction to one of the observation directions detected before the observation direction detector lost the observation direction and the observation direction that is moving at a change amount before the observation direction detector lost the observation direction; and
switch the immediately preceding mode to a fourth image pickup mode that is one of the other image pickup modes.

### Claim 38 (depends on 37)

The image pickup apparatus according to claim 37, wherein the at least one processor executes instructions in the memory device to widen a field angle of the image of the recording direction than a prescribed field angle in the fourth image pickup mode.

### Claim 39 (depends on 38)

The image pickup apparatus according to claim 38, wherein the switching of the mode is continuously active even after the mode is switched to one of the first, second, third, and fourth image pickup modes.

### Claim 40 (depends on 39)

The image pickup apparatus according to claim 39, wherein the at least one processor executes instructions in the memory device to restore the widened field angle to the prescribed field angle in a case where the fourth image pickup mode is switched to one of the first, second, third image pickup modes.

### Claim 41 (depends on 38)

The image pickup apparatus according to claim 38, wherein the at least one processor executes instructions in the memory device to:
notify the user of a detection error of the observation direction in a case where the observation direction detector cannot detect the observation direction; and
notify the user of the detection error in a case where the first image pickup mode is switched to one of the other image pickup modes.

### Claim 42 (depends on 2)

The image pickup apparatus according to claim 2, wherein the observation direction detector comprises a face direction detector that detects a face direction of the user,
wherein the at least one processor executes instructions in the memory device to:
detect a face direction of the user by the face direction detector;
calculate a first observation direction from the face direction detected;
estimate a second observation direction from a factor other than the face direction detected;
calculate reliability of the first observation direction;
determine the observation direction to the first observation direction in a case where the reliability is equal to or more than a threshold; and
determine the observation direction based on the first observation direction, the second observation direction, and the reliability in a case where the reliability is less than the threshold and the second observation direction is reliable.

### Claim 43 (depends on 2)

The image pickup apparatus according to claim 2, wherein a detection optical axis of the observation direction detector and an image pickup optical axis of the image pickup unit are directed in mutually different directions.

### Claim 44 (depends on 43)

The image pickup apparatus according to claim 43, wherein the detection optical axis of the observation direction detector is directed to a jaw of the user from the observing direction detector.

### Claim 45 (depends on 43)

The image pickup apparatus according to claim 43, wherein the image pickup optical axis of the image pickup unit is directed to a front direction of the user from the image pickup unit.

### Claim 46 (depends on 43)

The image pickup apparatus according to claim 43, wherein the image pickup apparatus in which the observation direction detector and the image pickup unit are integrally constituted is built in a camera body, and
wherein a lateral overall length of the image pickup apparatus is longer than its vertical overall length when seen from a front of the user in a state where the user wears the camera body.

### Claim 47 (depends on 46)

The image pickup apparatus according to claim 46, wherein the camera body is provided with fixing members that contact a user's body,
wherein the fixing members are respectively arranged in vicinities of right and left ends of the image pickup apparatus in the state where the user wears the camera body.

### Claim 48 (depends on 47)

The image pickup apparatus according to claim 47, further comprising contact angle adjusters that adjust angles of the fixing members with respect to the user's body.

### Claim 49 (depends on 46)

The image pickup apparatus according to claim 46, wherein the image pickup apparatus is connected with a neck hanging member for wearing the image pickup apparatus on a user's neck,
wherein the neck hanging member is connected to vicinities of right and left ends of the image pickup apparatus in the state where the user wears the camera body.

### Claim 50 (depends on 49)

The image pickup apparatus according to claim 49, wherein the neck hanging member is provided with a neck-hanging-angle adjuster that adjusts an angle of the neck hanging member with respect to the image pickup apparatus.

### Claim 51 (depends on 49)

The image pickup apparatus according to claim 49, wherein the neck hanging member is provided with a band part of which a sectional shape is not a perfect circle,
wherein distance between right and left portions of the band part that are symmetrical to the image pickup apparatus becomes shorter toward an upper side from a lower side in the state where the user wears the camera body.

### Claim 52 (depends on 50)

The image pickup apparatus according to claim 50, wherein the image pickup apparatus is connected with a power source unit through the neck hanging member,
wherein the power source unit is arranged behind the user's neck in the state where the user wears the camera body.

### Claim 53 (depends on 52)

The image pickup apparatus according to claim 52, wherein the image pickup apparatus is connected with the power source unit through a power supply member,
wherein the power supply member is arranged inside the neck hanging member.

### Claim 54 (depends on 21)

The image pickup apparatus according to claim 21, wherein the image pickup unit has an image pickup lens and an image sensor that convers an optical image formed by the image pickup lens into RAW data,
wherein the image pickup unit outputs RAW data read from a predetermined area of the image sensor as an image picked up by the image pickup unit,
wherein the at least one processor executes instructions in the memory device to:
extract data within an area, which is narrower than the predetermined area, including a target visual field in the recording direction and a margin around the target visual field from the RAW data; and
develop the data extracted.

### Claim 55 (depends on 54)

The image pickup apparatus according to claim 54, wherein the margin is a pixel area used for an image stabilization process.

### Claim 56 (depends on 54)

The image pickup apparatus according to claim 54, wherein the at least one processor executes instructions in the memory device to change a shape of the target visual field and a shape of the margin according to the recording direction and an optical property of the image pickup lens.

### Claim 57 (depends on 54)

The image pickup apparatus according to claim 54, wherein the at least one processor executes instructions in the memory device to record the data that is extracted and developed as an image of the recording direction without recording data that is not extracted from the predetermined area.

### Claim 58 (depends on 57)

The image pickup apparatus according to claim 57, wherein the at least one processor executes instructions in the memory device to transmit the image of the recording direction to an external appreciation device.

### Claim 59 (depends on 58)

The image pickup apparatus according to claim 58, wherein the external appreciation device applies an optical correction process and an image stabilization process to the image of the recording direction, and
wherein the at least one processor executes instructions in the memory device to transmit information required for the optical correction process and the image stabilization process together with the image of the recording direction.

### Claim 60

A portable device capable of wirelessly connecting to an image pickup apparatus, the portable device comprising:
a memory device that stores a set of instructions; and
at least one processor that executes the set of instructions to:
receive a video file in which metadata, which includes in-image position information that shows a position and a size of an image with respect to an image of each frame picked up by the image pickup apparatus, and an image of each frame are encoded;
extract the metadata from the video file;
extract the image of the frame encoded with the metadata extracted from the video file;
generate an image based on the image of the frame extracted from the video file using the metadata extracted from the video file; and
record the generated image as a video image.

### Claim 61

A control method for an image pickup apparatus that is wearable on a user, the control method comprising:
detecting an observation direction of the user by an observation direction detector;
picking up an image by an image pickup unit;
outputting an image based on the image picked up;
generating metadata including in-image position information that shows a position and a size of the output image with respect to the image picked up; and
generating a video file in which the metadata and the output image are encoded.

### Claim 62

A control method for a portable device capable of wirelessly connecting to an image pickup apparatus, the control method comprising:
receiving a video file in which metadata, which includes in-image position information that shows a position and a size of an image with respect to an image of each frame picked up by the image pickup apparatus, and an image of each frame are encoded;
extracting the metadata from the video file;
extracting the image of the frame encoded with the metadata extracted from the video file;
generating an image based on the image of the frame extracted using the metadata extracted; and
recording the generated image as a video image.

### Claim 63

A non-transitory computer-readable storage medium storing a control program causing a computer to execute a control method for an image pickup apparatus that is wearable on a user, the control method comprising:
detecting an observation direction of the user by an observation direction detector;
picking up an image by an image pickup unit;
outputting an image based on the image picked up;
generating metadata including in-image position information that shows a position and a size of the output image with respect to the image picked up; and
generating a video file in which the metadata and the output image are encoded.

### Claim 64

A non-transitory computer-readable storage medium storing a control program causing a computer to execute a control method for a portable device capable of wirelessly connecting to an image pickup apparatus, the control method comprising:
receiving a video file in which metadata, which includes in-image position information that shows a position and a size of an image with respect to an image of each frame picked up by the image pickup apparatus, and an image of each frame are encoded;
extracting the metadata from the video file;
extracting the image of the frame encoded with the metadata extracted from the video file;
generating an image based on the image of the frame extracted using the metadata extracted; and
recording the generated image as a video image.
