# US 2024/0342920 A1 · Control system, control method, and camera control device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0342920 A1
- Document ID: US20240342920A1
- Status: Published application
- Kind: A1
- Application number: 18/556,365
- Filed: 2021-12-28
- Published: 2024-10-17
- Earliest claimed: 2021-04-30
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: Ryuta HORIE
- Web page: https://patentgrep.com/patent/US20240342920A1

## CPC

- B25J 9/1697: {Vision controlled systems}
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- B25J 19/023: {including video camera means}
- G03B 7/093: for control of exposure time
- G03B 15/00: Special procedures for taking photographs; Apparatus therefor
- H04N 23/73: by influencing the exposure time
- H04N 23/80: Camera processing pipelines; Components thereof
- B25J 9/1682: {Dual arm manipulator; Coordination of several manipulators}
- G05B 2219/36436: Arm follows movement of handheld device, camera detects, analyses motion
- G05B 2219/39057: Hand eye calibration, eye, camera on hand, end effector
- G05B 2219/39114: Hand eye cooperation, active camera on first arm follows movement of second arm
- G05B 2219/40572: Camera combined with position sensor
- G05B 2219/40613: Camera, laser scanner on end effector, hand eye manipulator, local
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR (optical parts of such apparatus G02B; photosensitive materials or processes for photographic purposes G03C; apparatus for processing exposed photographic materials G03D)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Provided is a control system that controls an imaging condition of a camera equipped on a robot to capture an image in which a blur is suppressed. The control system includes a robot control unit that controls a motion of the robot equipped with the camera, and a camera control unit that controls an imaging operation of the camera. The camera control unit determines an exposure time for suppressing a blur that occurs in a captured image of the camera within an allowable pixel number for the moving speed of the camera, and in a case where the camera control unit cannot set the exposure time, the camera control unit instructs the robot control unit to update a speed limit value of the robot.

## Claims

### Claim 1

A control system comprising:
a robot control unit that controls a motion of a robot equipped with a camera; and
a camera control unit that controls an imaging operation of the camera, wherein
the camera control unit sets an imaging condition of the camera on a basis of a moving speed of the camera by the motion of the robot.

### Claim 2 (depends on 1)

The control system according to claim 1, wherein the camera control unit determines an exposure time for suppressing a blur that occurs in a captured image of the camera within an allowable pixel number for the moving speed of the camera.

### Claim 3 (depends on 2)

The control system according to claim 2, wherein when the moving speed of the camera increases, the camera control unit reduces the exposure time and controls the blur of the captured image to be within the allowable pixel number.

### Claim 4 (depends on 2)

The control system according to claim 2, wherein the camera control unit instructs the robot control unit to update a speed limit value of the robot in a case where the camera control unit cannot set the exposure time for suppressing the blur within the allowable pixel number for the moving speed of the camera.

### Claim 5 (depends on 4)

The control system according to claim 4, wherein the camera control unit instructs the robot control unit to decrease the speed limit value in a case where the moving speed of the camera cannot suppress the blur within the allowable pixel number in the exposure time to be set as a shortest possible exposure time.

### Claim 6 (depends on 2)

The control system according to claim 2, further comprising a position estimation unit that performs position estimation on a basis of the captured image of the camera, wherein
the camera control unit changes the allowable pixel number on a basis of a deviation between a target position input to the robot control unit and an estimated position estimated by the position estimation unit.

### Claim 7 (depends on 6)

The control system according to claim 6, wherein the camera control unit changes the allowable pixel number to a larger value as the deviation increases, and changes the allowable pixel number to a smaller value as the deviation decreases.

### Claim 8 (depends on 6)

The control system according to claim 6, wherein the camera control unit controls a frame rate of the camera on a basis of at least one of the deviation or the moving speed of the camera.

### Claim 9 (depends on 8)

The control system according to claim 8, wherein the camera control unit reduces a frame rate of the camera when the deviation is small or the moving speed of the camera is not fast.

### Claim 10 (depends on 1)

The control system according to claim 1, wherein the camera control unit determines the imaging condition of the camera every time a zoom operation of the camera is performed.

### Claim 11 (depends on 1)

The control system according to claim 1, wherein the moving speed of the camera includes a relative speed between a target object of the robot and the camera.

### Claim 12

A control method for controlling a robot and a camera equipped on the robot, the control method comprising steps of:
acquiring a moving speed of the camera by a motion of the robot;
setting an imaging condition of the camera on a basis of the moving speed of the camera; and
controlling the imaging operation of the camera on a basis of the imaging condition having been set.

### Claim 13

A camera control device comprising:
an acquirer that acquires a moving speed of a camera by a motion of a robot equipped with the camera;
a setting unit that sets an imaging condition of the camera on a basis of the moving speed of the camera; and
an execution unit that executes an imaging operation of the camera on a basis of the imaging condition having been set.
