# US 2021/0241467 A1 · Electronic apparatus and controlling method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241467 A1
- Document ID: US20210241467A1
- Status: Published application
- Kind: A1
- Application number: 17/144,488
- Filed: 2021-01-08
- Published: 2021-08-05
- Earliest claimed: 2020-02-05
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jaeha LEE; Taehyeong KIM; Hyunjung NAM; Jaebum PARK; Joonah PARK
- Web page: https://patentgrep.com/patent/US20210241467A1

## CPC

- G06T 7/248: {involving reference images or patches}
- G01S 17/08: for measuring distance only (indirect measurement G01S17/46; active triangulation systems G01S17/48)
- G01S 17/66: Tracking systems using electromagnetic waves other than radio waves
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- H04N 23/611: where the recognised objects include parts of the human body
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An electronic apparatus is provided. The electronic apparatus includes a camera, a Light Detection And Ranging (LiDAR) sensor, and a processor configured to track an object based on a plurality of images photographed by the camera sequentially, wherein, to track the object based on the plurality of images, the processor is configured to, based on the object being identified in a first image from among a plurality of images and subsequently, not being identified in a second image after the first image, control a photographing direction of the camera based on scanning information obtained by the LiDAR sensor.

## Claims

### Claim 1

An electronic apparatus, the apparatus comprising:
a camera;
a Light Detection And Ranging (LiDAR) sensor; and
a processor configured to track an object based on a plurality of images photographed by the camera sequentially,
wherein, to track the object based on the plurality of images, the processor is configured to, based on the object being identified in a first image from among a plurality of images and, subsequently, not being identified in a second image after the first image, control a photographing direction of the camera based on scanning information obtained by the LiDAR sensor.

### Claim 2 (depends on 1)

The apparatus as claimed in claim 1, further comprising:
a driving unit configured to control a photographing direction of the camera,
wherein the processor is further configured to control the driving unit to change a photographing direction of the camera based on the scanning information.

### Claim 3 (depends on 2)

The apparatus as claimed in claim 2, further comprising:
a memory,
wherein the processor is further configured to:
identify an object based on first scanning information obtained by the LiDAR sensor while the first image is obtained,
match an object obtained from the first image and an object obtained from the first scanning information, and
store location information of the matched object in the memory.

### Claim 4 (depends on 3)

The apparatus as claimed in claim 3, wherein the processor is further configured to:
identify at least one object based on second scanning information obtained by the LiDAR sensor while the second image is obtained,
identify an object corresponding to location information of an object stored in the memory from among the at least one object, and
control the driving unit to change a photographing direction of the camera based on the object corresponding to the location information of the object stored in the memory.

### Claim 5 (depends on 4)

The apparatus as claimed in claim 4, wherein the processor is further configured to, based on a plurality of objects being identified based on the second scanning information:
identify an object most adjacent to the object stored in the memory from among the plurality of objects based on the location information of the object stored in the memory, and
control the driving unit to change a photographing direction of the camera based on the most adjacent object.

### Claim 6 (depends on 4)

The apparatus as claimed in claim 4, wherein the processor is further configured to, based on the object being identified in a third image obtained by the camera after the photographing direction of the camera is changed, terminate control of the driving unit.

### Claim 7 (depends on 3)

The apparatus as claimed in claim 3, wherein the processor is further configured to obtain location information of the matched object based on the object obtained from the first scanning information.

### Claim 8 (depends on 1)

The apparatus as claimed in claim 1, wherein the processor is further configured to:
based on the object not being identified in the second image, obtain the scanning information by turning on the LiDAR sensor, and
based on a plurality of objects being identified from the scanning information, identify one of the plurality of objects based on a photographing direction of the camera.

### Claim 9 (depends on 1)

The apparatus as claimed in claim 1, wherein a case where the object is not identified in the second image includes a case where the object is not identified in an image photographed by the camera based on at least one of a motion of the object, a motion of the electronic apparatus or a quality of the image photographed by the camera.

### Claim 10 (depends on 1)

The apparatus as claimed in claim 1, wherein the scanning information obtained by the LiDAR sensor is information obtained by scanning a direction of 180° or 300° with reference to the electronic apparatus.

### Claim 11

A method of controlling an electronic apparatus, the method comprising:
tracking an object based on a plurality of images photographed by a camera of the electronic apparatus sequentially; and
based on the object being identified in a first image from among a plurality of images and, subsequently, not being identified in a second image after the first image, controlling a photographing direction of the camera based on scanning information obtained by a LiDAR sensor of the electronic apparatus.

### Claim 12 (depends on 11)

The method as claimed in claim 11, further comprising:
changing a photographing direction of the camera based on the scanning information.

### Claim 13 (depends on 12)

The method as claimed in claim 12, further comprising:
identifying an object based on first scanning information obtained by the LiDAR sensor while the first image is obtained;
matching an object obtained from the first image and an object obtained from the first scanning information; and
storing location information of the matched object in a memory of the electronic apparatus.

### Claim 14 (depends on 13)

The method as claimed in claim 13, wherein the controlling of the photographing direction of the camera comprises:
identifying at least one object based on second scanning information obtained by the LiDAR sensor while the second image is obtained;
identifying an object corresponding to location information of an object stored in the memory from among the at least one object; and
changing a photographing direction of the camera based on the object corresponding to the location information of the object stored in the memory.

### Claim 15 (depends on 14)

The method as claimed in claim 14,
wherein the identifying of the object corresponding to location information of an object stored in the memory comprises, based on a plurality of objects being identified based on the second scanning information, identifying an object most adjacent to the object stored in the memory from among the plurality of objects based on the location information of the object stored in the memory; and
wherein the changing of the photographing direction of the camera comprises changing a photographing direction of the camera based on the most adjacent object.

### Claim 16 (depends on 14)

The method as claimed in claim 14, further comprising:
based on the object being identified in a third image obtained by the camera after the photographing direction of the camera is changed, terminating control of the photographing direction of the camera.

### Claim 17 (depends on 13)

The method as claimed in claim 13, further comprising:
obtaining location information of the matched object based on the object obtained from the first scanning information.

### Claim 18 (depends on 11)

The method as claimed in claim 11, wherein the controlling of the photographing direction of the camera comprises:
based on the object not being identified in the second image, obtaining the scanning information by turning on the LiDAR sensor; and
based on a plurality of objects being identified from the scanning information, identifying one of the plurality of objects based on a photographing direction of the camera.

### Claim 19 (depends on 1)

The apparatus as claimed in claim 1, wherein the processor is further configured to:
identify another object in the second image,
compare the object in the first image and the other object in the second image, and
determine that the object is identified in the first image and, subsequently, not identified in the second image after the first image based on the comparison.

### Claim 20 (depends on 19)

The apparatus as claimed in claim 19, wherein the processor is further configured to:
assign a weight to the other object based on the second image and on the scanning information, and
control the photographing direction of the camera based on the assigned weight to the other object.
