# US 10,375,289 B2 · System and method for providing autonomous photography and videography > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,375,289 B2 - Document ID: US10375289B2 - Status: Granted - Kind: B2 - Application number: 15/637,954 - Filed: 2017-06-29 - Published: 2018-10-04 - Granted: 2019-08-06 - Earliest claimed: 2017-03-31 - Assignees: Hangzhou Zero Zero Technology Co., Ltd. - Applicants: Hangzhou Zero Zero Technology Co., Ltd. - Inventors: Mengqiu Wang; Lixin Liu; Tong Zhang; Jia Lu - Web page: https://patentgrep.com/patent/US10375289B2 ## CPC - H04N 7/185: {from a mobile camera, e.g. for remote control} - G01S 3/7864: {T.V. type tracking systems} - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0011: {associated with a remote control arrangement} - G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target} - G06V 20/17: taken from planes or by drones - G06V 40/166: {using acquisition arrangements} - H04N 23/60: Control of cameras or camera modules - H04N 23/62: Control of parameters via user interfaces - H04N 23/66: Remote control of cameras or camera parts, e.g. by remote control devices - B64U 2101/30: for imaging, photography or videography - B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS] - B64U 2201/20: Remote controls - G01S 17/02: Systems using the reflection of electromagnetic waves other than radio waves (G01S17/66 takes precedence) - G06V 20/13: Satellite images - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 2201/07: Target detection - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR ## Abstract An aerial system, including a processing system, an optical system, an actuation system and a lift mechanism, includes an autonomous photography and/or videography system 70, implemented, at least in part, by the processing system 22, the optical system 26, the actuation system 28 and the lift mechanism 32. The autonomous photograph and/or videography system performs the steps of establishing a desired flight trajectory, detecting a target, controlling the flight of the aerial system as a function of the desired flight trajectory relative to the target using the lift mechanism and controlling the camera to capture pictures and/or video. ## Claims ### Claim 1 An aerial system, comprising: a body; a lift mechanism coupled to the body; an optical system including an optical sensor mounted to the body, the optical sensor movable with respect to the body by an actuation system; an inertial measurement unit (IMU) coupled to the body including a vibration sensor configured to sense a vibration of the body; and a processing system coupled to the lift mechanism, the optical system, the IMU, and the actuation system, the processing system includes a processor configured to: establish a desired flight trajectory; detect a target; establish a position and a velocity of the target relative to the aerial system; control the flight of the aerial system to maintain the aerial system at a distance from the target as a function of the desired flight trajectory and the established position and velocity of the target using the lift mechanism; control the actuation system to automatically adjust an angle of the optical sensor relative to the body to maintain the target within a field of view of the optical sensor during the flight of the aerial system; control the optical system to capture pictures and/or video; retrieve a vibration measurement for each time period associated with each captured picture and/or video from the IMU; select a moving window time interval and calculate a variance/standard deviation of vibration within the moving window time interval; and automatically select pictures and/or videos by comparing the calculated the variance/standard deviation of vibration to a pre-defined threshold. ### Claim 2 (depends on 1) An aerial system, as set forth in claim 1, the system further including one or more position sensors, wherein the processing system is configured to integrate positioning data from the position sensors to establish a position of the aerial system, the processing system further including an auto-detection and tracking module, the auto-detection and tracking module being configured to: detect and locate the target from a video feed received from the optical system; and, responsively establish the position and the velocity of the target relative to the aerial system. ### Claim 3 (depends on 2) An aerial system, as set forth in claim 2, wherein the target is a person and the auto-detection and tracking module utilizes facial recognition to detect and track the target. ### Claim 4 (depends on 2) An aerial system, as set forth in claim 2, further including additional sensor(s) associated with the target, the additional sensor(s) being configured to establish position data associated with the target to and to communicate the position data to the processing system. ### Claim 5 (depends on 1) An aerial system, as set forth in claim 1, wherein the desired flight trajectory is from a list of possible flight trajectories. ### Claim 6 (depends on 5) An aerial system, as set forth in claim 5, wherein the desired flight trajectory is selected from the list of possible flight trajectories by a user. ### Claim 7 (depends on 5) An aerial system, as set forth in claim 5, wherein the processing system selects the desired flight trajectory from the list of possible flight trajectories. ### Claim 8 (depends on 5) An aerial system, as set forth in claim 5, wherein the list of possible flight trajectories includes one or more of the following: (a) a trajectory in which the aerial system follows the target from behind at a fixed distance, indefinitely or for a finite amount of time; (b) a trajectory in which the aerial system may lead the target at the front while aiming at the target, keeping a fixed distance, indefinitely or for a finite amount of time; (c) a trajectory in which the aerial system orbits around the target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time; (d) a trajectory in which the aerial system moves closer to or further away from the target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time; (e) a trajectory in which the aerial system moves in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the target, with a constant/varying speed, for a finite amount of time; (f) a trajectory in which one or more degrees of freedom associated with the aerial system are held constant and other degrees of freedom are used to track the target; (g) a trajectory which has been pre-recorded by a user; and, (h) a trajectory comprising of any of the above trajectories, in a pre-defined, random, or pseudo-random sequence. ### Claim 9 (depends on 1) An aerial system, as set forth in claim 1, wherein the processing system includes an auto-selection module, the auto-selection module being configured to automatically select picture(s) and/or video and/or truncated video clip(s) using a lower frequency vibration filter. ### Claim 10 (depends on 9) An aerial system, as set forth in claim 9, wherein the auto-selection module automatically selects picture(s) and/or video and/or truncated video clip(s) based on over/under exposure and/or composition. ### Claim 11 (depends on 10) An aerial system, as set forth in claim 10, wherein the processing system includes an auto-editing module, the auto-editing module being configured to edit the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters. ### Claim 12 A method for operating an aerial system, the aerial system including a body, a lift mechanism, an optical system, and a body, the optical system including an optical sensor mounted to the body and movable with respect to the body by an actuation system, an inertial measurement unit (IMU) coupled to the body including a vibration sensor configured to sense a vibration of the body, and a processing system, method including the steps of: establishing, by the processing system, a desired flight trajectory; detecting a target; establishing, by the processing system, a position and a velocity of the target relative to the aerial system; controlling, by the processing system, the flight of the aerial system to maintain the aerial system at a distance from the target as a function of the desired flight trajectory and the established position and velocity of the target using the lift mechanism; controlling, by the processing system, the actuation system to automatically adjust an angle of the optical sensor relative to the body to maintain the target within a field of view of the optical sensor during the flight of the aerial system; controlling, by the processing system, the optical system to capture pictures and/or video retrieving a vibration measurement for each time period associated with each captured picture and/or video from the IMU; selecting a moving window time interval and calculating a variance/standard deviation of vibration within the moving window time interval; and automatically selecting pictures and/or videos by comparing the calculated the variance/standard deviation of vibration to a pre-defined threshold. ### Claim 13 (depends on 12) A method system, as set forth in claim 12, including the steps of: integrating positioning data from position sensors to establish a position of the aerial system; detecting and locating the target from a video feed received from the optical system; and, responsively establishing the position and the velocity of the target relative to the aerial system. ### Claim 14 (depends on 13) A method, as set forth in claim 13, wherein the target is a person and the step of detecting and tracking the target utilizes facial recognition. ### Claim 15 (depends on 13) A method, as set forth in claim 13, further including the steps of establishing position data associated with the target using additional sensor(s) associated with the target to and communicating the position data to the aerial system. ### Claim 16 (depends on 12) A method aerial system, as set forth in claim 12, wherein the desired flight trajectory is from a list of possible flight trajectories. ### Claim 17 (depends on 16) A method, as set forth in claim 16, wherein the desired flight trajectory is selected from the list of possible flight trajectories by a user. ### Claim 18 (depends on 16) A method, as set forth in claim 16, wherein the aerial system automatically selects the desired flight trajectory from the list of possible flight trajectories. ### Claim 19 (depends on 16) A method, as set forth in claim 16, wherein the list of possible flight trajectories includes one or more of the following: (a) a trajectory in which the aerial system follows the target from behind at a fixed distance, indefinitely or for a finite amount of time; (b) a trajectory in which the aerial system may lead the target at the front while aiming at the target, keeping a fixed distance, indefinitely or for a finite amount of time; (c) a trajectory in which the aerial system orbits around the target at a fixed distance with a constant or varying speed, indefinitely or for a finite amount of time; (d) a trajectory in which the aerial system moves closer to or further away from the target at a specified camera aiming angle, with a constant/varying speed, for a finite amount of time; (e) a trajectory in which the aerial system moves in a certain direction, in world coordinates or in target coordinates, while the optical system is aimed the target, with a constant/varying speed, for a finite amount of time; (f) a trajectory in which one or more degrees of freedom associated with the aerial system are held constant and other degrees of freedom are used to track the target; (g) a trajectory which has been pre-recorded by a user; and, (h) a trajectory comprising of any of the above trajectories, in a pre-defined, random, or pseudo-random sequence. ### Claim 20 (depends on 12) A method, as set forth in claim 12, including the step of automatically selecting picture(s) and/or video and/or truncated video clip(s) using a lower frequency vibration filter. ### Claim 21 (depends on 20) A method, as set forth in claim 20, wherein the step of automatically selecting picture(s) and/or video and/or truncated video clip(s) includes selecting picture(s) and/or video and/or truncated video clip(s) based on over/under exposure and/or composition. ### Claim 22 (depends on 21) A method, as set forth in claim 21, including the step of automatically editing the selected picture(s), video, and/or video clip(s) based on predetermined editing parameters.