# US 10,455,215 B2 · System, method, and recording medium for a closed-loop immersive viewing technology coupled to drones > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,455,215 B2 - Document ID: US10455215B2 - Status: Granted - Kind: B2 - Application number: 15/189,141 - Filed: 2016-06-22 - Published: 2017-12-28 - Granted: 2019-10-22 - Earliest claimed: 2016-06-22 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: George W. Doerre; Michael S. Gordon; James R. Kozloski; Clifford A. Pickover - Web page: https://patentgrep.com/patent/US10455215B2 ## CPC - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)} - G05D 1/0027: {involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00)} - G05D 1/0094: {involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target} - G05D 1/104: {involving a plurality of aircrafts, e.g. formation flying (traffic control systems for aircraft G08G5/00)} - H04N 13/296: Synchronisation thereof; Control thereof - H04N 13/344: with head-mounted left-right displays - B64U 2101/30: for imaging, photography or videography - B64U 2201/102: adapted for flying in formations - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR ## Abstract A viewing technology method, system, and non-transitory computer readable medium including a display device associated with a user and at least two drones having an image capturing device, include a drone control circuit configured to control a flight path of the at least two drones such that the drones are separated by the inter-drone distance, a vergence angle determining circuit configured to determine a vergence angle of the pupils of the user relative to the image displayed on the display device, and a image control circuit configured to control a display of the image on the display device according to the vergence angle to cause the image to create a just-noticeable-difference in the image. ## Claims ### Claim 1 A viewing technology system including a display device associated with a user and at least two drones having an image capturing device, the system comprising: a drone separation calculating circuit configured to calculate an inter-drone distance between the at least two drones to create an image at a predetermined scale based on a relationship between the inter-drone distance and an inter-ocular distance between pupils of the user; a drone control circuit configured to control a flight path of the at least two drones by changing the flight path of the at least two drones to set the at least two drones at a new flight path such that the drones are separated by the inter-drone distance; a vergence angle determining circuit configured to determine a vergence angle of the pupils of the user relative to the image displayed on the display device; an image control circuit configured to control a display of the image on the display device according to the vergence angle to cause the image to create a just-noticeable-difference in the image; and a scale adjusting circuit configured to adjust the predetermined scale of the image based on a second scale input to the system by the user by causing the drone separation calculating circuit to calculate a new inter-drone distance for the at least two drones according to the second scale and causing the drone control circuit to control the flight path of the at least two drones to be separated by the new inter-drone distance, wherein the scale adjusting circuit causes the drone control circuit to control the flight path of at least two drones to be separated by the new inter-drone distance by moving the drones along lines of parallax by moving each drone a same amount along the lines of parallax at a same time from the inter-drone distance to the new inter-drone distance such that the user perceives the image to be growing or shrinking by maintaining a ratio between the inter-drone distance and the new inter-drone distance while the drones are moving from the inter-drone distance to the new inter-drone distance. ### Claim 2 (depends on 1) The system of claim 1, wherein the image control circuit creates the just-noticeable-difference to provide a scale model illusion image of variable depth of field. ### Claim 3 (depends on 1) The system of claim 1, wherein the display device comprises a head-mounted display worn by the user. ### Claim 4 (depends on 1) The system of claim 1, wherein the drone control circuit further controls the at least two drones to change the flight path based on a body movement of the user detected by the display device while maintaining the inter-drone distance. ### Claim 5 (depends on 1) The system of claim 1, wherein the drone control circuit further controls the at least two drones to change the flight path based on an eye movement of the user detected by the display device while maintaining the inter-drone distance. ### Claim 6 (depends on 1) The system of claim 1, wherein if the display device detects a tracking eye-type movement and a tracked object of the tracking eye-type movement is within a predetermined distance of a periphery of the image, the drone control circuit controls the at least two drones to change the flight path to capture the image in a direction of the tracking eye-type movement. ### Claim 7 (depends on 1) The system of claim 1, wherein the image control unit changes a color saturation of the image to create the just-noticeable-difference. ### Claim 8 (depends on 4) The system of claim 4, wherein the image control circuit causes a transition period scene to be displayed on the display device until the flight path of the at least two drones is finished changing. ### Claim 9 (depends on 1) The system of claim 1, wherein the image control circuit generates a virtual world representation image stored in a database if the inter-drone distance is unobtainable. ### Claim 10 (depends on 1) The system of claim 1, wherein the image control circuit generates a virtual world representation image depicting the image at the predetermined scale stored in a database until the flight path of the at least two drones reaches the inter-drone distance. ### Claim 11 (depends on 1) The system of claim 1, wherein the system includes a swarm of drones such that the at least two drones are interchangeable with other drones of the swarm of drones. ### Claim 12 (depends on 11) The system of claim 11, wherein the at least two drones are swapped with a drone in the drone swarm based on at least one of the vergence angle and the inter-drone distance. ### Claim 13 (depends on 1) The system of claim 1, wherein the image control circuit causes the images to be modified to omit a portion of the image based on geo-positional data of the user, the inter-drone distance, the predetermined scale, and the vergence angle according to a security setting. ### Claim 14 A viewing technology method including a display device associated with a user and at least two drones having an image capturing device, the method comprising: calculating an inter-drone distance between the at least two drones to create an image at a predetermined scale based on a relationship between the inter-drone distance and an inter-ocular distance between pupils of the user; controlling a flight path of the at least two drones by changing the flight path of the at least two drones to set the at least two drones at a new flight path such that the drones are separated by the inter-drone distance; determining a vergence angle of the pupils of the user relative to the image displayed on the display device; controlling a display of the image on the display device according to the vergence angle to cause the image to create a just-noticeable-difference in the image; and adjusting the predetermined scale of the image based on a second scale input to the system by the user by causing the calculating to calculate a new inter-drone distance for the at least two drones according to the second scale and causing the controlling to control the flight path of the at least two drones to be separated by the new inter-drone distance, wherein the adjusting causes the controlling to control the flight path of at least two drones to be separated by the new inter-drone distance by moving the drones along lines of parallax by moving each drone a same amount along the lines of parallax at a same time from the inter-drone distance to the new inter-drone distance such that the user perceives the image to be growing or shrinking by maintaining a ratio between the inter-drone distance and the new inter-drone distance while the drones are moving from the inter-drone distance to the new inter-drone distance. ### Claim 15 (depends on 14) The method of claim 14, wherein the controlling the display creates the just-noticeable-difference to provide a scale model illusion image of variable depth of field. ### Claim 16 A non-transitory computer-readable recording medium recording a viewing technology program including a display device associated with a user and at least two drones having an image capturing device, the program causing a computer to perform: calculating an inter-drone distance between the at least two drones to create an image at a predetermined scale based on a relationship between the inter-drone distance and an inter-ocular distance between pupils of the user; controlling a flight path of the at least two drones by changing the flight path of the at least two drones to set the at least two drones at a new flight path such that the drones are separated by the inter-drone distance; determining a vergence angle of the pupils of the user relative to the image displayed on the display device; controlling a display of the image on the display device according to the vergence angle to cause the image to create a just-noticeable-difference in the image; and adjusting the predetermined scale of the image based on a second scale input to the system by the user by causing the calculating to calculate a new inter-drone distance for the at least two drones according to the second scale and causing the controlling to control the flight path of the at least two drones to be separated by the new inter-drone distance, wherein the adjusting causes the controlling to control the flight path of at least two drones to be separated by the new inter-drone distance by moving the drones along lines of parallax by moving each drone a same amount along the lines of parallax at a same time from the inter-drone distance to the new inter-drone distance such that the user perceives the image to be growing or shrinking by maintaining a ratio between the inter-drone distance and the new inter-drone distance while the drones are moving from the inter-drone distance to the new inter-drone distance. ### Claim 17 (depends on 1) The system of claim 1, wherein each drone moves the same amount along the lines of parallax such that the display of the image is not distorted during movement. ### Claim 18 (depends on 1) The system of claim 1, wherein the display of the image follows a tracking eye-type movement of the user. ### Claim 19 (depends on 18) The system of claim 18, wherein if the display device detects the tracking eye-type movement and a tracked object of the tracking eye-type movement is within a predetermined distance of a periphery of the image, the drone control circuit controls the at least two drones to change the flight path to capture the image in a direction of the tracking eye-type movement. ### Claim 20 (depends on 1) The system of claim 1, wherein the image control circuit controls the display of the image to provide an illusion of variable depth of field and color saturation to reinforce a scale model illusion in which depth cues are mapped onto objects in the field of view to create the just-noticeable-difference.