# US 10,771,692 B2 · Imaging systems and methods > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,692 B2 - Document ID: US10771692B2 - Status: Granted - Kind: B2 - Application number: 16/118,303 - Filed: 2018-08-30 - Published: 2018-12-27 - Granted: 2020-09-08 - Earliest claimed: 2014-10-24 - Assignees: Bounce Imaging, Inc. - Applicants: Bounce Imaging, Inc. - Inventors: Pablo Alvarado-Moya; Laura Cabrera; Sietse Dijkstra; Francisco Aguilar - Web page: https://patentgrep.com/patent/US10771692B2 ## CPC - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 3/047: Fisheye or wide-angle transformations - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - H04N 13/243: using three or more two-dimensional [2D] image sensors - H04N 13/261: with monoscopic-to-stereoscopic image conversion - H04N 23/23: from thermal infrared radiation - H04N 23/51: Housings - H04N 23/56: provided with illuminating means - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - H04N 21/43637: {involving a wireless protocol, e.g. Bluetooth®, RF or wireless LAN [IEEE 802.11] (arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract At least one combined image may be created from a plurality of images captured by a plurality of cameras. A sensor unit may receive the plurality of images from the plurality of cameras. At least one processor in communication with the sensor unit may correlate each received image with calibration data for the camera from which the image was received. The calibration data may comprise camera position data and characteristic data. The processor may combine at least two of the received images from at least two of the cameras into the at least one combined image by orienting the at least two images relative to one another based on the calibration data for the at least two cameras from which the images were received and merging the at least two aligned images into the at least one combined image. ## Claims ### Claim 1 A system for creating at least one combined image from a plurality of images captured by a plurality of cameras, comprising: a memory configured to store predetermined calibration data for each of the plurality of cameras, the calibration data for each camera being derived from an image of a known training environment and compensating for: a camera position relative to each other camera of the plurality of cameras independent of the location of the camera in an environment, and a camera characteristic based on an intrinsic parameter of the camera, the calibration data comprising data for mapping pixels imaged by the camera inside a projection geometry and positioned about a virtual center point; a sensor unit configured to receive the plurality of images from the plurality of cameras; and at least one processor in communication with the sensor unit and the memory, the at least one processor configured to: correlate image data from each received image with the predetermined calibration data for the camera from which the image was received; and combine the image data from at least one of the cameras into the at least one combined image by: positioning pixels of the image data from the at least one of the cameras based on the data for mapping pixels; and merging the positioned pixels of the image data from the at least one of the cameras into the at least one combined image. ### Claim 2 (depends on 1) The system of claim 1, wherein the sensor unit comprises a throwable ball. ### Claim 3 (depends on 1) The system of claim 1, wherein at least one of the cameras comprises a fisheye lens. ### Claim 4 (depends on 1) The system of claim 1, wherein at least one of the cameras comprises a sensor and a lens. ### Claim 5 (depends on 1) The system of claim 1, wherein the camera characteristic for each camera is based on optical characteristics of the camera. ### Claim 6 (depends on 1) The system of claim 1, further comprising a receiver unit in communication with the processor and configured to display the at least one combined image. ### Claim 7 (depends on 6) The system of claim 6, wherein the receiver unit comprises a smartphone, a tablet, a personal computer, a server, or a combination thereof. ### Claim 8 (depends on 6) The system of claim 6, wherein the receiver unit is wirelessly coupled to the sensor unit. ### Claim 9 (depends on 1) The system of claim 1, wherein the processor is wirelessly coupled to the sensor unit. ### Claim 10 (depends on 1) The system of claim 1, wherein the at least one combined image is a video frame. ### Claim 11 (depends on 1) The system of claim 1, wherein positioning the pixels of the image data from the at least one of the cameras comprises correcting for a parallax effect in at least one of the images, determining a transformation factor for portions of the image data, relating pixel coordinate systems between the portions of the image data, estimating a global alignment between the portions of the image data, detecting a common distinctive feature in the portions of the image data, computing a globally consistent set of alignments for the portions of the image data, selecting a final compositing surface and a parameterization for the portions of the image data, or a combination thereof. ### Claim 12 (depends on 11) The system of claim 11, wherein the transformation factor comprises a scaling factor. ### Claim 13 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to generate the calibration data. ### Claim 14 (depends on 13) The system of claim 13, wherein generating the calibration data comprises: performing an intrinsic calibration for each of the plurality of cameras; and performing an extrinsic calibration for the plurality of cameras. ### Claim 15 (depends on 14) The system of claim 14, wherein the at least one processor is configured to perform the intrinsic calibration for each of the plurality of cameras by: positioning the camera in a plurality of locations relative to a calibration object; receiving an image generated by the camera at each of the plurality of locations; and varying at least one intrinsic parameter based on the images generated by the camera at each of the plurality of locations until an optimum is found. ### Claim 16 (depends on 14) The system of claim 14, wherein the at least one processor is configured to perform the intrinsic calibration for each of the plurality of cameras by: creating a projection model for the camera; determining a ray originating at a coordinate system of the camera that contains a corresponding point in space; performing a great circle fitting; defining an objective function for the camera; performing an optimization; or a combination thereof. ### Claim 17 (depends on 14) The system of claim 14, wherein the at least one processor is configured to perform the extrinsic calibration for the plurality of cameras by: positioning each camera in a plurality of locations relative to a calibration object; receiving an image generated by each camera at each of the plurality of locations; for each camera, finding a rotation and translation between a coordinate system of the camera and a reference coordinate system based on the images generated by the camera at each of the plurality of locations; and deriving camera positions and orientations relative to each other based on the rotations and translations. ### Claim 18 (depends on 14) The system of claim 14, wherein the at least one processor is configured to perform the extrinsic calibration for each of the plurality of cameras by: performing an optimization; calibrating the camera using a plurality of images of a calibration object; creating a translation vector for the camera; relating the camera to each of the other cameras; or a combination thereof. ### Claim 19 (depends on 1) The system of claim 1, wherein the calibration data is generated prior to the sensor unit receiving the plurality of images. ### Claim 20 (depends on 1) The system of claim 1, wherein: the known training environment comprises a three-dimensional calibration object that expresses a known set of lines in space with precisely measured coordinates in substantially all directions around the plurality of cameras; and the known set of lines in space establish the projection geometry about the virtual center point. ### Claim 21 (depends on 1) The system of claim 1, wherein the projection geometry comprises a sphere. ### Claim 22 (depends on 1) The system of claim 1, further comprising at least one orientation sensor, wherein the at least one processor is configured to orient the at least one combined image to an upright position based on orientation data determined by the at least one orientation sensor. ### Claim 23 (depends on 1) The system of claim 1, wherein at least one of the plurality of cameras is a thermal camera. ### Claim 24 A method for creating at least one combined image from a plurality of images captured by a plurality of cameras, comprising: receiving, at a sensor unit, the plurality of images from the plurality of cameras; correlating, with at least one processor in communication with the sensor unit, image data from each received image with predetermined calibration data for the camera from which the image was received, the calibration data for each camera being derived from an image of a known training environment and compensating for: a camera position relative to each other camera of the plurality of cameras independent of the location of the camera in an environment, and a camera characteristic based on an intrinsic parameter of the camera, the calibration data comprising data for mapping pixels imaged by the camera inside a projection geometry and positioned about a virtual center point; and combining, with the at least one processor, the image data from at least one of the cameras into the at least one combined image by: positioning pixels of the image data from the at least one of the cameras based on the data for mapping pixels; and merging the positioned pixels of the image data from the at least one of the cameras into the at least one combined image. ### Claim 25 (depends on 24) The method of claim 24, wherein the sensor unit comprises a throwable ball. ### Claim 26 (depends on 24) The method of claim 24, wherein at least one of the cameras comprises a fisheye lens. ### Claim 27 (depends on 24) The method of claim 24, wherein at least one of the cameras comprises a sensor and a lens. ### Claim 28 (depends on 24) The method of claim 24, wherein the camera characteristic for each camera is based on optical characteristics of the camera. ### Claim 29 (depends on 24) The method of claim 24, further comprising displaying, with a receiver unit in communication with the processor, the at least one combined image. ### Claim 30 (depends on 29) The method of claim 29, wherein the receiver unit comprises a smartphone, a tablet, a personal computer, a server, or a combination thereof. ### Claim 31 (depends on 29) The method of claim 29, wherein the receiver unit is wirelessly coupled to the sensor unit. ### Claim 32 (depends on 24) The method of claim 24, wherein the processor is wirelessly coupled to the sensor unit. ### Claim 33 (depends on 24) The method of claim 24, wherein the at least one combined image is a video frame. ### Claim 34 (depends on 24) The method of claim 24, wherein positioning the pixels of the image data from the at least one of the cameras comprises correcting for a parallax effect in at least one of the images, determining a transformation factor for portions of the image data, relating pixel coordinate systems between the portions of the image data, estimating a global alignment between the portions of the image data, detecting a common distinctive feature in the portions of the image data, computing a globally consistent set of alignments for the portions of the image data, selecting a final compositing surface and a parameterization for the portions of the image data, or a combination thereof. ### Claim 35 (depends on 34) The method of claim 34, wherein the transformation factor comprises a scaling factor. ### Claim 36 (depends on 24) The method of claim 24, further comprising generating, with the at least one processor, the calibration data. ### Claim 37 (depends on 36) The method of claim 36, wherein generating the calibration data comprises: performing an intrinsic calibration for each of the plurality of cameras; and performing an extrinsic calibration for the plurality of cameras. ### Claim 38 (depends on 37) The method of claim 37, wherein performing the intrinsic calibration for each of the plurality of cameras comprises: positioning the camera in a plurality of locations relative to a calibration object; receiving an image generated by the camera at each of the plurality of locations; and varying at least one intrinsic parameter based on the images generated by the camera at each of the plurality of locations until an optimum is found. ### Claim 39 (depends on 37) The method of claim 37, wherein performing the intrinsic calibration for each of the plurality of cameras comprises: creating a projection model for the camera; determining a ray originating at a coordinate system of the camera that contains a corresponding point in space; performing a great circle fitting; defining an objective function for the camera; performing an optimization; or a combination thereof. ### Claim 40 (depends on 37) The method of claim 37, wherein performing the extrinsic calibration for the plurality of cameras comprises: positioning each camera in a plurality of locations relative to a calibration object; receiving an image generated by each camera at each of the plurality of locations; for each camera, finding a rotation and translation between a coordinate system of the camera and a reference coordinate system based on the images generated by the camera at each of the plurality of locations; and deriving camera positions and orientations relative to each other based on the rotations and translations. ### Claim 41 (depends on 37) The method of claim 37, wherein performing the extrinsic calibration for the plurality of cameras comprises: performing an optimization; calibrating the camera using a plurality of images of a calibration object; creating a translation vector for the camera; relating the camera to each of the other cameras; or a combination thereof. ### Claim 42 (depends on 24) The method of claim 24, wherein the calibration data is generated prior to the sensor unit receiving the plurality of images. ### Claim 43 (depends on 24) The method of claim 24, wherein: the known training environment comprises a three-dimensional calibration object that expresses a known set of lines in space with precisely measured coordinates in substantially all directions around the plurality of cameras; and the known set of lines in space establish the projection geometry about the virtual center point. ### Claim 44 (depends on 24) The method of claim 24, wherein the projection geometry comprises a sphere. ### Claim 45 (depends on 24) The method of claim 24, further comprising: determining, with at least one orientation sensor, orientation data; and orienting, with the at least one processor, the at least one combined image to an upright position based on the orientation data. ### Claim 46 (depends on 24) The method of claim 24, wherein at least one of the plurality of cameras is a thermal camera.