Three dimensional media streaming and broadcasting system and method
AssigneeKarinca Teknoloji Ve Ilet. San. Tic. Ltd. Sti
InventorHamit MERMERKAYA
Life of this patent
Estimating the term…
Abstract
A three-dimensional media streaming and broadcasting system and method, which enables different media, such as concerts and sports events, to be transferred in a live and three-dimensional manner, and the transferred image to be broadcast and viewed with devices such as virtual reality goggles.
Claims11
Three-dimensional media streaming and broadcasting method, which enables different environments, such as concerts and sports events, to be streamed live and in three dimensions, and the streamed image to be broadcast and viewed by monitoring devices such as virtual reality goggles, characterized by:
- a server grouping all the images coming from each camera set within the same time interval in terms of depth and color, and upon grouping, generating image frames (110);
- the server computing X, Y and Z axis coordinates of each pixel coming from a depth-sensing camera, and creating a point cloud from the grouped pixels (120);
- after the point cloud is created, in accordance with the depth-sensing camera chosen as reference, rotating the images of other depth-sensing cameras according to the reference camera set and aligning the point cloud, and obtaining the key point cloud required to generate the first 3D key frame (130);
- the newly aligned and combined point cloud coming from the depth-sensing cameras and the key point cloud for the 3D key frame are continuously renewed within a preferred time period;
- when a new point cloud comes from the depth-sensing cameras, 3D key frame renewal time is controlled;
- when the new point cloud comes, if a new 3D key frame is to be obtained (if the surface of the point cloud is to be constructed (mesh creation)) going to the next step, and additionally with the last 3D key frame generated asynchronously, going to step 190, or if a 3D key frame will not be generated and the old frame will be accepted, going to step 190 (140);
- if the point cloud is accepted as the key point cloud to generate the 3D key frame, filtering the point cloud respectively (150);
- constructing the surface of the filtered point cloud (mesh creation) (160);
- performing surface filtering of the point cloud whose surfaces are created (170);
- texturing the surface-filtered point cloud and broadcasting the image (180);
- if the new 3D key frame is not accepted, determining the skeleton information of the living and objects moving on the last generated 3D key frame together with the joints by using the current point cloud via the artificial intelligence algorithms running on the server (190);
- animating the living organisms and the objects moving on the current key frame according to their joints, and broadcasting them for the monitoring devices;
- (200);
- as long as the event continues, returning to the first process step.
Three-dimensional media streaming and broadcasting system (1), which enables different environments, such as concerts and sports events, to be streamed live and in three dimensions, and the streamed image to be broadcast and viewed by monitoring devices such as virtual reality goggles, comprising:
- camera sets, which enable to capture images from the event venue in three dimensions, and includes at least one depth-sensing camera, at least one color camera and at least one computing module;
- depth-sensing camera, which receives depth information for each pixel, captures multiple pixel images at up to 30 frames per second, and is adapted to enable detection of the depths of the images of the venue and all people and objects present in the scene, where digital recording is performed, separately for each pixel;
- a color camera which is adapted to enable capturing colored image of the scene that is recorded and the entire site that is captured at high resolution and high frame rate (fbs:
- frame per second);
- a computing module with GPU capability, which is adapted to capture and compress the depth and color images received from the depth-sensing camera and the color camera simultaneously and after synchronizing, transmitting them to the server via Internet connection;
- a server which is adapted to generate image frames by grouping the images coming from each camera set within the same time interval in terms of depth and color;
- server which is adapted to compute the X, Y and Z axis coordinates of each pixel coming from the camera set and create the point cloud from the grouped pixels, and after the point cloud is created, in accordance with the camera set chosen as reference, to rotate the images of the other camera cameras according to the reference camera set and aligning the point cloud, and thereby obtaining the first key frame;
- server which is adapted to continuously renew the key frame within a preferred time period with the new images coming from the camera sets, and when new images are received from the camera set, to control whether they came within the period of key frame renewal time;
- server which, when the new images are received, is adapted to create the surface of the point cloud (mesh creation) if a new key frame is to be obtained, or if the old frame will be accepted as the point cloud, to determine the skeleton information of the living and objects moving on the key frame together with their joints and animating the living organisms and the objects moving on the current key frame according to their joints and to broadcasting the image.
Dependencies are inferred from the claim text.