# US 2021/0343081 A1 · Three dimensional media streaming and broadcasting system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343081 A1
- Document ID: US20210343081A1
- Status: Published application
- Kind: A1
- Application number: 17/281,144
- Filed: 2019-10-07
- Published: 2021-11-04
- Earliest claimed: 2019-07-05
- Assignees: Karinca Teknoloji Ve Ilet. San. Tic. Ltd. Sti
- Applicants: Karinca Teknoloji Ve Ilet. San. Tic. Ltd. Sti
- Inventors: Hamit MERMERKAYA
- Web page: https://patentgrep.com/patent/US20210343081A1

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G06T 7/55: from multiple images
- G06T 13/00: Animation
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 15/00: Three-dimensional [3D] image rendering
- H04N 13/194: Transmission of image signals
- H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/344: with head-mounted left-right displays
- H04N 21/2187: Live feed
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2210/56: Particle system, point based geometry or rendering
- H04N 2013/0096: {Synchronisation or controlling aspects}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

A three-dimensional media streaming and broadcasting system and method, which enables different media, such as concerts and sports events, to be transferred in a live and three-dimensional manner, and the transferred image to be broadcast and viewed with devices such as virtual reality goggles.

## Claims

### Claim 1

Three-dimensional media streaming and broadcasting method, which enables different environments, such as concerts and sports events, to be streamed live and in three dimensions, and the streamed image to be broadcast and viewed by monitoring devices such as virtual reality goggles, characterized by:
a server grouping all the images coming from each camera set within the same time interval in terms of depth and color, and upon grouping, generating image frames (110);
the server computing X, Y and Z axis coordinates of each pixel coming from a depth-sensing camera, and creating a point cloud from the grouped pixels (120);
after the point cloud is created, in accordance with the depth-sensing camera chosen as reference, rotating the images of other depth-sensing cameras according to the reference camera set and aligning the point cloud, and obtaining the key point cloud required to generate the first 3D key frame (130);
the newly aligned and combined point cloud coming from the depth-sensing cameras and the key point cloud for the 3D key frame are continuously renewed within a preferred time period; when a new point cloud comes from the depth-sensing cameras, 3D key frame renewal time is controlled; when the new point cloud comes, if a new 3D key frame is to be obtained (if the surface of the point cloud is to be constructed (mesh creation)) going to the next step, and additionally with the last 3D key frame generated asynchronously, going to step 190, or if a 3D key frame will not be generated and the old frame will be accepted, going to step 190 (140);
if the point cloud is accepted as the key point cloud to generate the 3D key frame, filtering the point cloud respectively (150);
constructing the surface of the filtered point cloud (mesh creation) (160);
performing surface filtering of the point cloud whose surfaces are created (170);
texturing the surface-filtered point cloud and broadcasting the image (180);
if the new 3D key frame is not accepted, determining the skeleton information of the living and objects moving on the last generated 3D key frame together with the joints by using the current point cloud via the artificial intelligence algorithms running on the server (190);
animating the living organisms and the objects moving on the current key frame according to their joints, and broadcasting them for the monitoring devices; (200);
as long as the event continues, returning to the first process step.

### Claim 2 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of creating the point cloud from the grouped pixels, the server computes the horizontal, vertical and depth axis (X, Y and Z axis) coordinates of each pixel by using the focus land and principal point data, and pixel depth information of each depth-sensing camera, and then creates the point cloud according to the coordinates.

### Claim 3 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of aligning the point cloud and obtaining the first key point cloud, the server locates the common points detected by the depth-sensing camera pairs and performs alignment by overlapping the detected common points.

### Claim 4 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of accepting the point cloud as the key point cloud or not, generating a new 3D key frame with the point cloud image in every 2-5 seconds, and using the last 3D key frame within the time period wherein a 3D key frame is not generated.

### Claim 5 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of accepting the point cloud as the key point cloud or not, performing the steps of point cloud filtering, mesh creation, surface filtering and texturing respectively in order to generate a new key frame with the point cloud image in every 2-5 seconds.

### Claim 6 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of accepting the point cloud as the key point cloud or not, the step of using the last 3D key frame within the time period wherein 3D key frame is not generated and determining the skeleton information of the living organisms and objects moving on the 3D key frame together with the joints, and the step of animating the living organisms and objects moving on the current key image according to their joints and broadcasting the image.

### Claim 7 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of determining the skeleton information of the moving living organisms and objects together with the joints, the server determines the skeleton structure of the people, living organisms and objects moving on the key frame.

### Claim 8 (depends on 1)

Three-dimensional media streaming and broadcasting method according to claim 1, wherein, in the step of animating the living organisms and objects moving on the key image according to their joints, the server moving only the moving objects from their joints on the key frame and broadcasting the said image.

### Claim 9

Three-dimensional media streaming and broadcasting system (1), which enables different environments, such as concerts and sports events, to be streamed live and in three dimensions, and the streamed image to be broadcast and viewed by monitoring devices such as virtual reality goggles, comprising:
camera sets, which enable to capture images from the event venue in three dimensions, and includes at least one depth-sensing camera, at least one color camera and at least one computing module;
depth-sensing camera, which receives depth information for each pixel, captures multiple pixel images at up to 30 frames per second, and is adapted to enable detection of the depths of the images of the venue and all people and objects present in the scene, where digital recording is performed, separately for each pixel;
a color camera which is adapted to enable capturing colored image of the scene that is recorded and the entire site that is captured at high resolution and high frame rate (fbs: frame per second);
a computing module with GPU capability, which is adapted to capture and compress the depth and color images received from the depth-sensing camera and the color camera simultaneously and after synchronizing, transmitting them to the server via Internet connection;
a server which is adapted to generate image frames by grouping the images coming from each camera set within the same time interval in terms of depth and color;
server which is adapted to compute the X, Y and Z axis coordinates of each pixel coming from the camera set and create the point cloud from the grouped pixels, and after the point cloud is created, in accordance with the camera set chosen as reference, to rotate the images of the other camera cameras according to the reference camera set and aligning the point cloud, and thereby obtaining the first key frame;
server which is adapted to continuously renew the key frame within a preferred time period with the new images coming from the camera sets, and when new images are received from the camera set, to control whether they came within the period of key frame renewal time;
server which, when the new images are received, is adapted to create the surface of the point cloud (mesh creation) if a new key frame is to be obtained, or if the old frame will be accepted as the point cloud, to determine the skeleton information of the living and objects moving on the key frame together with their joints and animating the living organisms and the objects moving on the current key frame according to their joints and to broadcasting the image.

### Claim 10 (depends on 9)

Three-dimensional media streaming and broadcasting system according to claim 9, characterized by the server which is adapted to compute the horizontal, vertical and depth axis (X, Y and Z axis) coordinates of each pixel by using the focus land and principal point data, and pixel depth information of each camera set, and then to create the point cloud according to the coordinates.

### Claim 11 (depends on 9)

Three-dimensional media streaming and broadcasting system according to claim 9, characterized by the server which is adapted to determine the skeleton structure of the people, living organisms and objects moving on the key frame via the artificial intelligence algorithms running thereon.
