# US 12,217,368 B2 · Extended field of view generation for split-rendering for virtual reality streaming

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,368 B2
- Document ID: US12217368B2
- Status: Granted
- Kind: B2
- Application number: 17/532,942
- Filed: 2021-11-22
- Published: 2022-06-02
- Granted: 2025-02-04
- Earliest claimed: 2020-12-01
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Madhukar Budagavi; Youngkwon Lim; Ravindranath Kanamangala
- Web page: https://patentgrep.com/patent/US12217368B2

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- H04N 19/597: specially adapted for multi-view video sequence encoding
- H04N 21/21805: {enabling multiple viewpoints, e.g. using a plurality of cameras}
- H04N 21/234345: {the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment}
- H04N 21/234363: {by altering the spatial resolution, e.g. for clients with a lower screen resolution}
- H04N 21/4126: {The peripheral being portable, e.g. PDAs or mobile phones}
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/42202: {environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes}
- H04N 21/44213: {Monitoring of end-user related data (arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33)}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N 21/6332: directed to client
- H04N 21/6587: Control parameters, e.g. trick play commands, viewpoint selection
- H04N 21/816: {involving special video data, e.g 3D video}
- G06V 20/44: {Event detection}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An electronic device, a client device, and a method for split rendering video content are disclosed. The method includes receiving user data including viewpoint and motion data associated with a client device. The method also includes identifying a first portion of the video content corresponding to a field of view (FoV) of the client device based on the viewpoint. The method further includes identifying speed and direction of the client device based on the motion data. Additionally, the method includes identifying, based on the speed and the direction, a second portion of the video content extending from the first portion of the video content and expanding the FoV. The method also includes scaling the second portion of the video content using a scaling factor that is determined based on the speed. The method further includes transmitting the first and second portions of the video content and the scaling factor.

## Claims

### Claim 1

An electronic device comprising:
a communication interface configured to receive user data including viewpoint data and motion data associated with a client device;
a processor operably coupled to the communication interface, the processor configured to:
identify a first portion of video content corresponding to a field of view (FoV) of the client device based on the viewpoint data,
identify a speed and a direction of the client device based on the motion data,
identify, based on the speed and the direction, a second portion of the video content extending from the first portion of the video content and expanding the FoV, and
compare the speed to a threshold speed;
scale a spatial resolution of the second portion of the video content using a scaling factor that is determined based on the comparison of the speed to the threshold speed; and
the communication interface is further configured to transmit the first portion of the video content, the scaled second portion of the video content, and the scaling factor to the client device, wherein the first portion of the video content and the scaled second portion of the video content are disposed within a same video frame.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the second portion of the video content extends asymmetrically along the direction of the motion data.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein the processor is further configured to:
set the scaling factor to that of the first portion of the video content based on the speed being less than the threshold speed.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the processor is further configured to:
set the scaling factor to apply additional scaling than that of the first portion of the video content based on the speed being greater than the threshold speed.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein:
the user data corresponds to multiple consecutive time instances;
the processor is further configured to:
identify the FoV based on the viewpoint data at a current time instance of the multiple consecutive time instances; and
identify the speed and the direction based on the motion data corresponding to a set of consecutive time instances including the current time instance of the multiple consecutive time instances.

### Claim 6 (depends on 1)

The electronic device of claim 1, wherein:
the video content is three dimensional extended reality content; and
the processor is further configured to:
render the three dimensional extended reality content into two-dimensional video frames,
encode the first and second portions using the two-dimensional video frames, and
transmit the two-dimensional video frames to the client device.

### Claim 7 (depends on 1)

The electronic device of claim 1, wherein the processor is further configured to:
identify an event that occurs within the video content based on obtained content information, the event located in a third portion of the video content;
determine whether to transmit the third portion of the video content based on the content information; and
in response to determining to transmit the third portion of the video content, cause the communication interface to transmit the first, second and third portions of the video content to the client device.

### Claim 8 (depends on 7)

The electronic device of claim 7, wherein:
the user data includes preferences of a user of the client device; and
the processor is further configured to:
determine whether to transmit the third portion of the video content based on the content information and the preferences, and
in response to determining to transmit the third portion of the video content, cause the communication interface to transmit the first, second and third portions of the video content to the client device.

### Claim 9

A method for split rendering of video content, the method comprises:
receiving user data including viewpoint data and motion data associated with a client device;
identifying a first portion of the video content corresponding to a field of view (FoV) of the client device based on the viewpoint data;
identifying a speed and a direction of the client device based on the motion data;
identifying, based on the speed and the direction, a second portion of the video content extending from the first portion of the video content and expanding the FoV;
comparing the speed to a threshold speed;
scaling a spatial resolution of the second portion of the video content using a scaling factor that is determined based on the comparison of the speed to the threshold speed; and
transmitting the first portion of the video content, the scaled second portion of the video content, and the scaling factor to the client device, wherein the first portion of the video content and the scaled second portion of the video content are disposed within a same video frame.

### Claim 10 (depends on 9)

The method of claim 9, wherein the second portion of the video content extends asymmetrically along the direction of the motion data.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
setting the scaling factor to that of the first portion of the video content based on the speed being less than the threshold speed.

### Claim 12 (depends on 9)

The method of claim 9, further comprising:
setting the scaling factor to apply additional scaling than that of the first portion of the video content based on the speed being greater than the threshold speed.

### Claim 13 (depends on 9)

The method of claim 9, wherein:
the user data corresponds to multiple consecutive time instances;
identifying the FoV is based on the viewpoint data at a current time instance of the multiple consecutive time instances; and
identifying the speed and the direction is based on the motion data corresponding to a set of consecutive time instances including the current time instance of the multiple consecutive time instances.

### Claim 14 (depends on 9)

The method of claim 9, wherein:
the video content is three dimensional extended reality content; and
the method further comprises:
rendering the three dimensional extended reality content into two-dimensional video frames
encoding the first and second portions using the two-dimensional video frames, and
transmitting the two-dimensional video frames to the client device.

### Claim 15 (depends on 9)

The method of claim 9, further comprising:
identifying an event that occurs within the video content based on obtained content information, the event located in a third portion of the video content;
determining whether to transmit the third portion of the video content based on the content information; and
in response to determining to transmit the third portion of the video content, transmitting the first, second and third portions of the video content to the client device.

### Claim 16 (depends on 15)

The method of claim 15, wherein:
the user data includes preferences of a user of the client device; and
the method further comprises
determining whether to transmit the third portion of the video content based on the content information and the preferences, and
in response to determining to transmit the third portion of the video content, transmitting the first, second and third portions of the video content to the client device.