LSTM based personalization view point estimation apparatus and method
AssigneeFdn. For Res.&Bus., Seoul Nat. Univ. Of Sci.&Tech.
InventorsDong Ho Kim, Bong-Seok Seo, Changjong Hyun
Life of this patent
Estimating the term…
Abstract
Disclosed are an LSTM based personalization viewpoint estimation apparatus and method. According to an embodiment of the present disclosure, a viewpoint of a user is estimated using LSTM with respect to a 360-degree raw image and a viewpoint image that is a region of interest of the user and a tile including the estimated viewpoint is transmitted over a network, so that the amount of data to transmit compared to the network bandwidth may be reduced and a viewpoint of each individual may be estimated.
Claims9
A Long Short Term Memory (LSTM) based personalization viewpoint estimation apparatus, comprising:
- a raw image acquisition unit acquiring 360-degree raw images of a predetermined number of frames through multiple cameras;
- a viewpoint image acquisition unit acquiring a user viewpoint image that is a region of interest of a user among the acquired raw images of the respective frames, through degrees of freedom (DoF) of a head mounted display (HMD);
- a preprocessing unit extracting a 1D feature of the raw image and the user viewpoint image for each frame by using a convolutional neural network algorithm, and combining the extracted features and user positions into a series of vectors to generate input data;
- an LSTM execution unit estimating a user viewpoint in a predetermined future by performing learning on the input data of the preprocessing unit;
- and
- a transmission unit transmitting tile information including the estimated user viewpoint of the future over a network.
A Long Short Term Memory (LSTM) based personalization viewpoint estimation method, comprising steps of:
- acquiring 360-degree raw images in an equirectangular projection (ERP) form of a predetermined number of frames through multiple cameras;
- acquiring a viewpoint image from the acquired raw images of the respective frames, through degrees of freedom (DoF) of a head mounted display (HMD);
- preprocessing for extracting a 1D feature of the raw image and the viewpoint image for each frame by using a convolutional neural network algorithm, and combining the extracted features and user positions into a series of vectors to generate input data;
- estimating a user viewpoint in a predetermined future by performing learning on the input data of the preprocessing step based on LSTM;
- and
- transmitting tile information including the estimated user viewpoint of the future over a network.
A non-transitory computer-readable recording medium having a program recorded thereon, the program being for executing a Long Short Term Memory (LSTM) based personalization viewpoint estimation method comprising steps of:
- acquiring 360-degree raw images in an equirectangular projection (ERP) form of a predetermined number of frames through multiple cameras;
- acquiring a viewpoint image from the acquired raw images of the respective frames, through degrees of freedom (DoF) of a head mounted display (HMD);
- preprocessing for extracting a 1D feature of the raw image and the viewpoint image for each frame by using a convolutional neural network algorithm, and combining the extracted features and user positions into a series of vectors to generate input data;
- estimating a user viewpoint in a predetermined future by performing learning on the input data of the preprocessing step based on LSTM;
- and
- transmitting tile information including the estimated user viewpoint of the future over a network.
Dependencies are inferred from the claim text.