Back to results
US 11,694,419 B2Granted
Image analysis and gaze redirection using characteristics of the eye
AssigneeKickback Space Inc.
InventorRocco Haro
Life of this patent
Estimating the term…
Abstract
In a method for performing gaze redirection, a video stream of a user is received, each video frame from the video stream including a depiction of a gaze of the user. For each video frame from the video stream, an estimated gaze direction of the gaze of the user in that video frame is determined. A field of view (FOV) for a virtual representation associated with the user in a virtual environment is determined. An updated video frame including a modified gaze direction of the user different from the estimated gaze direction of the gaze of the user is generated.
Claims20
A method, comprising:
- receiving, via a processor of a first compute device and from a second compute device, a signal representing eye data associated with at least one eye of a first user;
- determining, via the processor and in response to receiving the signal representing the eye data, that the eye data is sufficient to perform gaze direction correction for the first user;
- sending, via the processor and to the second compute device, a signal indicating that the eye data is sufficient to perform the gaze direction correction;
- receiving, via the processor and from the second compute device, a signal representing a first video frame of the first user;
- estimating, via the processor and using the eye data, a gaze direction of the first user in the first video frame;
- determining, via the processor, a field of view of a first virtual representation of the first user in a virtual environment, the first virtual representation (1) based on an appearance of the first user and (2) controllable by the first user;
- comparing, via the processor, the gaze direction of the first user in the first video frame and the field of view of the first virtual representation, to predict a target gaze direction for the first user;
- inputting, via the processor, representations of the first video frame, the gaze direction of the first user, the target gaze direction, and a normalizing factor into a processing pipeline to generate a second video frame, a gaze direction of the first virtual representation in the second video frame being different than the gaze direction of the first user in the first video frame;
- generating, via the processor and using the second video frame, a modified video frame that represents the first virtual representation from a perspective of a second virtual representation in the virtual environment;
- and
- causing, via the processor, the modified video frame to be displayed in the virtual environment, at a third compute device, and to a second user associated with the second virtual representation.
8 more elements
An apparatus, comprising:
- a memory;
- and
- a processor operatively coupled to the memory, the processor configured to:
- receive, at a first compute device and from a second compute device, a signal representing a first video frame of a first user captured at a first time;
- estimate a first gaze direction of the first user in the first video frame;
- determine a first field of view of a first virtual representation of the first user in a virtual environment, the first virtual representation based on an appearance of the first user, the first field of view including a second virtual representation included in the virtual environment and not including a third virtual representation included in the virtual environment;
- generate a second video frame that shows the first virtual representation looking at the second virtual representation and not looking at the third virtual representation;
- send, at a second time subsequent to the first time, the second video frame to a third compute device to cause the third compute device to display the second video frame within the virtual environment;
- receive, at the first compute device and from the second compute device, a signal representing a third video frame of the first user captured at a third time;
- estimate a second gaze direction of the first user in the second video frame;
- determine a second field of view of a first virtual representation in the virtual environment, the second field of view including the third virtual representation and not the second virtual representation;
- generate a fourth video frame that shows the first virtual representation looking at the third virtual representation and not the second virtual representation;
- and
- send, at a fourth time subsequent to the third time, the fourth video frame to the third compute device to cause the third compute device to display the fourth video frame within the virtual environment.
11 more elements
A non-transitory, processor-readable medium storing code representing instructions executable by a processor, the code comprising code to cause the processor to:
- receive a video stream of a user, each video frame from the video stream including a depiction of a gaze of the user;
- and
- for each video frame from the video stream:
- determine, substantially in real time as that video frame is received, an estimated gaze direction of the gaze of the user in that video frame;
- determine a field of view for a virtual representation associated with the user in a virtual environment;
- generate an updated video frame including a modified gaze direction of the user different from the gaze direction of the user;
- and
- cause the updated video frame to be displayed.
5 more elements
Dependencies are inferred from the claim text.