# US 10,321,123 B2 · Gaze correction of multi-view images > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,321,123 B2 - Document ID: US10321123B2 - Status: Granted - Kind: B2 - Application number: 15/397,951 - Filed: 2017-01-04 - Published: 2017-07-06 - Granted: 2019-06-11 - Earliest claimed: 2016-01-05 - Assignees: RealD Spark, LLC - Applicants: RealD Spark, LLC - Inventors: Eric Sommerlade; Michael G. Robinson - Web page: https://patentgrep.com/patent/US10321123B2 ## CPC - G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06V 40/193: {Preprocessing; Feature extraction} - H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)} - H04N 7/15: Conference systems - H04N 13/344: with head-mounted left-right displays - H04N 13/117: the virtual viewpoint locations being selected by the viewers or determined by viewer tracking - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Gaze is corrected by adjusting multi-view images of a head. Image patches containing the left and right eyes of the head are identified and a feature vector is derived from plural local image descriptors of the image patch in at least one image of the multi-view images. A displacement vector field representing a transformation of an image patch is derived, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector produced by machine learning. The multi-view images are adjusted by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. ## Claims ### Claim 1 A device for adjusting multi-view images of a head to correct gaze, the device comprising an image processor arranged to process the multi-view images of a head by: in each image, identifying image patches containing the left and right eyes of the head, respectively; in respect of the image patches containing the left eyes of the head in each image of the multi-view images, and also in respect of the image patches containing the right eyes of the head in each image of the multi-view images, performing the steps of: deriving a feature vector from plural local image descriptors of the image patch in at least one image of the multi-view images; and deriving a displacement vector field representing a transformation of an image patch, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; and adjusting each image of the multi-view images by transforming the image patches containing the left and right eyes of the head in accordance with the derived displacement vector field. ### Claim 2 (depends on 1) A device according to claim 1, further comprising a camera system arranged to capture the multi-view images. ### Claim 3 (depends on 1) A device according to claim 1, further comprising a telecommunications interface arranged to transmit the adjusted images over a telecommunications network to a destination device for display thereon. ### Claim 4 (depends on 1) A device according to claim 1, wherein the multi-view images are a stereoscopic pair of images. ### Claim 5 (depends on 4) A device according to claim 4, wherein said plural local image descriptors are plural local image descriptors in both images of the stereoscopic pair; and wherein the image processor is further arranged to process the multi-view images such that the deriving of the displacement vector fields further comprises: deriving an initial displacement vector field representing a notional transformation of a notional image patch in a notional image having a notional camera location relative to the camera locations of the multi-view images, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; and deriving displacement vector fields representing a transformation of the image patches in each image of the multi-view images by transforming the initial displacement vector field in accordance with an estimate of the optical flows between the notional image patches in the notional images and the image patches in the images of the multi-view images. ### Claim 6 (depends on 5) A device according to claim 5, wherein the notional camera location is between the camera locations of the images of the stereoscopic pair. ### Claim 7 (depends on 1) A device according to claim 1, wherein said plural local image descriptors are plural local image descriptors in each images of the multi-view images, and said reference data comprising pairs of reference displacement vector fields for each image of the multi-view images, which pairs of reference displacement vector fields are associated with possible values of the feature vector. ### Claim 8 (depends on 1) A device according to claim 1, wherein said plural local image descriptors are plural local image descriptors in one image of the multi-view images; and wherein the image processor is further arranged to process the multi-view images such that the deriving of the displacement vector fields further comprises: deriving a displacement vector field representing a transformation of the image patch in said one image of the multi-view images, using the derived feature vector to look up reference data comprising reference displacement vector fields associated with possible values of the feature vector; and deriving a displacement vector field representing a transformation of the image patch in the other multi-view image or images by transforming the derived displacement vector field representing a transformation of the image patch in said one image of the multi-view images in accordance with an estimate of the optical flow between the image patches in said one image of the multi-view images and the other multi-view image or images. ### Claim 9 (depends on 1) A device according to claim 1, wherein the image processor is further arranged to process the multi-view images such that: the deriving of the displacement vector fields further comprises deriving displacement vector fields in respect of the image patches in each image of the multi-view images, and the transforming of the image patches containing the left and right eyes of the head is performed in accordance with the displacement vector fields derived in respect of the image patches in each image of the multi-view images. ### Claim 10 (depends on 1) A device according to claim 1, wherein the image processor is further arranged to process the multi-view images such that: the deriving of the displacement vector field further comprises deriving displacement vector fields in respect of the image patches in each image of the multi-view images, and further deriving a merged displacement vector field from the displacement vector fields derived in respect of the image patches in each image of the multi-view images, and the transforming of the image patches containing the left and right eyes of the head is performed in accordance with the merged displacement vector field. ### Claim 11 (depends on 10) A device according to claim 10, wherein the reference displacement vector fields are further associated with confidence values; wherein the image processor is further arranged to process the multi-view images such that the deriving of the displacement vector fields further comprises: deriving a confidence value associated with each derived displacement vector field, and deriving the merged displacement vector field as an average of the displacement vector fields derived in respect of the image patches in each image of the multi-view images weighted by the derived confidence values. ### Claim 12 (depends on 1) A device according to claim 1, wherein the image processor is further arranged to process the multi-view using location data representing an observer location relative to a display location of the multi-view images; wherein said reference data comprises plural sets of reference displacement vector fields associated with possible values of the feature vector, which sets are associated with different observer locations; and wherein the image processor is further arranged to process the multi-view images such that the deriving of the displacement vector fields further comprises using the derived feature vector to look up the set of reference displacement vector fields associated with the observer location represented by the location data. ### Claim 13 (depends on 1) A device according to claim 1, wherein the local image descriptors comprise local image descriptors derived in a normalized coordinate system, and the reference and derived displacement vector fields are displacement vector fields in the same normalized coordinate system. ### Claim 14 (depends on 1) A device according to claim 1, wherein the feature vector comprises features that are values derived from the local image descriptors in a discrete space, the reference displacement vector fields comprise a reference displacement vector field associated with each possible value of the feature vector in the discrete space, and the image processor is further arranged to process the multi-view images such that the deriving of the displacement vector fields further comprises selecting the reference displacement field associated with the actual value of the derived feature vector. ### Claim 15 (depends on 1) A device according to claim 1, wherein the feature vector comprises features that are discrete values of the local image descriptors in a continuous space, and the image processor is further arranged to process the multi-view images such that the deriving of the displacement vector fields further comprises interpolating a displacement vector field from the reference displacement vector fields based on the relationship between the actual value of the derived feature vector and the values of the feature vector associated with the reference displacement vector fields.