# US 10,372,994 B2 · Method, system and apparatus for selecting a video frame > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,994 B2 - Document ID: US10372994B2 - Status: Granted - Kind: B2 - Application number: 15/585,020 - Filed: 2017-05-02 - Published: 2017-11-16 - Granted: 2019-08-06 - Earliest claimed: 2016-05-13 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Sammy Chan; Ka Ming Leung; Mark Ronald Tainsh - Web page: https://patentgrep.com/patent/US10372994B2 ## CPC - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/47: {Detecting features for summarising video content} - G06V 40/168: {Feature extraction; Face representation} - G06V 40/18: Eye characteristics, e.g. of the iris - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method of selecting at least one video frame of a video sequence. A plurality of faces is detected in at least one video frame of the video sequence. An orientation of the detected faces is tracked over a series of subsequent video frames to determine whether a first detected face is turning towards a second detected face. The method then determines, using the tracked orientation of the detected faces, a portion of the video sequence in which the first and second detected faces are oriented towards each other for at least a predetermined number of frames defining a gaze fixation of the detected faces. At least one video frame is selected from the determined portion of the video sequence, the selected video frame capturing the gaze fixation of the detected faces. ## Claims ### Claim 1 A method of selecting at least one video frame of a video sequence in a database of video sequences, the method comprising: detecting, by a processor, a plurality of faces in at least one video frame of the video sequence, wherein the plurality of faces detected in the at least one video frame is determined to exceed a predetermined size threshold based on pixel size of a detected face relative to the at least one video frame; tracking, by the processor, an orientation of the detected faces over a series of subsequent video frames to determine whether a first detected face is turning towards a second detected face, wherein the orientation of the detected faces is relative to a camera; determining, by the processor, using the tracked orientation of the detected faces, a portion of the video sequence in which the first and second detected faces are oriented towards each other for at least a predetermined number of frames defining a gaze fixation of the detected faces; and selecting, by the processor, at least one video frame from the determined portion of the video sequence in the database of video sequences, said selected video frame capturing the gaze fixation of the detected faces. ### Claim 2 (depends on 1) The method according to claim 1, wherein the first detected face is turning towards the second detected face while the second detected face remains stationary. ### Claim 3 (depends on 1) The method according to claim 1, wherein the gaze fixation is dependent on a threshold period. ### Claim 4 (depends on 1) The method according to claim 1, wherein the gaze fixation is dependent on a minimum turning period. ### Claim 5 (depends on 1) The method according to claim 1, wherein faces outside a centre region of the selected frame are ignored. ### Claim 6 (depends on 1) The method according to claim 1, further comprising printing the selected frame in a photobook. ### Claim 7 (depends on 1) The method according to claim 1, further comprising displaying the selected frame in a summary representation of the video sequence. ### Claim 8 (depends on 1) The method according to claim 1, wherein the first detected face is oriented forward relative to the second face and the second face is oriented backward relative to the first face in the frames of the portion of the video sequence. ### Claim 9 (depends on 1) The method according to claim 1, wherein detecting the plurality of faces further comprises detecting the plurality of faces having the same depth in the at least one video frame. ### Claim 10 A system for selecting at least one video frame of a video sequence in a database of video sequences, the system comprising: a memory for storing data and a computer program; a processor coupled to the memory for executing the computer program, the computer program comprising instructions for: detecting a plurality of faces in at least one video frame of the video sequence, wherein the plurality of faces detected in the at least one video frame is determined to exceed a predetermined size threshold based on pixel size of a detected face relative to the at least one video frame; tracking an orientation of the detected faces over a series of subsequent video frames to determine whether a first detected face is turning towards a second detected face, wherein the orientation of the detected faces is relative to a camera; determining, using the tracked orientation of the detected faces, a portion of the video sequence in which the first and second detected faces are oriented towards each other for at least a predetermined number of frames defining a gaze fixation of the detected faces; and selecting at least one video frame from the determined portion of the video sequence in the database of video sequences, said selected video frame capturing the gaze fixation of the detected faces. ### Claim 11 An apparatus for selecting at least one video frame of a video sequence in a database of video sequences, the apparatus comprising: means for detecting a plurality of faces in at least one video frame of the video sequence, wherein the plurality of faces detected in the at least one video frame is determined to exceed a predetermined size threshold based on pixel size of a detected face relative to the at least one video frame; means for tracking an orientation of the detected faces over a series of subsequent video frames to determine whether a first detected face is turning towards a second detected face, wherein the orientation of the detected faces is relative to a camera; means for determining, using the tracked orientation of the detected faces, a portion of the video sequence in which the first and second detected faces are oriented towards each other for at least a predetermined number of frames defining a gaze fixation of the detected faces; and means for selecting at least one video frame from the determined portion of the video sequence in the database of video sequences, said selected video frame capturing the gaze fixation of the detected faces. ### Claim 12 A non-transitory computer readable medium having a computer program stored on the medium for selecting at least one video frame of a video sequence in a database of video sequences, the program comprising: code for detecting a plurality of faces in at least one video frame of the video sequence, wherein the plurality of faces detected in the at least one video frame is determined to exceed a predetermined size threshold based on pixel size of a detected face relative to the at least one video frame; code for tracking an orientation of the detected faces over a series of subsequent video frames to determine whether a first detected face is turning towards a second detected face, wherein the orientation of the detected faces is relative to a camera; code for determining, using the tracked orientation of the detected faces, a portion of the video sequence in which the first and second detected faces are oriented towards each other for at least a predetermined number of frames defining a gaze fixation of the detected faces; and code for selecting at least one video frame from the determined portion of the video sequence in the database of video sequences, said selected video frame capturing the gaze fixation of the detected faces.