# US 2020/0005079 A1 · Systems and Methods of Person Recognition in Video Streams > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0005079 A1 - Document ID: US20200005079A1 - Status: Published application - Kind: A1 - Application number: 16/565,365 - Filed: 2019-09-09 - Published: 2020-01-02 - Earliest claimed: 2017-05-30 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Akshay R. Bapat; George Alban Heitz; Rizwan Ahmed Chaudhry; Navneet Dalal; James Edward Stewart; Jennifer Bush; Joe Delone Venters; Kara Gates; Timothy Butler; Yohannes Kifle; Anton Bastov - Web page: https://patentgrep.com/patent/US20200005079A1 ## CPC - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - G06F 18/23: Clustering techniques - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/53: {Recognition of crowd images, e.g. recognition of crowd congestion} - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V 40/173: {face re-identification, e.g. recognising unknown faces across different face tracks} - G08B 13/19645: {Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over} - G08B 13/1966: {Wireless systems, other than telephone systems, used to communicate with a camera} - G08B 13/19663: {Surveillance related processing done local to the camera} - G08B 13/19697: {Arrangements wherein non-video detectors generate an alarm themselves} - G06F 16/45: Clustering; Classification - G06V 20/44: {Event detection} - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/168: {Feature extraction; Face representation} - G06V 40/172: {Classification, e.g. identification} - G06V 40/179: {metadata assisted face recognition} - G06V 2201/10: Recognition assisted with metadata - H04N 23/611: where the recognised objects include parts of the human body - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS ## Abstract The various implementations described herein include systems and methods for recognizing persons in video streams. In one aspect, a method includes: (1) obtaining a live video stream; (2) detecting person(s) in the stream; and (3) determining, from analysis of the live video stream, first information of the detected person(s); (4) determining, based on the first information, that the first person is not known to the computing system; (5) in accordance with the determination that the first person is not known: (a) storing the first information; and (b) requesting a user to classify the first person; and (6) in accordance with a determination that a response was received classifying the first person as a stranger, deleting the stored first information. ## Claims ### Claim 1 A method comprising: at a computing system having one or more processors and memory: obtaining a live video stream; detecting a first person in the live video stream; determining, from analysis of the live video stream, first information that identifies an attribute of the first person; determining, based on at least some of the first information, that the first person is not a known person to the computing system; in accordance with the determination that the first person is not a known person: storing at least some of the first information; and requesting a user to classify the first person; and in accordance with (i) a determination that a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, deleting the stored first information. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the first information comprises: selecting one or more images of the first person from the live video stream; and characterizing a plurality of features of the first person based on the one or more images. ### Claim 3 (depends on 2) The method of claim 2, further comprising: identifying a pose of the first person in each of the one or more images; and for each identified pose, in accordance with a determination that an image of the first person in the identified pose has not previously been stored, storing (i) the image of the first person in the identified pose, and (ii) corresponding pose information. ### Claim 4 (depends on 2) The method of claim 2, wherein the plurality of features includes one or more features that can personally identify the first person. ### Claim 5 (depends on 2) The method of claim 2, wherein the plurality of features includes one or more facial features of the first person. ### Claim 6 (depends on 1) The method of claim 1, further comprising: receiving, from the user, a designation that the first person is not a person; and responsive to the designation, storing the designation in association with the first information of the first person. ### Claim 7 (depends on 1) The method of claim 1, further comprising, in accordance with the determination that the first person is not a known person, generating a timely notification of detection of an unknown person. ### Claim 8 (depends on 1) The method of claim 1, further comprising: detecting a second person in the live video stream; characterizing a plurality of features of the second person based on one or more images of the second person in the live video stream; and determining whether the second person is a same person as the first person by comparing the plurality of characterized features of the second person with the stored first information regarding a plurality of characterized features of the first person. ### Claim 9 A computing system comprising: one or more processors; and memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: obtaining a live video stream; detecting a first person in the live video stream; determining, from analysis of the live video stream, first information that identifies an attribute of the first person; determining, based on at least some of the first information, whether the first person is a known person to the computing system; in accordance with a determination that the first person is not a known person: storing at least some of the first information; and requesting a user to classify the first person; and in accordance with (i) a determination that a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, deleting the stored first information. ### Claim 10 (depends on 9) The system of claim 9, wherein detecting the the first person comprises detecting the first person within a first predefined threshold time. ### Claim 11 (depends on 9) The system of claim 9, wherein the one or more programs further include instructions for comparing the first information of the first person with stored first information of the first person previously classified by the user; and wherein determining that the first person is not a known person comprises determining that the first information of the first person does not match, within a predetermined threshold, the stored first information of the first person previously classified by the user. ### Claim 12 (depends on 9) The system of claim 9, wherein the one or more programs further include instructions for: determining that the first person is a known person to the computing system; and in accordance with the determination that the first person is a known person, forgoing requesting the user to classify the first person. ### Claim 13 (depends on 12) The system of claim 12, wherein the one or more programs further include instructions for forgoing generating a timely notification in accordance with the determination that the first person is a known person. ### Claim 14 (depends on 12) The system of claim 12, wherein the one or more programs further include instructions for generating a timely notification identifying the first person in accordance with the determination that the first person is a known person. ### Claim 15 (depends on 14) The system of claim 14, wherein the one or more programs further include instructions for: determining a location of the first person; determining whether the first person has previously been authorized by the user to access the location; and wherein generating the timely notification identifying the first person comprises generating the timely notification identifying the first person and the location in accordance with a determination that the first person has not previously been authorized to access the location. ### Claim 16 A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to: obtain a live video stream; detect a first person in the live video stream; determine, from analysis of the live video stream, first information that identifies an attribute of the first person; determine, based on at least some of the first information, whether the first person is a known person to the computing system; in accordance with a determination that the first person is not a known person: store at least some of the first information; and request a user to classify the first person; and in accordance with (i) a determination that a response was not received from the user, or (ii) a determination that a response was received from the user classifying the first person as a stranger, delete the stored first information. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to: detect the first person by performing facial detection on a first frame in the live video stream; and track the first person through time by analyzing a plurality of video frames in the live video stream, the plurality of video frames subsequent to the first frame. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to: for each video frame of the first frame and the plurality of video frames, determine an identification quality factor for the video frame; select one or more video frames of the first frame and the plurality of video frames based on the determined identification quality factors; and wherein determining the first information comprises determining first information based on an analysis of the selected one or more images. ### Claim 19 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to: determine an activity of the first person based on the tracking of the first person; and send a notification with identification of the first person and the determined activity. ### Claim 20 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to provide a notification to the first person in accordance with the determination that the first person is not a known person.