# US 2018/0025230 A9 · Method and System for Motion Vector-Based Video Monitoring and Event Categorization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025230 A9
- Document ID: US20180025230A9
- Status: Published application
- Kind: A9
- Application number: 14/821,597
- Filed: 2015-08-07
- Published: 2018-01-25
- Earliest claimed: 2014-07-07
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Jason N. Laska; Gregory R. Nelson; Greg Duffy
- Web page: https://patentgrep.com/patent/US20180025230A9

## CPC

- H04L 63/0428: {wherein the data content is protected, e.g. by encrypting or encapsulating the payload}
- G06F 3/048: Interaction techniques based on graphical user interfaces [GUI]
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0485: Scrolling or panning
- G06F 3/04855: Interaction with scrollbars
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 16/447: {Temporal browsing, e.g. timeline}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G08B 13/194: using image scanning and comparing systems
- G08B 13/196: using television cameras
- G08B 13/19604: {involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change}
- G08B 13/19606: {Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan}
- G08B 13/19613: {Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion}
- G08B 13/19615: {wherein said pattern is defined by the user}
- G08B 13/19676: {Temporary storage, e.g. cyclic memory, buffer storage on pre-alarm}
- G08B 13/19682: {Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system}
- G08B 13/19684: {Portable terminal, e.g. mobile phone, used for viewing video remotely}
- G11B 27/005: {Reproducing at a different information rate from the information rate of recording (for television signals H04N5/783)}
- G11B 27/028: with computer assistance
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/105: {of operating discs}
- G11B 27/30: on the same track as the main recording
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- H04L 9/0822: {using key encryption key}
- H04L 9/085: {Secret sharing or secret splitting, e.g. threshold schemes}
- H04L 9/0869: {involving random numbers or seeds}
- H04L 63/083: {using passwords (cryptographic mechanisms or cryptographic arrangements for entity authentication using a predetermined code H04L9/3226)}
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04N 5/144: {Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20)}
- H04N 5/93: Regeneration of the television signal or of selected parts thereof
- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N 7/183: {for receiving images from a single remote source}
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- H04N 7/186: {Video door telephones}
- H04N 9/87: Regeneration of colour television signals (H04N9/80 takes precedence)
- H04N 21/2187: Live feed
- H04N 21/239: Interfacing the upstream path of the transmission network, e.g. prioritizing client {content} requests (hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60)
- H04N 21/2393: {involving handling client requests (scheduling or organising the servicing of application requests H04L67/60)}
- H04N 21/2743: Video hosting of uploaded data from client
- H04N 21/4222: {Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy (remote control device for a television receiver integrated into a mobile phone H04M1/72415)}
- H04N 21/431: Generation of visual interfaces {for content selection or interaction}; Content or additional data rendering
- H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations}
- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- H04N 21/4334: {Recording operations (recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27)}
- H04N 21/4335: Housekeeping operations, e.g. prioritizing content for deletion because of storage space restrictions {(storage management, e.g. defragmentation G06F3/0604; unloading stored programs G06F9/445; storage management in file systems G06F16/10; buffering arrangements in a network node or in an end terminal in packet networks H04L49/90)}
- H04N 21/4438: {Window management, e.g. event handling following interaction with the user interface}
- H04N 21/4622: {Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet (web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859)}
- H04N 23/6811: {based on the image signal}
- H04N 23/6815: {by distinguishing pan or tilt from motion}
- H04W 12/02: Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
- H04W 12/033: of the user plane, e.g. user's traffic
- H04W 12/04: Key management, e.g. using generic bootstrapping architecture [GBA]
- H04W 12/06: Authentication
- H04W 12/08: Access security
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30232: Surveillance
- G06V 20/44: {Event detection}
- H04L 2209/80: Wireless
- H04L 2463/062: applying encryption of the keys
- H04N 21/2347: involving video stream encryption
- H04N 21/2541: {Rights Management (protecting software against unauthorised usage in a vending or licensing environment G06F21/10; security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08)}
- H04N 21/42204: {User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor (constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205)}
- H04N 21/4314: {for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid}
- H04N 21/4408: involving video stream encryption, e.g. re-encrypting a decrypted video stream for redistribution in a home network
- H04N 21/4627: Rights management {associated to the content (security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08)}
- H04N 21/4753: {for user identification, e.g. by entering a PIN or password (cryptographic authentication protocols H04L9/32; networks authentication protocols H04L63/08)}
- H04N 23/65: Control of camera operation in relation to power supply
- H04N 23/651: {for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera}
- H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
- H04W 12/50: Secure pairing of devices
- H04W 84/12: WLAN [Wireless Local Area Networks]
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A computer system processes a video stream to detect a start of a first motion event candidate in the video stream, and in response to detecting the start of the first motion event candidate in the video stream, initiates event recognition processing on a first video segment associated with the start of the first motion event candidate. Initiating the event recognition processing further includes: determining a motion track of a first object identified in the first video segment; generating a representative motion vector for the first motion event candidate based on the motion track of the first object; and sending the representative motion vector for the first motion event candidate to an event categorizer, where the event categorizer assigns a respective motion event category to the first motion event candidate based on the representative motion vector of the first motion event candidate.

## Claims

### Claim 1

A method of processing a video stream, comprising:
processing the video stream to detect a start of a first motion event candidate in the video stream, wherein processing comprises:
obtaining a profile of a motion pixel count for a current frame sequence in the video stream;
in response to determining that the obtained profile meets a predetermined trigger criterion, determining that the current frame sequence includes a motion event candidate;
identifying a beginning time for a portion of the profile meeting the predetermined trigger criterion; and
designating the identified beginning time to be the start of the first motion event candidate; and
in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining that the obtained profile meets the predetermined trigger criterion includes determining that the motion pixel count satisfies a threshold motion pixel count.

### Claim 3 (depends on 1)

The method of claim 1, wherein the start of the first motion event candidate is the time at which the motion pixel count meets the predetermined trigger criterion.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining that the obtained profile meets the predetermined trigger criterion is with respect to a predetermined length of the current frame sequence.

### Claim 5 (depends on 1)

The method of claim 1, further comprising adjusting the predetermined trigger criterion over time based on a performance feedback.

### Claim 6 (depends on 5)

The method of claim 5, wherein the performance feedback indicates a number of false-positive motion events detected in processing the video stream.

### Claim 7 (depends on 6)

The method of claim 6, wherein the adjusting comprises increasing the predetermined trigger criterion when the number of false positives satisfies a corresponding threshold.

### Claim 8 (depends on 5)

The method of claim 5, wherein the performance feedback indicates a number of motion events that failed to be detected in processing the video stream.

### Claim 9 (depends on 8)

The method of claim 8, wherein the adjusting comprises decreasing the predetermined trigger criterion when the number of motion events that failed to be detected satisfies a corresponding threshold.

### Claim 10

A computing system for processing a video stream, comprising:
one or more processors; and
memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
processing the video stream to detect a start of a first motion event candidate in the video stream, wherein processing comprises:
obtaining a profile of a motion pixel count for a current frame sequence in the video stream;
in response to determining that the obtained profile meets a predetermined trigger criterion, determining that the current frame sequence includes a motion event candidate;
identifying a beginning time for a portion of the profile meeting the predetermined trigger criterion; and
designating the identified beginning time to be the start of the first motion event candidate; and
in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the operations further comprise adjusting the motion pixel count based on a camera state change that causes a change in pixel values in the current frame sequence, wherein the change in pixel values does not correspond to the occurrence of motion in the current frame sequence.

### Claim 12 (depends on 11)

The computing system of claim 11, wherein the operations further comprise suppressing designation of the identified beginning time as the start of the first motion event candidate in accordance with the start of the first motion event candidate overlapping with a time corresponding to the camera state change.

### Claim 13 (depends on 11)

The computing system of claim 11, wherein adjusting the motion pixel count comprises adjusting the motion pixel count by a fraction.

### Claim 14 (depends on 11)

The computing system of claim 11, wherein adjusting the motion pixel count comprises resetting the motion pixel count after the camera state change.

### Claim 15 (depends on 10)

The computing system of claim 10, wherein processing the video stream further comprises:
detecting a rate of change for the motion pixel count of the profile; and
if the detected rate of change satisfies a predetermined rate, suppressing designation of the identified beginning time as the start of the first motion event candidate.

### Claim 16 (depends on 15)

The computing system of claim 15, wherein the operations further comprise, after the suppressing:
executing a recovery process to determine whether the identified beginning time corresponds to a valid motion event candidate; and
in accordance with determining that the identified beginning time corresponds to a valid motion event candidate, recovering the first motion event candidate by designating the identified beginning time to be the start of the first motion event candidate.

### Claim 17 (depends on 15)

The computing system of claim 15, wherein the operations further comprise resetting the motion pixel count if the detected rate of change satisfies the predetermined rate.

### Claim 18

A method of processing a video stream, comprising:
processing the video stream to detect a start of a first motion event candidate in the video stream; and
in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate, wherein initiating the event recognition processing further comprises:
determining a motion track of a first object identified in the first video segment; and
generating a representative motion vector for the first motion event candidate based on the motion track of the first object, wherein a respective motion event category is assigned to the first motion event candidate based on the representative motion vector of the first motion event candidate;
wherein determining the motion track of the object identified in the first video segment further comprises, based on a frame sequence of the first video segment:
building a histogram that specifies a frame count for pixel locations in a scene of the first video segment;
segmenting the filtered histogram into the one or more motion regions; and
selecting one or more dominant motion regions from the one or more motion regions based on a predetermined dominance criterion, wherein at least one of the one or more dominant motion regions corresponds to the respective motion track of the object.

### Claim 19 (depends on 18)

The method of claim 18, wherein the frame count for a respective pixel location is a sum of the motion pixel count at the respective pixel location in frames of the first video segment.

### Claim 20 (depends on 19)

The method of claim 19, further comprising creating an event mask that includes one or more of the pixel locations having less than a threshold motion pixel count.

### Claim 21 (depends on 20)

The method of claim 20, wherein the threshold motion pixel count is with respect to a predefined number of frames of the first video segment.

### Claim 22 (depends on 20)

The method of claim 20, wherein one or more of the pixel locations not included in the event mask include at least the one or more dominant motion regions.

### Claim 23 (depends on 20)

The method of claim 20, wherein the event mask is a non-binary mask.

### Claim 24 (depends on 18)

The method of claim 18, wherein the predetermined dominance criterion is a threshold frame count.

### Claim 25 (depends on 18)

The method of claim 18, wherein the predetermined dominance criterion is a threshold motion pixel count.

### Claim 26

A computing system for processing a video stream, comprising:
one or more processors; and
memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
processing the video stream to detect a start of a first motion event candidate in the video stream; and
in response to detecting the start of the first motion event candidate in the video stream, initiating event recognition processing on a first video segment associated with the start of the first motion event candidate, wherein initiating the event recognition processing further comprises:
determining a motion track of a first object identified in the first video segment; and
generating a representative motion vector for the first motion event candidate based on the motion track of the first object, wherein a respective motion event category is assigned to the first motion event candidate based on the representative motion vector of the first motion event candidate;
wherein determining the motion track of the object identified in the first video segment further comprises, based on a frame sequence of the first video segment:
building a histogram that specifies a frame count for pixel locations in a scene of the first video segment;
segmenting the filtered histogram into the one or more motion regions; and
selecting one or more dominant motion regions from the one or more motion regions based on a predetermined dominance criterion, wherein at least one of the one or more dominant motion regions corresponds to the respective motion track of the object.

### Claim 27 (depends on 26)

The computing system of claim 26, wherein the operations further comprise determining a track length of the detected motion track.

### Claim 28 (depends on 27)

The computing system of claim 27, wherein the operations further comprise suppressing the motion track of the first object in accordance with the determined track length satisfying a predetermined threshold.

### Claim 29 (depends on 28)

The computing system of claim 28, wherein the suppression of the motion track occurs before generating the representative motion vector, and wherein the operations further comprise foregoing generation of the representative motion vector in accordance with suppression of the motion track.

### Claim 30 (depends on 26)

The computing system of claim 26, wherein determining the motion track of the object further comprises performing object segmentation to identify one or more foreground objects in the first video segment.

### Claim 31 (depends on 30)

The computing system of claim 30, wherein the one or more foreground objects identified in the first video segment include the object for which the motion track is determined.