# US 2017/0249500 A1 · Identification and de-identification within a video sequence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0249500 A1
- Document ID: US20170249500A1
- Status: Published application
- Kind: A1
- Application number: 15/595,441
- Filed: 2017-05-15
- Published: 2017-08-31
- Earliest claimed: 2012-01-04
- Assignees: AIC Innovations Group, Inc.
- Applicants: AIC Innovations Group, Inc.
- Inventors: Adam Hanina; Lei Guan
- Web page: https://patentgrep.com/patent/US20170249500A1

## CPC

- G16H 20/10: relating to drugs or medications, e.g. for ensuring correct administration to patients
- G06N 20/00: Machine learning
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/66: Trinkets, e.g. shirt buttons or jewellery items (recognising microscopic objects G06V20/69)
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 30/40: for processing medical images, e.g. editing
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and system for de-identifying a video sequence are provided. The method may include the steps of capturing a video sequence, comprising a number of individual frames, including one or more users performing one or more actions, and using activity recognition to recognize one of the one or more actions. One or more of the plurality of frames may be defined as comprising the recognized one or more actions, and a portion of the one or more of the plurality of frames may be identified to remain visible. The non-identified portions of the one or more of the plurality of frames and the non-defined frames may be de-identified. This method may be applied to the determine of whether a user has ingested a medication pill.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A method for de-identifying a video sequence, comprising:
capturing, in a video capture device, a video sequence comprising a plurality of individual frames, the plurality of individual frames including images of a desired action;
designating, by a processor, one or more frames of the plurality of frames as comprising a sequence of the desired action, wherein at least one of the plurality of frames is non-designated;
identifying a first portion of the one or more designated frames to remain visible, wherein the first portion comprises the desired action;
de-identifying a second portion of the one or more designated frames and de-identifying the at least one non-designated frame so that the first portion comprising the desired action remains visible; and
de-identifying a further portion of the first portion of the one or more designated frames determined to include one or more faces other than faces of the one or more users.

### Claim 22 (depends on 21)

The method of claim 21, wherein the de-identifying comprises blurring the image.

### Claim 23 (depends on 21)

The method of claim 21, further comprising highlighting the visible portions of the one or more plurality of frames.

### Claim 24 (depends on 21)

The method of claim 21, wherein the desired action comprises administration of medication.

### Claim 25 (depends on 21)

The method of claim 21, further comprising transmitting the de-identified one or more of the plurality of frames to a remote storage and computing location.

### Claim 26 (depends on 21)

The method of claim 21, further comprising performing, by a processor, activity recognition on the captured video sequence to recognize the desired activity within the plurality of individual frames;
wherein performing activity recognition to recognize the desired activity further comprises:
determining, through object recognition, the identity of an object for a first plurality of frames of the video sequence; and
determining, through object recognition, the identity of the object and its surroundings in a second plurality of frames of the video sequence.

### Claim 27 (depends on 21)

The method of claim 21, wherein the desired activity comprises micro movements of one or more body parts.

### Claim 28 (depends on 21)

The method of claim 21, further comprising performing, by a processor, activity recognition on the captured video sequence to recognize the desired activity within the plurality of individual frames;
wherein the performing activity recognition to recognize the desired activity further comprises:
defining a plurality of detection methods for recognizing the desired activity; and
applying each of the plurality of detection methods to a subset of the frames of the captured video sequence.

### Claim 29 (depends on 28)

The method of claim 28, wherein each detection method comprises searching for an object positioned at a different orientation.

### Claim 30

A method for de-identifying a video sequence of a user performing a desired activity, comprising the steps of:
capturing a first video sequence in a video capture device, the first video sequence comprising a plurality of individual frames, the plurality of individual frames including images of one or more users performing a desired activity;
designating, by a processor, one or more frames of the plurality of frames as comprising the desired activity, wherein at least one frame is non-designated;
identifying a first portion of the one or more designated frames to remain visible, wherein the first portion comprises the recognized desired activity; and
de-identifying a second portion of the one or more designated frames and de-identifying the at least one non-designated frame so that the first portion comprising the desired activity remains visible; and
de-identifying a further portion of the first portion of the one or more designated frames determined to include one or more faces other than faces of the one or more users.

### Claim 31 (depends on 30)

The method of claim 30, wherein the de-identifying comprises blurring the image.

### Claim 32 (depends on 30)

The method of claim 30, further comprising highlighting the visible portions of the one or more plurality of frames.

### Claim 33 (depends on 30)

The method of claim 30, further comprising transmitting the de-identified second portion and at least one non-designated frame to a remote storage and computing location.

### Claim 34 (depends on 30)

The method of claim 30, further comprising performing, by a processor, activity recognition on the captured video sequence to recognize the desired activity;
wherein performing activity recognition further comprises:
determining, through object recognition, the identity of an object for a first plurality of frames of the video sequence; and
determining, through object recognition, the identity of the object and its surroundings in a second plurality of frames of the video sequence.

### Claim 35 (depends on 30)

The method of claim 30, further comprising capturing a second video sequence, the second video sequence comprising a plurality of individual frames, the plurality of individual frames of the second video sequence including images of one or more users performing a second desired activity;
performing, by the processor, activity recognition on the second video sequence to recognize one or more micro movements of one or more body parts associated with the second desired activity by the user;
designating one or more frames of the second video sequence as comprising the recognized one or more micro movements of the one or more body parts of the user, wherein at least one frame of the second video sequence is non-designated;
identifying a first portion of the one or more frames of the second video sequence to remain visible, wherein first portion of the one or more frames of the second video sequence comprises the recognized one or more micro movements of the one or more body parts of the user; and
de-identifying a second portion of the one or more frames of the second video sequence and de-identifying the at least one non-designated frame of the second video sequence so that the first portion of the one or more frames of the second video sequence remains visible; and
de-identifying a further portion of the first portion of the one or more designated frames of the second video sequence determined to include one or more faces other than faces of the one or more users.

### Claim 36 (depends on 30)

The method of claim 30, further comprising performing, by a processor, activity recognition on the captured video sequence to recognize the second desired activity;
wherein performing activity recognition to recognize the performance of the second desired activity further comprises:
defining a plurality of detection methods for recognizing the second desired activity; and
applying each of the plurality of detection methods to a subset of the frames of the captured first video sequence.

### Claim 37 (depends on 36)

The method of claim 36, wherein each of the plurality of detection methods comprise searching for an object positioned at a different orientation.

### Claim 38

A system for de-identifying a video sequence of a user performing a desired activity, the system comprising:
a video capture device operable to capture a video sequence comprising a plurality of individual frames, the plurality of individual frames including images of one or more users performing a first desired activity;
a processor associated with the video capture device operable to perform activity recognition on the captured video sequence to recognize the desired activity by:
determining, through object recognition, the identity of a first object for a first plurality of frames of the video sequence,
determining, through object recognition, the identity of a second object and its surroundings in a second plurality of frames of the video sequence,
designating one or more frames of the captured video sequence as comprising the recognized action of the desired activity, wherein at least one frame is non-designated, identifying a first portion of the one or more designated frames of the captured video sequence to remain visible, wherein the first portion comprises the recognized placement of the second object,
de-identifying a second portion of the one or more designated frames and de-identifying the at least one non-designated frame so that the first portion remains visible; and
de-identifying a further portion of the first portion of the one or more designated frames of the second video sequence determined to include one or more faces other than faces of the one or more users.

### Claim 39 (depends on 38)

The system of claim 38, wherein the de-identifying comprises blurring the image.

### Claim 40 (depends on 38)

The system of claim 38, wherein the processor is further operable to apply each of a plurality of detection methods for recognizing the placement of the desired activity to a subset of the frames of the captured video sequence.
