Integrated sensor and video motion analysis method

An integrated sensor and video motion analysis method comprising:

• obtaining a video captured from a camera;
• obtaining one or more objects of interest and one or more distinguishing visual characteristics for each of said one or more objects of interest using a computer, wherein
• at least one of said one or more objects of interest includes a visual marker;
• said visual marker has a pattern with no rotational symmetries;
• said one or more distinguishing visual characteristics of said at least one of said one or more objects of interest comprise said pattern of said visual marker;
• selecting a plurality of frames from said video for analysis;
• preprocessing said plurality of frames to facilitate image analysis of said plurality of frames, wherein said preprocessing said plurality of frames comprises one or more of noise removal, flicker removal, smoothing, color space conversion, color or brightness balancing, and shadow removal;
• transforming each of said plurality of frames except a first frame of said plurality of frames to an aligned frame that estimates a view of said plurality of frames except said first frame from said camera pose for a previous frame of said plurality of frames by
• calculating a frame translation that minimizes pixel differences between a center region of a frame of said plurality of frames and a center region of the previous frame after applying said frame translation;
• dividing each of said plurality of frames into tiles;
• and,
• calculating a local tile translation for each of said tiles that minimizes the pixel differences between a tile and a corresponding tile of the previous frame after applying said frame translation and said local tile translation;
• calculating frame differences between each of said aligned frames and the previous frame;
• calculating high motion regions from said frame differences;
• searching said high motion regions of said plurality of frames for said one or more objects of interest, yielding
• one or more identified objects;
• and,
• one or more regions within said plurality of frames, wherein
• each of said one or more regions is associated with one of said one or more identified objects;
• and,
• pixels of each of said one or more regions match said one or more distinguishing visual characteristics for said one of said one or more identified objects associated with said regions;
• estimating a camera pose of said camera for each of said plurality of frames;
• estimating an object pose relative to said camera pose for one or more of said identified objects in one or more of said plurality of frames from a location and pixel contents of said one or more regions, by
• estimating a ray on which said one or more identified objects lies from the location of said one or more regions in said plurality of frames for said one or more identified objects;
• estimating a relative distance of said one or more identified objects from said camera from a relative size of said one or more regions in said plurality of frames for said one or more identified objects;
• estimating an orientation of said one or more identified objects relative to said camera by calculating a rotation that minimizes a difference between a reference pixel pattern for said one or more identified objects and said pixel contents of said one or more regions for said one or more identified objects;
• and,
• wherein said reference pixel pattern corresponds to one of said one or more distinguishing visual characteristics for said one or more of identified objects, or said reference pixel pattern is obtained from said pixel contents of said one or more regions for a designated reference frame;
• transforming said object pose in each of said plurality of frames to a common coordinate system, using said camera pose for each of said plurality of frames;
• obtaining motion capture data for one or more of said one or more objects of interest;
• obtaining one or more event signatures, each comprising
• an event type;
• a sensor data signature associated with said event type;
• a video signature associated with said event type;
• locating a prospective event matching one of said sensor data signature of said one or more event signatures in said motion capture data or matching one of said video signature of said one or more event signatures in said plurality of frames;
• assigning a prospective event type to said prospective event;
• eliminating said prospective event as a false positive if the prospective event does not match both of said sensor data signature for said prospective event type and said video signature for said prospective event type;
• if said prospective event is not eliminated as a false positive, synchronizing said motion capture data with said plurality of frames by aligning a motion capture data timestamp corresponding to said sensor data signature for said prospective event with a frame of said plurality of frames at which said video signature for said prospective event occurs in said video;
• generating one or more motion metrics for said one or more identified objects using sensor fusion to combine changes in said object pose across said plurality of frames with said motion capture data;
• obtaining one or more signatures of activities of interest, each of said one or more signatures corresponding to one or more of said one or more motion metrics;
• searching said plurality of frames for said one or more signatures of activities of interest;
• selecting one or more highlight frames or fail frames that include frames at which said one or more signatures of activities of interest occur, and optionally include frames that are proximal to the frames at which said one or more signatures of activities of interest occur;
• overlaying said one or more highlight frames or fail frames with data or graphics representing values of one or more of said one or more motion metrics at each of said one or more highlight frames or fail frames;
• transmitting said one or more highlight frames or fail frames to one or more of a repository, a viewer, a server, a computer, a social media site, a mobile device, a network, and an emergency service;
• and,
• wherein
• said motion capture data and said one or more motion metrics comprise one or more of linear position, linear velocity, linear acceleration, trajectory, orientation, angular velocity, angular acceleration, time of motion, time elapsed between positions, time elapsed between start of motion and arriving at a position, and time of impact;
• said one or more distinguishing visual characteristics comprise one or more of shape, curvature, size, color, luminance, hue, saturation, texture, and pixel pattern;
• and,
• said one or more objects of interest comprise one or more of equipment and persons.