# US 2023/0410504 A9 · System and method for sentence directed video object codetection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410504 A9 - Document ID: US20230410504A9 - Status: Published application - Kind: A9 - Application number: 17/135,995 - Filed: 2020-12-29 - Published: 2023-12-21 - Earliest claimed: 2016-06-06 - Assignees: Purdue Research Foundation - Applicants: Purdue Research Foundation - Inventors: Jeffrey Mark Siskind; Haonan Yu - Web page: https://patentgrep.com/patent/US20230410504A9 ## CPC - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/29: {Graphical models, e.g. Bayesian networks} - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/73: using feature-based methods - G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis - G06V 10/761: {Proximity, similarity or dissimilarity measures} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and method for determining the locations and types of objects in a plurality of videos. The method comprises pairing each video with one or more sentences describing the activity or activities in which those objects participate in the associated video, wherein no use is made of a pretrained object detector. The object locations are specified as rectangles, the object types are specified as nouns, and sentences describe the relative positions and motions of the objects in the videos referred to by the nouns in the sentences. The relative positions and motions of the objects in the video are described by a conjunction of predicates constructed to represent the activity described by the sentences associated with the videos. ## Claims ### Claim 1 A method for determining the locations and types of objects in a plurality of videos, comprising: using a computer processor, receiving a plurality of videos; pairing each of the videos with one or more sentences; using the processor, describing one or more activities in which those objects participate in a corresponding video; and wherein no use is made of a pretrained object detector. ### Claim 2 (depends on 1) The method of claim 1, wherein locations of the objects are specified by the processor as rectangles in frames of the videos, the object types are specified as nouns, and sentences describe the relative positions and motions of the objects in the videos referred to by the nouns in the sentences. ### Claim 3 (depends on 1) The method of claim 1, wherein the relative positions and motions of the objects in the video are described by a conjunction of predicates constructed to represent the activity described by the sentences associated with the videos. ### Claim 4 (depends on 1) The method of claim 1, wherein the locations and types of the objects in the plurality of videos are determined by: a. using one or more object proposal mechanisms to propose locations for possible objects in one or more frames of the videos; b. using one or more object trackers to track the positions of the proposed object locations forward or backward in time; c. collecting the tracked proposal positions for each proposal into a tube; d. computing features for each tube based on image features for the portion of the images inside the tubes; or e. forming a graphical model, wherein: i. one or more noun occurrences in sentences associated with a video are associated with vertices in the model; ii. the set of potential labels of each vertex is the set of proposal tubes for the associated video; iii. pairs of vertices that are associated with occurrences of the same noun in two sentences associated with different videos are attached by a binary factor computed as a similarity measure between the tubes selected for from the label sets for the two vertices; iv. collections of vertices that are associated with occurrences of different nouns in the same sentence associated with a video are attached by a factor whose arity is the arity of a predicate in the conjuction of the predicates used to represent the activity described by the sentence where the score of said represents the degree to which the collection of tubes selected for those vertices exhibits the properties of that predicate; or v. the graphical model is solved by selecting a single proposal tube for each vertex from the set of potential labels for that vertex that collectively maximizes a combination of the similarity measure for all pairs of vertices connected by a similarity factor and the predicate scores of all collections of vertices connected by a predicate factor. ### Claim 5 (depends on 1) The method of claim 1, wherein the proposal generation mechanism is MCG. ### Claim 6 (depends on 1) The method of claim 1, wherein the proposal generation mechanism is EdgeBoxes. ### Claim 7 (depends on 1) The method of claim 1, wherein the proposals are tracked by CamShift. ### Claim 8 (depends on 1) The method of claim 1, wherein moving proposals are tracked in HSV color space and allowed to change size. ### Claim 9 (depends on 1) The method of claim 1, wherein stationary proposals are tracked in RGB color space and are required to remain of constant size. ### Claim 10 (depends on 1) The method of claim 1, wherein PHOW features are used as image/tube features. ### Claim 11 (depends on 1) The method of claim 1, wherein HOG features are used as image/tube features. ### Claim 12 (depends on 1) The method of claim 1, wherein similarity is measured using a chi-squared distance between image/tube features. ### Claim 13 (depends on 1) The method of claim 1, wherein similarity is measured using Euclidean distance between image/tube features. ### Claim 14 (depends on 1) The method of claim 1, wherein the set of proposal is augmented with proposals rotated by multiples of 90 degrees. ### Claim 15 (depends on 1) The method of claim 1, wherein the similarity measures and predicate scores are combined by summation. ### Claim 16 (depends on 1) The method of claim 1, wherein the similarity measures and predicate scores are combined by taking their product. ### Claim 17 (depends on 1) The method of claim 1, wherein the graphical model is solved using Belief Propagation. ### Claim 18 (depends on 1) The method of claim 1, wherein the set of proposals is augmented by detections produced by a pretrained object detector. ### Claim 19 (depends on 1, 18) The method of claim 18, wherein the method of claim 1 is first applied and then the method of claim 18 is applied in one or more subsequent iterations, each iteration using an object detector trained on the proposals selected in earlier iterations.