System, method and computer program product for performing unstructured information management and automatic text analysis, including a search operator functioning as a Weighted AND (WAND)

A data processing system for processing stored data, comprising:

• data storage for storing a collection of data units;
• and
• coupled to the data storage, a search engine responsive to a query for retrieving at least one data unit from said data storage;
• where
• the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum,
• where said data units comprise documents;
• and
• where said data processing system comprises an inverted file system for storing annotations derived from a tokenization of document data, a list comprising occurrences of respective annotations and, for each listed occurrence of a respective annotation, a set comprised of a plurality of token locations spanned by said respected annotation.

A data processing system for processing stored document data, comprising:

• data storage for storing a collection of document data;
• and
• coupled to the data storage, a search engine responsive to a query for retrieving at least one document from said data storage;
• where
• the query comprises a Boolean predicate that functions as a Weighted AND (WAND), the WAND taking as arguments a list of Boolean variables X1, X2, . . . , Xk, a list of associated positive weights, w1, w2, . . . , wk, and a threshold θ, where:
• (WAND) (X1, w1, . . . Xk, wk, θ)
• is true if:
• ∑
• 1
• ≤
• i
• ≤
• k
• ⁢
• x
• i
• ⁢
• w
• i
• ≥
• θ
• ,
• where xi is the indicator variable for Xi, where
• x
• i
• =
• {
• 1
• ,
• if
• ⁢
• ⁢
• X
• i
• ⁢
• ⁢
• is
• ⁢
• ⁢
• true
• 0
• ,
• otherwise
• .
• said search engine comprising an output for outputting a result of a search of the collection of document data using the query.

A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

• a computer program segment for storing a collection of data units;
• and
• a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;
• where
• the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum;
• and
• where said search operator comprises a weighted AND function, where varying the threshold weight value varies the operation of the weighted AND function from being substantially a logical OR function to being substantially a logical AND function.

A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

• a computer program segment for storing a collection of data units;
• and
• a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;
• where
• the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum;
• where said data units comprise documents;
• and
• further comprising a computer program segment for implementing an inverted file system for storing annotations derived from a tokenization of document data, a list comprising occurrences of respective annotations and, for each listed occurrence of a respective annotation, a set comprised of a plurality of token locations spanned by said respected annotation.

A computer program product embodied on a computer-readable medium and comprising program code for directing operation of a text intelligence system in cooperation with at least one application, comprising:

• a computer program segment for storing a collection of data units;
• and
• a computer program segment implementing a search engine that is responsive to a query for retrieving at least stored one data unit;
• where
• the query comprises a search operator comprised of a plurality of search sub-expressions each having an associated weight value, and where said search engine returns a data unit having a weight value sum that exceeds a threshold weight value sum;
• where the query comprises a Boolean predicate that functions as a Weighted AND (WAND), the WAND taking as arguments a list of Boolean variables X1, X2, . . . , Xk, a list of associated positive weights, w1, w2, . . . , wk, and a threshold θ, where:
• (WAND) (X1, w1, . . . Xk, wk, θ)
• is true if:
• ∑
• 1
• ≤
• i
• ≤
• k
• ⁢
• x
• i
• ⁢
• w
• i
• ≥
• θ
• ,
• where xi is the indicator variable for Xi, where
• x
• i
• =
• {
• 1
• ,
• ⁢
• if
• ⁢
• ⁢
• X
• i
• ⁢
• ⁢
• is
• ⁢
• ⁢
• true
• 0
• ,
• ⁢
• otherwise
• .

A method for processing document data, comprising:

• receiving a query;
• and
• responding to the query for retrieving at least one document from a data storage;
• where
• the query comprises a Boolean predicate that functions as a Weighted AND (WAND), the WAND taking as arguments a list of Boolean variables X1, X2, . . . , Xk, a list of associated positive weights, w1, w2, . . . , wk, and a threshold θ, where:
• (WAND) (X1, w1, . . . Xk, wk, θ)
• is true if:
• ∑
• 1
• ≤
• i
• ≤
• k
• ⁢
• x
• i
• ⁢
• w
• i
• ≥
• θ
• ,
• where xi is the indicator variable for Xi, where
• x
• i
• =
• {
• 1
• ,
• if
• ⁢
• ⁢
• X
• i
• ⁢
• ⁢
• is
• ⁢
• ⁢
• true
• 0
• ,
• otherwise
• .
• further comprising outputting as a result of using the query a retrieved at least one document.