Classifies a sequence of nodes based on training data. The training data is in the form of a classifier specification, which is generated from the output of cts:train. Returns labels for each of the input documents in the same order as the input document.

cts:classify classifies a sequence of nodes using the output from cts:train. The $data-nodes and $classifier parameters are respectively the nodes to be classified and the specification output from cts:train. cts:classify can use either supports or weights forms of the $classifier output from cts:train (see Output Formats). If the supports form is used, the training nodes must be passed as the 4th parameter. The $options parameter is an options element in the cts:classify namespace.

The output is a sequence of label elements of the form:

<cts:label> 
  <cts:class name="Example 1" val="-0.003"/>
  <cts:class name="Example 2" val="1.4556"/>
  ...
</cts:label>

Each label corresponds to the data node in the corresponding position in the input sequence. There will be a <class> child for each class where the document passed the class threshold. The val attribute gives the class membership value for the data node in the given class. Values greater than zero indicate likely class membership, values less than zero indicate likely non-membership. Adjusting thresholds can give more or less selective classification. Increasing the threshold leads to a more selective classification (that is, decreases the likelihood of classification in the class). Decreasing the threshold gives less selective classification.

let $firsthalf := xdmp:directory("/shakespeare/plays/", "1")[1 to 19]
let $secondhalf := xdmp:directory("/shakespeare/plays/", "1")[20 to 37]
let $classifier :=  
  let $labels := for $x in $firsthalf 
         return
         <cts:label>
           <cts:class name={xdmp:document-properties(xdmp:node-uri($x))
                 //playtype/text()}/>
         </cts:label>
  return
  cts:train($firsthalf, $labels, 
          <options xmlns="cts:train">
            <classifier-type>supports</classifier-type>
          </options>)
return
cts:classify($secondhalf, $classifier, 
             <options xmlns="cts:classify"/>,
             $firsthalf)

  => ( <label>...</label>,... )

Compute precision, recall, the F measure, and thresholds for the classes computed by the classifier, by comparing with the labels for the same set.

You use the output of cts:thresholds to determine the best thresholds values for your data, based on the first pass through the first part of your training data. The output of cts:thresholds provides you with precision and recall measurements at the calculated thresholds for each class. The following are the definitions of the attributes of the thresholds element returned by cts:thresholds:

name

The name of the class.

threshold

The threshold that is computed by the classifier to give the best results. The threshold is used by cts:classify when classifying documents, and is defined to be the positive or negative distance from the hyperplane which represents the edge of the class.

precision

A number which represents the fraction of nodes identified in a class that are actually in that class. As this aproaches 1, there is a higher probability that you over-classified.

recall

A number which represents the the fraction of nodes in a class that were identified by the classifier as being in that class. As this aproaches 1, there is a higher probability that you under-classified.

F (the F-measure)

A measure which represents if the classification at the given threshold is closer to recall or closer to precision. A value of 1 indicates that precision and recall have equal weight. A value of 0.5 indicates that precision is weighted 2x recall. A value of 2 indicates that recall is weighted 2x prcision. A value of 0 indicates that the weighting is precision only, and a value of +INF (xs:double('+INF')) indicates that weighting is recall only.

let $firsthalf := xdmp:directory("/shakespeare/plays/", "1")[1 to 19]
let $labels := for $x in $firsthalf 
       return
       <cts:label>
         <cts:class name={xdmp:document-properties(xdmp:node-uri($x))
                 //playtype/text()}/>
       </cts:label>
let $classifylabels :=
  let $secondhalf := 
          xdmp:directory("/shakespeare/plays/", "1")[20 to 37]
  let $classifier :=  
    cts:train($firsthalf, $labels, 
          <options xmlns="cts:train">
            <classifier-type>supports</classifier-type>
          </options>)
  return
  cts:classify($firsthalf, $classifier, 
             <options xmlns="cts:classify"/>,
             $firsthalf)
return
cts:thresholds($classifylabels, $labels)
(: 
   This returns the computed thresholds for the plays in a 
   Shakespeare database.  For example:

<thresholds xmlns="http://marklogic.com/cts">
  <class name="TRAGEDY" threshold="-0.0192594" precision="1"
         recall="1" f="1" count="8"/>
  <class name="COMEDY" threshold="0.934239" precision="1" 
         recall="0.8" f="0.888889" count="5"/>
  <class name="HISTORY" threshold="0.101927" precision="1" 
         recall="1" f="1" count="6"/>
</thresholds>
:)

Produces a set of classifiers from a list of labeled training documents.

The elements in the label sequence should match one for one with the nodes in the training node sequence. The first label element describes the first node in the training node sequence, the second label element describes the second node in the training node sequence, and so on. If there are more labels than training nodes or more training nodes than labels, an error is raised.

The format of each label element is:

  <cts:label name="Node1">
    <cts:class name="Example1"/>
    <cts:class name="Example2" val="-1"/>
        :   :
  </cts:label>

Each class listed indicates whether the corresponding node in the training sequence is in the given class. Examples are taken to be positive examples unless specified otherwise (with a val attribute of -1). The document is assumed to be a negative example of any classes that are not explicitly listed. The name attribute on the label element is an optional name for the labelled node. It is purely for human consumption to help in tuning the classification parameters.

Output Formats

A linear classifier is defined by a weight vector w on terms, and an offset value b. The <weights/> node encodes the weight vector directly. Its children are the classes, and each class includes a list of terms. The term node uses an internal id to identify the term and a term weight:

<weights>
  <class name="Example1" offset="2.04">
    <term id="43587329645324245" val="0.3423432"/>
    <term id="47893427895432534" val="-0.12345556"/>
      :                           :
  </class>
      :
</weights>

The weight vector w is a linear combination of the documents themselves, and it may be more convenient to express the classifier in this way. For instance, if the number of terms is not limited, the <weights/> node will be extremely large. The weight vector form may not be used if the classifier kernel is non-linear, that is, with the Gaussian or geodesic kernel.

The support vector representation of the classifier includes a supports node that has <class/> children for each class. Here the class elements contain a list of doc elements which identify the specific training nodes using an internal key. This internal key is valid across queries only for nodes in the database. Each doc element has an attribute encoding the weight of that document and an error attribute which shows how well the document fit the classifier. Large positive or negative errors (greater than about 1.5) are potentially mis-classified documents.

<supports>
  <class name="Example1" offset="2.04">
    <doc id="155584958759" name="Node102" val="-0.00334163" err="1.4"/>
    <doc id="594064848864" name="Node57" val="0.025341234" err="-2.3"/>
      :                             :
  </class>
      :    
</supports>

Each class is identified by a unique name.

let $firsthalf := xdmp:directory("/shakespeare/plays/", "1")[1 to 19]
let $labels := for $x in $firsthalf 
         return
         <cts:label>
           <cts:class name={xdmp:document-properties(xdmp:node-uri($x))
                //playtype/text()}/>
          </cts:label>
return
cts:train($firsthalf, $labels, 
       <options xmlns="cts:train">
         <classifier-type>supports</classifier-type>
       </options>)

  =>  <cts:classifier>...