Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: INNER | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## norsys.netica
Class NetTester

    java.lang.Object
      |
      +--norsys.netica.NetTester

public class NetTester
extends java.lang.Object

A tool for grading a Bayes net by using a set of real cases to see how well the predictions or diagnosis of the net match the actual cases. It is not for decision networks.

The following outlines how you typically use a NetTester object.

1. Choose the nodes that you wish the Net to predict and get rated on, and then place them in a NodeList. These nodes are called the "test" nodes. Their values in the case file are treated as though they were hidden from the Bayes net.

2. Choose any additional nodes you do not wish the network to know the value of during its inference. For example, if the network is for medical diagnosis, you might select the disease nodes and nodes representing other unobservable internal states. These nodes are called the "unobserved" nodes. It is okay if this list is empty or null when passed to the NetTester constructor.

3. Construct the NetTester. E.g.: NetTester tester = new NetTester (testNodes, unObsNodes, -1);

4. Run the method testWithFile(), supplying the case file to use for testing. Netica will read the case file, processing the cases one-by-one. When Netica reads a case, it will ignore any findings for the unobserved nodes. It then does belief updating to generate beliefs for each of the unobserved nodes. It goes back and checks the true value for those nodes as supplied by the case file (if they are supplied for that case), and compares them with the beliefs it generated. It accumulates all the comparisons into summary statistics.

5. Retrieve any of the summary statistics for the nodes you are interested in. Call any of getConfusion, getErrorRate, getLogLoss, or getQuadraticLoss, as desired.

6. Repeat steps 4 and 5 as often as desired, on possibly new case files, thus accumulating the cases and observing how the statistics change.

7. Optionally, cleanup up all the resources allocated for testing by calling the `finalize` method.

A coding example of the above is given in the example for NetTester.

Since:
  2.08
Version:
  2.26 - October 1, 2002

Constructor Summary
`NetTester(NodeList testNodes, NodeList unObsNodes, int tests)` Creates a NetTester object which is a tool for grading a Bayes net (it is not for decision networks) using a set of real cases to see how well the predictions or diagnosis of the net match the actual cases.

Method Summary
`void` | `finalize()` Informs the native NetTester that it is no longer required, so that it may free-up its resources.
`double` | `getConfusion(Node node, int predictedState, int actualState)` Returns the number of times the Net predicted predictedState for node, but the case file actually held actualState as the value of that node.
`double` | `getErrorRate(Node node)` Returns the cumulative "error rate" of node under the tests previously performed.
`double` | `getLogLoss(Node node)` Returns the "logarithmic loss" of node under the tests previously performed.
`double` | `getQuadraticLoss(Node node)` Returns the "quadratic loss" of node under the tests previously performed.
`void` | `testWithFile(Streamer inStream)` Perform the tests on the Bayes net using the case data in inStream.

Methods inherited from class java.lang.Object
`clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### NetTester

    public NetTester(NodeList testNodes,
                     NodeList unObsNodes,
                     int tests)
              throws NeticaException

  Creates a NetTester object which is a tool for grading a Bayes net (it is not for decision networks) using a set of real cases to see how well the predictions or diagnosis of the net match the actual cases.

  testNodes are nodes that the Bayes net will predict and get rated on. Their values in the case file are all hidden from the Bayes net (ie, unobserved) whenever a case is read. For each such case, the Bayes net does a prediction and compares that prediction with the true value from the case file, cumulating statistics as it goes.

  If unObsNodes is non-null, then the nodes it contains will also be unobserved.

  Pass -1 for tests.
  Parameters:

  NodeList |  | testNodes |  | the nodes that the Bayes net will predict and get rated on
  NodeList |  | unObsNodes |  | a list of nodes that will will also be unobserved. If there are no such nodes, you may pass null.
  int |  | tests |  | a parameter for future use; pass -1 for now.

  Version:
  Versions 2.08 and later have this method.
  See Also:

  testWithFile |  | Accumulate case data into the test
  getConfusion |  | Get elements of the confusion matrix
  getErrorRate |  | Get fraction of test cases where prediction failed
  getLogLoss |  | Get the "logarithmic loss" score of the test
  getQuadraticLoss |  | Get the "quadratic loss" score of the test
  finalize |  | Free up tester and all its resources

  Example:

        Net net = new Net (new Streamer ("BookBags.dne"));
        net.compile();
        NodeList testNodes = new NodeList (net);
        Node testNode = net.getNode ("Book_Bag");
        testNodes.add (testNode);
        NetTester tester = new NetTester (testNodes, null, -1);
        tester.testWithFile (new Streamer ("BookBags100.cas"));
        printConfusionMatrix (tester, testNode); // defined in example for  getConfusion
        System.out.println ("Error rate for " + node.getName() + " = " + tester.getErrorRate(node));
        tester.finalize();

Method Detail

### finalize

    public void finalize()
                  throws NeticaException

  Informs the native NetTester that it is no longer required, so that it may free-up its resources.

  If you override this method, be sure to call the base class method (super.finalize();).
  Version:

  Versions 2.08 and later have this method.
  In the C Version of the API, this function is named DeleteNetTester_bn.
  See Also:

  NetTester |  | Construct the NetTester object

  Overrides:
    `finalize` in class `java.lang.Object`

### testWithFile

    public void testWithFile(Streamer inStream)
                      throws NeticaException

  Perform the tests on the Bayes net using the case data in inStream.

  The net must be compiled (see compile) before calling this.

  This method can be called multiple times with different files to accumulate the results of all the cases.

  Calls to this method can be intermingled with calls to getConfusion, getErrorRate, getLogLoss, and getQuadraticLoss.
  Parameters:

  Streamer |  | inStream |  |

  Version:
  Versions 2.08 and later have this method.
  In the C Version of the API, this function is named TestWithFile_bn.
  See Also:

  NetTester |  | Construct the NetTester object

### getConfusion

    public double getConfusion(Node node,
                               int predictedState,
                               int actualState)
                        throws NeticaException

  Returns the number of times the Net predicted predictedState for node, but the case file actually held actualState as the value of that node. These are the entries of a table traditionally called the "confusion matrix".

  The prediction is formed by reading all of the observed node values from the case file, updating beliefs, and then picking the state of this node which has the highest resultant (posterior) probability.

  node must have been in the testNodes list originally passed to NetTester.
  Parameters:

  Node |  | node |  | The node being examined.
  int |  | predictedState |  | The state predicted by this net, as compared to the actualState in the case file.
  int |  | actualState |  | the actual state in the case file, as compared to the predictedState

  Version:
  Versions 2.08 and later have this method.
  In the C Version of the API, this function is named GetTestConfusion_bn.
  See Also:

  NetTester |  | Construct the NetTester object
  getErrorRate |  | Get fraction of test cases where prediction failed
  getLogLoss |  | Get the "logarithmic loss" score of the test
  getQuadraticLoss |  | Get the "quadratic loss" score of the test

  Example:

        /*
         * Print a confusion matrix table.
         */
        public static void printConfusionMatrix (NetTester nt, Node node) throws NeticaException {
            int numStates = node.getNumStates();
            System.out.println("Confusion matrix for " + node.getName() + ":");

            for (int i=0;  i < numStates;  ++i){
                System.out.print( "\t" + node.state(i).getName() );
            }
            System.out.println( "\t" + "Actual" );

            for (int a=0;  a < numStates;  ++a){
                for (int p=0;  p < numStates;  ++p){
                    System.out.print( "\t" + (int) (nt.getConfusion(node, p, a )) );
                }
                System.out.println( "\t" + node.state(a).getName() );
            }
        }

        Sample output from running the above method might be:

            Confusion matrix for Book_Bag:
                    bag_1   bag_2   Actual
                    36      7       bag_1
                    7       50      bag_2

### getErrorRate

    public double getErrorRate(Node node)
                        throws NeticaException

  Returns the cumulative "error rate" of node under the tests previously performed. This is the fraction of times the Net predicted or diagnosed states incorrectly for node, out of all the cases which provided a value for node.

  The prediction is formed by reading all of the observed node values from the case file, updating beliefs, and then picking the state of this node which has the highest resultant (posterior) probability.

  A result of 0.0 means no prediction errors, whereas a result of 1.0 means all predictions were in error.

  node must have been in the testNodes list originally passed to NetTester.
  Parameters:

  Node |  | node |  | The node being examined.

  Version:
  Versions 2.08 and later have this method.
  In the C Version of the API, this function is named GetTestErrorRate_bn.
  See Also:

  NetTester |  | Construct the NetTester object
  getConfusion |  | Get elements of the confusion matrix
  getLogLoss |  | Get the "logarithmic loss" score of the test
  getQuadraticLoss |  | Get the "quadratic loss" score of the test

### getLogLoss

    public double getLogLoss(Node node)
                      throws NeticaException

  Returns the "logarithmic loss" of node under the tests previously performed.

  The "logarithmic loss" is defined as: MOAC [- log (Pc)],

  where MOAC stands for the mean (average) over all cases (i.e., all cases
  for which the case file provides a value for the node in question), and

  where log(Pc) is the natural logarithm of the probability predicted for the state that turns out to be correct.

  node must have been in the testNodes list originally passed to NetTester.
  Parameters:

  Node |  | node |  | The node being examined.

  Version:
  Versions 2.08 and later have this method.
  In the C Version of the API, this function is named GetTestLogLoss_bn.
  See Also:

  NetTester |  | Construct the NetTester object
  getConfusion |  | Get elements of the confusion matrix
  getErrorRate |  | Get fraction of test cases where prediction failed
  getQuadraticLoss |  | Get the "quadratic loss" score of the test
  `Sensitivity.getMutualInfo` |  | Use a Sensitivity measurer to find the mutual info (entropy reduction) between two nodes

### getQuadraticLoss

    public double getQuadraticLoss(Node node)
                            throws NeticaException

  Returns the "quadratic loss" of node under the tests previously performed.

  The "quadratic loss" is defined as: MOAC [1 - 2 * Pc + sum[j=1 to n] (Pj ^ 2)],

  where MOAC stands for the mean (average) over all cases (i.e., all cases
  for which the case file provides a value for the node in question),

  where Pc is the probability predicted for the state that turns out to be correct,

  where Pj is the probability predicted for state j, and

  where n is the number of states of the node.

  node must have been in the testNodes list originally passed to NetTester.
  Parameters:

  Node |  | node |  | The node being examined.

  Version:
  Versions 2.08 and later have this method.
  In the C Version of the API, this function is named GetTestQuadraticLoss_bn.
  See Also:

  NetTester |  | Construct the NetTester object
  getConfusion |  | Get elements of the confusion matrix
  getErrorRate |  | Get fraction of test cases where prediction failed
  getLogLoss |  | Get the "logarithmic loss" score of the test

Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: INNER | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD