Aug
14

Performance Testing your Classifier

Once you have a *.xml file that you created from training the classifier then you can test the performance of that classifier cascade against your prositive test or you could create a validation test set to test performance against. OpenCV comes with a built in performance utility

C:\OpenCV2.2\bin>opencv_performance.exe -data "C:\My Documents\HaarClassifier\haarcascade\haarcascade.xml" -info "C:\My Documents\Positive Test Set\positivesoutput.txt"

Explanation of the code above:

opencv_performance.exe is a utility distributed with the OpenCV library. This is usually contained int the “bin” folder in the directory you installed OpenCV.
ex. C:\OpenCV2.2\bin

Below are the options available for this utility as well as what the defaults are.

C:\OpenCV2.2\bin>opencv_performance.exe
Usage: opencv_performance.exe
  -data <classifier_directory_name>
  -info <collection_file_name>
  [-maxSizeDiff <max_size_difference = 1.500000>]
  [-maxPosDiff <max_position_difference = 0.300000>]
  [-sf <scale_factor = 1.200000>]
  [-ni]
  [-nos <number_of_stages = -1>]
  [-rs <roc_size = 40>]
  [-w <sample_width = 24>]
  [-h <sample_height = 24>]

-ni – If specified it will not output the resultant image files with detections. If NOT specified it WILL create a new image for every image it opens and it will save it as det-[image filename] and it will contain boxes around the positive detections.

A small excerpt from the performance output:

+================================+======+======+======+
|            File Name           | Hits |Missed| False|
+================================+======+======+======+
|   20100627_001220_1024_0193.jpg|     0|     1|    19|
+--------------------------------+------+------+------+
|   20100627_002044_1024_0193.jpg|     1|     0|    16|
+--------------------------------+------+------+------+
|   20100627_004432_1024_0193.jpg|     1|     0|    17|
+--------------------------------+------+------+------+
|   20100627_005956_1024_0193.jpg|     1|     0|    20|
+--------------------------------+------+------+------+
|   20100627_010132_1024_0193.jpg|     1|     0|    16|
+--------------------------------+------+------+------+
|   20100627_011308_1024_0193.jpg|     0|     1|    22|
+--------------------------------+------+------+------+
|   20100627_014044_1024_0193.jpg|     1|     0|    21|
+--------------------------------+------+------+------+
|   20100627_015444_1024_0193.jpg|     1|     0|    18|
+--------------------------------+------+------+------+
|   20100627_015956_1024_0193.jpg|     1|     0|    17|
+--------------------------------+------+------+------+
|   20100627_021220_1024_0193.jpg|     1|     0|    18|
+--------------------------------+------+------+------+
|   20100627_021932_1024_0193.jpg|     1|     0|    18|
+--------------------------------+------+------+------+
...
...
+--------------------------------+------+------+------+
|                           Total|  1009|   352| 10532|
+================================+======+======+======+
Number of stages: 18
Number of weak classifiers: 98
Total time: 9.734000
18
        41      93      0.431579        0.978947
        41      93      0.431579        0.978947
        41      93      0.431579        0.978947
        39      38      0.410526        0.400000
        38      28      0.400000        0.294737
        38      22      0.400000        0.231579
        34      16      0.357895        0.168421
        ...
        ...

For metrics and data analysis this can be very helpful as it will not only give you the performance but also the number of stages the classifier used and the number of weak classifiers used in this cascade.
This is an easy way of seeing your classifiers performance right away, and based on the results you can tell whether your classifier is really performing poorly (No Hits and/or Alot of False Hits). If this is the case it would be best to start the process over and maybe get more positive and negative training samples.

Example of image file created when I did not use the -ni option. (It will mark all the objects it calculates as”positive” areas, so hits and false hits will show up).
det-20101231_200008_1024_0193

Tags: , , , ,

Posted by Athena Command Prompt, OpenCV, Thesis Research Subscribe to RSS feed

7 Comments on Performance Testing your Classifier

  1. Szilvi says:
    February 7, 2012 at 10:54 am

    Hi,
    can you please explain me the output of the performance.exe? What does the second part mean?
    41 93 0.431579 0.978947
    41 93 0.431579 0.978947
    41 93 0.431579 0.978947

    Reply
    • Athena says:
      February 20, 2012 at 1:22 pm

      Those are the values that can be used for a ROC plot graph. Its a way of showing the sensitivity and specificity of your classifier.
      For example:

      [Hits] [False Alarms] [True Positive Rate (TPR)] [False Positive Rate (FPR)]
      41 93 0.431579 0.978947

      ROC Plots

      Hope this helps!

      Reply
  2. shenhangyu says:
    September 5, 2012 at 2:32 am

    My results show as follows:

    +================================+======+======+======+
    | File Name | Hits |Missed| False|
    +================================+======+======+======+
    |3_28-3-326-348_20021120143759.bm| 0| 1| 0|
    +——————————–+——+——+——+
    | face02425.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | face02426.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | face02427.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | face02428.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | face02429.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | GULF_159.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | GULF_160.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | GULF_161.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | GULF_162.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | GULF_163.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | GULF_164.bmp| 0| 1| 0|
    +——————————–+——+——+——+
    | Total| 0| 12| 0|
    +================================+======+======+======+
    Number of stages: 7
    Number of weak classifiers: 82
    Total time: 0.017000
    7
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000
    0 0 0.000000 0.000000

    Can you tell me why this case happens?

    Reply
  3. shenhangyu says:
    September 5, 2012 at 2:37 am

    Can you tell me the format of the file “positivesoutput.txt”?
    Whether the format is “face02407.bmp 1 0 0 20 20”?

    Reply
  4. Tony says:
    June 10, 2013 at 12:33 pm

    hi!
    using the performance.exe with the parameter “-ni” works out just fine, but when I also want to save the images, I get an error saying “could not save the image”. I didn’t compile the performance.exe by myself, so I cannot debug it -could you upload a working performance.exe for me somewhere and share the link? that would be just great !!
    Thank you,
    Tony

    Reply
  5. Akash says:
    October 16, 2014 at 10:01 pm

    Can any one tell me the positiveoutput.txt file format????

    thanks

    Reply

Leave a Reply