Performance Testing Accuracy

When creating a classifier to use for detection of solar cavities, the steps taken are essentially creating the samples to train with, train, and then test the performance. In testing the performance, I began looking through the output, and noticed some oddity. For some of the images the performance output would classify a MISS, so I decided to do some comparison of those results by looking at the images and do a visual inspection of the performance classification results and the actual classification results. I was getting misses based on the size difference of my marked image (what I denoted as the bounds or area of the positive object) versus the classifiers size and position of what it calculates as the positive object.

Below is an excerpt from the performance output originally and for this specific image the output shows that there were no HITS and 1 MISS.

+================================+======+======+======+
|            File Name           | Hits |Missed| False|
+================================+======+======+======+
[...]
+--------------------------------+------+------+------+
|   20101231_231508_1024_0193.jpg|     0|     1|     9|
+--------------------------------+------+------+------+

But this below excerpt is when the performance tool is run specifying to not be so strict on the size and position difference when comparing that positive classification with the marked image. Now we get the HIT with no MISSES.

+================================+======+======+======+
|            File Name           | Hits |Missed| False|
+================================+======+======+======+
[...]
+--------------------------------+------+------+------+
|   20101231_231508_1024_0193.jpg|     1|     0|     7|
+--------------------------------+------+------+------+

Here is an example of the output image, I changed some colors to highlight which one is which. The GREEN rectangle shows essentially the marked image, while the BLUE rectangles are the classifiers detections. You can notice that the BLUE and GREEN area are not exact, but the area that the BLUE rectangle covers does still in fact include the cavity so it should have really been a positive HIT from the beginning.
So what does all this mean?
Pretty simply put, the performance values initially obtained did not truly reflect the performance of the classifier.

Tags: opencv, performance, testing outcomes, thesis