These pages contain the databases and evaluation code required to run the experiments described in the paper:
print_paperentry_bytitle: more than 2 or no entry matches.
Becauase I am no longer at Columbia, I have not had as much time as I would have liked to make these evaluation tools as portable and easy to use as possible. If you encounter any problems trying to use my code, please contact me and I will try to help. E-mail: simonb@cs.cmu.edu
Modifying Your Edge Detector
The first thing you need to do use these evaluation tools is to modify your edge detector to comply with the following requirements:
- Inputs:
- (1) A standard grey-scale .pgm file.
- (2) The number of edges to detect.
- Command Line Interface:
- Should be of the form:
sobel image.pgm #edges
- Operation:
- Your detector should sort the detected edges by their confidences (most confident first) and output as many of them as were requested in the command line.
- Output:
- For each detected edge, a line of ASCII text should be written to standard output. Each line consists of 4 real-valued fields, separated by spaces:
- (1) The x-coordinate of the location of the edge.
- (2) The y-coordinate of the location of the edge.
- (3) The angle of orientation of the edge in degrees.
- (4) The confidence measure.
- Coordinate Frame:
- The location and orientation of the edges must be specified in the following coordinate frame:
- (1) The origin of the coordinate frame is the middle of the bottom-left pixel.
- (2) The distance between the centers of neighbouring pixels is one unit.
- (3) The x-axis points to the right.
- (4) The y-axis points upwards.
- (5) Angles are measures anti-clockwise in degrees, and the zero direction is along the x-axis.
Of these requirements, the most important one is that the origin is at the center of the bottom left-most pixel.
- Examples:
- To help you fullfill these requirements, here are a few examples:
- (1) Example C++ code for a sobel detector which fullfills the requirements. Feel free to use the input/output parts of this example. Alternatively, use it as a template.
- (2) An example .pgm image d100.pgm from one of the benchmarks
- (3) The output written to standard output by this sobel detector on this image when called with the command line "sobel d100.pgm 100"
Evaluation Code
The next thing to do is to down-load the evaluation code and compile it. If the Makefile does not work and you can't figure out how to modify it, compiling the programs by hand should be straightforward. They should compile with pretty much any C++ compiler. The only libraries they need is the standard i/o and math libraries. Once you have compiled the code, you will need to add the appropriate directory to your path.
There are 7 executables concerned with the evaluation of the edge detectors, and 2 concerned with averaging over multiple images. The 7 evaluation executables are:
- (1) gmc1_colinear_v1 - the first variant of the first (colinearity) global measure of coherence which uses single edges including their orientation
- (2) gmc1_colinear_v2 - the second variant of the first (colinearity) global measure of coherence which uses two edges and inores the orientations
- (3) gmc2_intersection - the second global measure of coherence, the one based on a common intersection point
- (4) gmc3_coparallel - the third global measure of coherence, the one based on the edges all be coparallel
- (5) gmc4_ellipse5 - the first variant of the fourth (ellipse) measure of coherence which uses 5 points on the ellipse and no tangency constraints
- (6) gmc4_ellipse4 - the second variant of the fourth (ellipse) measure of coherence which uses 4 points on the ellipse and one tangency constraint
- (7) gmc4_ellipse3 - the third variant of the fourth (ellipse) measure of coherence which uses 3 points on the ellipse and two tangency constraints
All of the programs read from standard input and write to standard output. They have one command line argument, namely, the number of edges to work on. They can be combined with an edge detector in the following manner:
sobel d100.pgm 100 | gmc1_colinear_v1 100
The output is a curve of the confidence measure against the global measure of coherence. An example of the output obtained using the above command line, and the sobel detector and image give above is contained in this file. The curve in the second column of this file is an example of one of those shown in Figure 3 of the paper.
The other 2 exectables are concerned with averaging over several images:
- (1) gmc_mean - used to average over several different images to create graphs like that in Figure 4 in the paper
- (2) gmc_diff - used to compare two detectors on the same image to produce graphs like that in Figure 5 in the paper
Hopefully, you should not need to touch these files.
Finally, the evaluation code contains a number of scripts which are used to put all the pieces together:
- (1) gmc_benchmark - this script takes 3 arguments: (1) the edge detector, (2) the evaluation program from one of the 7 above, and (3) the number of edges to apply it to. Called from within one of the bechmark database directories below it generates a curve like those in Figure 4 in the paper. Called with the command line
gmc_benchmark sobel gmc1_colinear_v1 1000
in the first bechmark database below it generates the curve in the results directory. Column 2 of this file is the mean measure of coherence, and column 3 is the variance.
- (2) gmc_comparison - this script takes 4 arguments: (1) the first edge detector, (2) the second edge detector, (3) the evaluation program from one of the 7 above, and (3) the number of edges to apply it to. Called from within one of the bechmark database directories below it generates a curve like those in Figure 5 in the paper. Called with the command line
gmc_comparison sobel roberts gmc1_colinear_v1 1000
in the first bechmark database below it generates the curve in the results directory. Column 2 of this file is the mean relative measure of coherence, and column 3 is the variance.
Benchmark Databases
There are 7 benchmark databases. Once downloaded and uncompressed, change directory into the benchmark directory so that the image, results, and tmp directories are visible. Then, either of the scripts gmc_benchmark or gmc_comparison can be called. Any edge detector can be used with any benchmark. However, only certain evaluation programs can be used with each benchmark. Similarly, each benchmark should be called with a different number of edges. For each benchmark, I described which evaluation programs can be used and give a guideline number of edges.
- Benchmark 1 - Use with either gmc1_colinear_v1 or gmc_colinear_v2 and around 1000 edges. Example use: "gmc_benchmark sobel gmc1_colinear_v1 1000"
- Benchmark 2 - Use with either gmc1_colinear_v1 or gmc_colinear_v2 and around 1000 edges. Example use: "gmc_benchmark sobel gmc1_colinear_v1 1000"
- Benchmark 3 - Use with either gmc1_colinear_v1 or gmc_colinear_v2 and around 1000 edges. Example use: "gmc_benchmark sobel gmc1_colinear_v1 1000"
- Benchmark 4 - Use with gmc2_intersection and around 1500 edges. Example use: "gmc_benchmark sobel gmc2_intersection 1500"
- Benchmark 5 - Use with gmc2_intersection and around 1500 edges. Example use: "gmc_benchmark sobel gmc2_intersection 1500"
- Benchmark 6 - Use with gmc3_coparallel and around 3000 edges. Example use: "gmc_benchmark sobel gmc3_coparallel 3000"
- Benchmark 7 - Use with gmc4_ellipse3, gmc4_ellipse4, or gmc_ellipse5 and around 2000 edges. Example use: "gmc_benchmark sobel gmc4_ellipse3 2000"
The images which comprise each benchmark are in the images directory, tmp is used as a temporary workspace, and the results are placed in results.
Edge Detector Code
The code for the edge detectors that I tested is contained here. Getting this to compile may be substantially more difficult than the evaluation code above. But, you are welcome to try. Once you have downloaded the code and uncompressed it, the first thing you need to do is edit the "BASEDIR" in the Makeconf file. Then try "make install." If all goes well, first add the appropriate subdirectory of bin to your path.
Then, there is just one more thing you need to do before you can run the executables in bin/ops_sys. You need to generate the data for the Baker-Nayar-Murase detector. Change directory to the "features" directory and run: "generate -o se.pf -f Step_Edge -w Square_5x5 -s Rectangular_Averaging -n 50000 -d 8". Assuming everything has compiled OK, and you have set your path correctly, this will generate a step edge detector. The other executables have been hard-coded to find the file $BASEDIR/features/se.pf so do not move the directory or touch the file se.pf.
There are 5 detectors: pfd (Baker-Nayar-Murase), nalwa (Nalwa-Binford), canny (a simple canny-like detector), sobel, and roberts_cross. All of them need to be appended with one of the following three appendixes: _nms (non-maximal suppression), _nms_spl (non-maximum suppression and sub-pixel localization), or _raw (neither nms or spl).
To Be Added
There are two things from the paper that I have yet to add to this page:
- (1) The results I obtained for the detector I implemented.
- (2) The scripts and the data to produce graphs like that in Figure 7 where a single detector is compared across equivalent datasets captured under different conditions.
I hope to find time to add these components as soon as possible.