Ranking
For each problem (CSP, Max-CSP and WCSP) and each combination of instance and solver category,
there will be a ranking of solvers.
The categories are described in Section 4.
The main criteria for ranking the solvers are as follows.
- CSP:
- Solvers claiming incorrect results in a given category will be disqualified from this category.
Of the remaining solvers,
the solver solving the most problems will be declared the winner.
Ties will be broken
by considering the minimum total solution time.
- Max-CSP/WCSP:
- Solvers claiming incorrect results in a given category will be disqualified from this category.
The remaining solvers will be ranked according to the following:
- Incomplete solvers
- will be ranked in increasing order of
their average relative error
(i.e. the average normalised difference between the cost
of solutions found by the solver and the cost
of the best known solutions).
- Complete solvers
- will be ranked in decreasing order of the number of
instances for which a solution has been proved to be OPTIMUM.
To break ties, we will rank the solvers in increasing order
of their average relative errors.
Marc van Dongen
2009-03-10
