How to submit
The submissions will be sent by the end of september directly to ludovic[dot]denoyer[at]lip6[dot]fr. Each submission will be composed by one TXT file formatted as presented below:
Submission example for classification
# Authors # description of the model # ... # end of description id_document category:score category:score category:score ....
The sumbission file is composed of one line for each document. Each line contains the score of all the categories for the corresponding document.
We will use two evaluation measures:
- The mean average precision computed for each document
- The classification error computed by considering that a document belongs to a category if its score is > 0.5
We propose to evaluate two aspects of the submissions • The capacity of the models to find the good categories for each document: for that we will compute the number of classification errors. Typically, we will compute one recall for each possible category and then compute the (micro and macroaverage) mean of the recall over all the classes. • The capacity of the models to assign high score to real categories. For that, we will consider the average precision by considering that, for each document, the system returns a list of ranked categories. Average precision emphasizes returning more relevant categories earlier. The average precision will be computed for each document and then averaged over all the documents.
Note that we are opened to any evaluation suggestions.
Please ask any question to Ludovic DENOYER