Output details

Repeatable and reliable search system evaluation using crowdsourcing

<16> Originality: First-ever demonstration of the repeatability and reliability of crowdsourced evaluation of semantic-web search-engines using multiple trials and comparing with expert judgements.

Significance: Describes an ongoing evaluation campaign of semweb search system performance with submissions from academic and commercial participants. Repeatability enables comparison of performance across time for deployed search engines. Supports work towards fully-automated develop-test-evaluate loop by showing high agreement between distinct judge pools separated in time. Subsequent competitions have reused this methodology (http://www.websemanticsjournal.org/index.php/ps/article/view/336).

Rigour: Demonstrates reliability using multiple evaluation and relevance metrics. Introduces objective criteria for rejecting unreliable crowdsourced judges. Acceptance rate for this SIGIR was 19.9%.