Call for Replication, Benchmark, Data and Software

Science, to a large degree is like building a house of ideas on the foundations laid by others. In some scientific disciplines the laying of foundations is explicitly seen as part of the innovative activity of the discipline. In the Semantic Web and Linked Data field there has been a bias towards presenting novel ideas in research papers. The goal of the Replication, Benchmark and Data Track is to cater the reviewing and paper evaluation process towards the specific needs of papers that make useful contributions without proving a hypothesis or making a novel contribution. Specifically, this track seeks work in the following areas: 

Replication focuses on replicating a prior published approach in order to shed light on some important, possibly overlooked aspect. Replicating a result, or failing to, is a useful contribution to our collective knowledge, and good replication papers will challenge some previously accepted trusim, expose some limitation in the assumptions or confounds chosen, or confirm (or question) the internal validity of the results. For example: 
    • Jens Dittrich, Lukas Blunschi, and Marcos Antonio Vaz Salles. 2008. Dwarfs in the rearview mirror: how big are they really?. VLDB Endow. 1, 2 (August 2008). [PDF] 
Review Criteria: Is the replicated work significant, has it been done before, was there an important/relevant lesson to be learned from the replication, were hidden assumptions of the original experiment exposed? 

Benchmarks make available to the community a new class of resources, metrics or software that can be used to measure the performance of systems in some dimension. Any data and software should be made public through a reasonable access mechanism, to enable the community to use it. Ideally a benchmark paper will also provide some baseline of performance, or may further serve the community by surveying the performance of existing systems according to the benchmark. The key here would be that the systems evaluated are not being presented in the paper as the contribution. For example: 
    • Guo, Yuanbo, Heflin, Jeff and Pan, Zhengxiang . Benchmarking DAML+OIL Repositories. ISWC 2003. [PDF]
Review Criteria: Does the benchmark measure something significant (is it relevant and sufficiently general), are the proposed performance metrics sufficiently broad and relevant, is there already a similar benchmark (if yes, how does it differ), can others use the data and software, how can it advance the state of the art, if a survey was done, was the coverage of systems reasonable, or were any obvious choices missing? 

Data introduces an important data set to the community. This highly important task is often difficult to publish, as its main contribution lies in providing others the means for accomplishing their goals. Even though dbpedia and wordnet are some of the most valuable and widely used resources in our community, and have made an invaluable contribution to our science, they were very difficult to publish as papers. For example: 
    • S Auer, C Bizer, G Kobilarov, J Lehmann, R Cyganiak, Z Ives. Dbpedia: A nucleus for a web of open data. ISWC 2007. [PDF]
Review Criteria: Is there a similar data source? Is the source of interest to the semantic web community (and society in general)? Is the source semantic, linked, etc.? Does it use URIs. Is it available to the community? Was the data used for something scientific, practical, etc.? Is the data likely to be repurposed for other uses? 

Software Frameworks advance science by sharing with the community software that can easily be extended or adapted to support scientific study and experimentation. Jena, Sesame, hadoop, and many other software frameworks have clearly impacted our community but were, similarly, difficult to publish.  For example:

    • Brian McBride. Jena: A Semantic Web Toolkit. IEEE Internet Computing. November, 2002. [PDF]

Review Criteria: Does the framework solve a problem that is useful to the ISWC community?  Are there similar frameworks? How general is the framework in terms of applicability? What is the novelty of the abstractions chosen and are they useful? What can it not do and what is the rationale for the exclusion of that functionality? Does it use open standards, when applicable, or have good reason not to?  Is it open-source, freely available, extendable? How applicable is it to a wide variety of problems?  

We encourage the authors to carefully read the calls for the other tracks, the Research track, the  In Use track and the Industry track and consider submitting to the most 
appropriate track. Multiple submissions of the same paper to different tracks are not acceptable.
 

Topics of Interest

All topics addressed in any of the other ISWC tracks that present work without a clear hypothesis or novelty, but that present Replication, Benchmark, or Data studies are of interest to this track. 
 

Submission

Pre-submission of abstracts is a strict requirement. All papers and abstracts have to be submitted electronically via the EasyChair conference submission System at https://www.easychair.org/conferences/?conf=iswc2014rdb .
 
All research submissions must be in English, and no longer than 16 pages. Papers that exceed this limit will be rejected without review. Submissions must be in PDF formatted in the style of the Springer Publications format for Lecture Notes in Computer Science (LNCS). For details on the LNCS style, see Springer’s Author Instructions. ISWC-2014 submissions are not anonymous.

Authors of accepted papers will be required to provide semantic annotations for the abstract of their submission, which will be made available on the conference web site. Details will be provided at the time of acceptance. 

Accepted papers will be distributed to conference attendees and also published by Springer in the printed conference proceedings, as part of the Lecture Notes in Computer Science series. At least one author of each accepted paper must register for the conference and present the paper there.
 

Prior Publication And Multiple Submissions

ISWC 2014 will not accept research papers that, at the time of submission, are under review for or have already been published in or accepted for publication in a journal, another conference, or another ISWC track. The conference organizers may share information on submissions with other venues to ensure that this rule is not violated.
 

Submission of a Poster or Demo together with your Accepted Research Paper

Authors of accepted papers are invited to submit also a poster or a demo to the Posters and Demo track. The submission format is the same as for normal poster and demo submissions but the submission must cite the corresponding paper from the research track.
 

Important Dates

  • Abstracts: May 1, 2014
  • Full Paper Submission: May 9, 2014  May 12, 2014
  • Author Rebuttals: June 9-11, 2014 
  • Notifications: July 3, 2014  
  • Camera-Ready Versions: August 1, 2014
  • Conference: October 19-23, 2014
 
All deadlines are Hawaii time.
 

Program Chairs

Abraham Bernstein, University of Zurich, Switzerland
Chris Welty, IBM Research, USA
 

Program Committee

The program committee list will can be found here