About the journal

• Home page
• Contact information
• Aims and scope
• Indexing information
• Editorial policies
• ComSIS consortium
• Journal boards
• Managing board

For authors

• Information for contributors
• Paper submission
• Article submission through OJS
• Copyright transfer form
• Download section

For readers

• Forthcoming articles
• Current issue
• Archive

For reviewers

• View and review submissions

News

• Journal's Facebook page
• Calls for special issues
• New issue notification

A Scale for Crawler Effectiveness on the Client-Side Hidden Web

Vıctor M. Prieto¹, Manuel Alvarez¹, Rafael Lopez-Garcıa¹ and Fidel Cacheda¹

1. Comunication and Information Technologies Department, University of A Coruna
   Campus de Elvina s/n - 15071, A Coruna, Spain
   {victor.prieto, manuel.alvarez, rafael.lopez, fidel.cacheda}@udc.es

Abstract

The main goal of this study is to present a scale that classifies crawling systems according to their effectiveness in traversing the “clientside” Hidden Web. First, we perform a thorough analysis of the different client-side technologies and the main features of the web pages in order to determine the basic steps of the aforementioned scale. Then, we define the scale by grouping basic scenarios in terms of several common features, and we propose some methods to evaluate the effectiveness of the crawlers according to the levels of the scale. Finally, we present a testing web site and we show the results of applying the aforementioned methods to the results obtained by some open-source and commercial crawlers that tried to traverse the pages. Only a few crawlers achieve good results in treating client-side technologies. Regarding standalone crawlers, we highlight the open-source crawlers Heritrix and Nutch and the commercial crawler WebCopierPro, which is able to process very complex scenarios. With regard to the crawlers of the main search engines, only Google processes most of the scenarios we have proposed, while Yahoo! and Bing just deal with the basic ones. There are not many studies that assess the capacity of the crawlers to deal with client-side technologies. Also, these studies consider fewer technologies, fewer crawlers and fewer combinations. Furthermore, to the best of our knowledge, our article provides the first scale for classifying crawlers from the point of view of the most important client-side technologies.

Key words

Web Search, Crawlers, Hidden Web, Web Spam, JavaScript

Digital Object Identifier (DOI)

https://doi.org/10.2298/CSIS111215015P

Publication information

Volume 9, Issue 2 (June 2012)
Year of Publication: 2012
ISSN: 2406-1018 (Online)
Publisher: ComSIS Consortium

Full text

Available in PDF
Portable Document Format

How to cite

Prieto, V. M., Alvarez, M., Lopez-Garcıa, R., Cacheda, F.: A Scale for Crawler Effectiveness on the Client-Side Hidden Web. Computer Science and Information Systems, Vol. 9, No. 2, 561-583. (2012), https://doi.org/10.2298/CSIS111215015P