Title: Intelligent focused crawler: Learning which links to crawl
Abstract:A web crawler is defined as an automated program that methodically scans through Internet pages and downloads any page that can be reached via links. With the exponential growth of the Web, fetching i...A web crawler is defined as an automated program that methodically scans through Internet pages and downloads any page that can be reached via links. With the exponential growth of the Web, fetching information about a special-topic is gaining importance. A focused crawler is a web crawler that attempts to download only web pages that are relevant to a predefined topic or set of topics. In order to determine a web page is about a particular topic, focused crawlers use classification techniques. In this study we focus on the classification of links instead of downloaded web pages to determine relevancy. We combine a Naïve Bayes classifier for classification of URLs with a simple URL scoring optimization to improve the system performance. Our results demonstrate that proposed approach performs better.Read More
Publication Year: 2011
Publication Date: 2011-06-01
Language: en
Type: article
Indexed In: ['crossref']
Access and Citation
Cited By Count: 23
AI Researcher Chatbot
Get quick answers to your questions about the article from our AI researcher chatbot