What is a crawler?

Crawlers have a wide variety of uses on the internet. They automatically search through documents online. Website operators are mainly familiar with web crawlers from search engines such as Google or Bing; however, crawlers can also be used for malicious purposes and do harm to companies.

At one look

01. A definition of what a crawler is
02. How does a crawler work?
03. What types of crawlers are there?
04. Crawler vs scraper: a comparison
05. How are crawlers blocked and managed?
06. What hazards do spam crawlers pose?
07. Crawlers: What you need to know

A definition of what a crawler is

Crawlers are computer programs that are programmed to search the internet. Typically, developers program a crawler so that it repeats the same actions over and over again. This is how search is automated, which is why “robots” is also another name for crawlers. “Spider” is also another name because they crawl across the World Wide Web.

Google and other search engines use crawlers to index websites. For a site to appear in Google’s search results requires the Google crawler to first visit the site and list it.

How does a crawler work?

A crawler works through several pre-defined steps one after the other. That’s why it is vital to define these steps before the crawl. Typically, a crawler visits the different URLs of a website one by one and then saves the results in an index. How this index looks depends on the specific algorithm, for example, the Google algorithm specified the order in which results appear for a specific search query.

What types of crawlers are there?

Developers use crawlers in a variety of ways:

Search engines

Particularly widespread and well known is the use of crawlers by search engines like Google or Bing. The work of these search engines is based on the use of web crawlers. This is because they are the ones who prepare search results for users by creating an index.

Focused crawler

“Focused crawlers” are the topic-related counterpart to the universal search machine. They limit themselves to specific areas of the internet, for example, sites with a specific topic or current reports/news and create a detailed index of this.

Web analysis

Webmasters also use crawlers to analyze websites relating to data such as site visits or links. Most employ special web analysis tools.

Price comparison

The prices for many products, such as flights or electronics products, vary depending on vendor. Price comparison websites use crawlers to provide their users an overview of current prices.

Crawler vs scraper: a comparison

At first glance, a scraper operates similarly to a crawler. They both collect data from other websites for reuse. However, cybercriminals often use scrapers for malicious purposes and scrape the entire content of a site that is visible to the user. While crawlers primarily collect and organize the metadata of a URL, scrapers typically copy the entire content of other websites to then make it accessible via a different URL.

How are crawlers blocked and managed?

Under certain circumstances, it can make sense to block crawlers in general or block specific crawlers on your website. Using the robots.txt file, webmasters can block specific crawlers. This is a good idea if, for example, the website would otherwise be negatively impacted from crawling activity.

However, website operators cannot use the robots.txt file to completely prevent the indexing of a URL in search engines. If you want to prevent search engines from indexing a specific URL, such as SEA landing pages exclusively optimized for advertising, then the noindex meta tag is the right choice.

What hazards do spam crawlers pose?

Webmasters closely monitor the traffic on their websites. However, crawlers pose a problem here because they skew the numbers. Since a large number of crawlers is active on the internet, in many cases crawlers are responsible for a considerable share of spam-based traffic. Crawler referrer spam, in particular, poses a risk factor here because this type of crawler ignores the robots.txt file and accesses the website directly.

Crawlers: What you need to know

As a website operator, you always have to keep an eye on crawler activities on your site. Along with valuable crawlers such as search engine web crawlers, there are other types of crawlers that negatively impact website performance. Using a professional bot management system, you can control the activities of crawlers so that website performance is ensured, especially during peak times such as shopping events.

To Myra Application Security

Want to learn more about our solutions, use cases and best practices for attack defense? In our download area you will find product sheets, fact sheets, white papers and case studies.

Go to download section

Responsible Disclosure Legal notice Privacy policy Privacy Settings Terms and Conditions