Blog, Let’s understand how to use User-agent properly and resolve indexing errors.

It is important to accurately identify the types of robots when writing a robots.txt file.

 

 

Since Googlebot crawls using smartphones, redirection errors may occur. 

Therefore, let’s properly understand how to use User-agent.

 

 

Googlebot   (Desktop only)

Googlebot is Google’s desktop web crawler, which navigates and indexes web pages to reflect them in search engine results. Googlebot plays a crucial role in analyzing website content to provide users with the most relevant information.

Googlebot-Mobile   (Mobile only)

Googlebot-Mobile is Google’s mobile web crawler that crawls mobile-optimized web pages and reflects them in search engine results. This crawler is specifically designed to optimize the mobile user experience.

AdsBot-Google

AdsBot-Google is Google’s advertising crawler, used to assess whether advertisers can utilize their websites in Google Ads. 

This crawler plays a crucial role in determining the quality and suitability of web pages.

 

 

 

AdsBot-Google-Mobile

AdsBot-Google-Mobile is a specialized crawler used by Google to evaluate mobile web pages for advertising purposes.

 

 

Usage example

Allow all robots to access all documents.

User-agent: *

Allow: / 

 

User-agent: Googlebot

User-agent: Googlebot-Mobile

User-agent: AdsBot-Google

User-agent: AdsBot-Google-Mobile

 

 

Googlebot: Google's web crawler (desktop version)

Googlebot-Mobile: Google's mobile crawler

Bingbot: Bing search engine's crawler

Slurp: Yahoo's web crawler

Baidu: Baidu search engine's crawler (China)

Yandex: Yandex's web crawler (Russia)

DuckDuckBot: DuckDuckGo's web crawler

Exabot: Exabot's web crawler

Facebot: Facebook's crawler

Twitterbot: Twitter's crawler

 

 

 

Copyright2021olympithecus