正如所有国内SEO所知,百度蜘蛛名称叫作BaiduSpider,而谷歌蜘蛛名称叫作GoogleBot。但是不限于此,谷歌不同产品的抓取UA(User-Agent)是不一样的。可以理解为每个产品都是独立项目在运行。当然,它们的名称也可能是一样,比如谷歌PC版蜘蛛和移动端蜘蛛的名称都是Googlebot。以下为谷歌所有产品抓取工具的蜘蛛名称(谷歌官方称之为:用户代理令牌或产品令牌)与User-Agent(完整的用户代理字符串)。
抓取工具 (产品)
用户代理令牌
(产品令牌)(蜘蛛名称)
完整的用户代理字符串 (User-Agent)
APIs-Google APIs-Google APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) AdSense Mediapartners-Google Mediapartners-Google AdsBot 移动网络 Android(检查 Android 网页广告质量) AdsBot-Google-Mobile Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) AdsBot 移动网络(检查 iPhone 网页广告质量) AdsBot-Google-Mobile Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) AdsBot(检查桌面设备网页广告质量) AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html) Googlebot Images Googlebot-Image Googlebot
Googlebot-Image/1.0 Googlebot News Googlebot-News Googlebot
Googlebot-News Googlebot Video Googlebot-Video Googlebot
Googlebot-Video/1.0 Googlebot(桌面版) Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
或(很少使用):
Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot(智能手机版) Googlebot Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mobile AdSense Mediapartners-Google (各类移动设备)(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) 移动应用 Android(检查 Android 应用页面广告质量。遵守 AdsBot-Google 漫游器规则。) AdsBot-Google-Mobile-Apps AdsBot-Google-Mobile-Apps Feedfetcher FeedFetcher-Google 不遵循 robots.txt 规则
FeedFetcher-Google; (+http://www.google.com/feedfetcher.html) Google Read Aloud Google-Read-Aloud 不遵循 robots.txt 规则
现用代理: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)
曾用代理(已弃用):
google-speakr