<input type="hidden" id="_wpnonce" name="_wpnonce" value="554d87a88e" /><input type="hidden" name="_wp_http_referer" value="/blog/wp-json/wp/v2/posts/3282" /><input type="hidden" id="_wpnonce" name="_wpnonce" value="554d87a88e" /><input type="hidden" name="_wp_http_referer" value="/blog/wp-json/wp/v2/posts/3282" />{"id":3282,"date":"2015-03-26T10:37:42","date_gmt":"2015-03-25T23:37:42","guid":{"rendered":"https:\/\/www.businesslegions.com\/blog\/?p=3282"},"modified":"2015-03-26T10:37:42","modified_gmt":"2015-03-25T23:37:42","slug":"best-way-to-stop-crawler-bots","status":"publish","type":"post","link":"https:\/\/www.businesslegions.com\/blog\/2015\/03\/26\/best-way-to-stop-crawler-bots\/","title":{"rendered":"Best way to stop crawler bots"},"content":{"rendered":"<!--CusAds0--><p>I recently discovered one of my VPS servers was running constantly at 1-2 CPU load and received notifications from the VPS provider that it was\u00a0using more than my fair share of CPU resources and temporarily suspended the VPS to prevent it from impacting my neighbouring customers &#8211; which was fair enough.<\/p>\n<p>Upon investigation there were several reasons why the CPU load was high and one of them being that the sight was being crawled by different bots, Google, Bing, Yahoo, Ahref, Yandex, Twitter and the list goes on. So in order to reduce the load I decided to investigate what the best way was to prevent all the bots except for Google to crawl through my websites (please note that on the same VPS I&#8217;m hosting multiple websites).<\/p>\n<p>After some researching and testing, the best way I was able to stop all bots except for Google was to include the following in the robots.txt file:<\/p>\n<p>User-Agent: *<br \/>\nDisallow: \/<\/p>\n<p>User-Agent: Googlebot<br \/>\nAllow: \/<\/p>\n<p>User-Agent: Googlebot-Mobile<br \/>\nAllow: \/<\/p>\n<p>User-Agent: Googlebot-Image<br \/>\nAllow: \/<\/p>\n<p>User-Agent: Mediapartners-Google<br \/>\nAllow: \/<\/p>\n<p>User-Agent: Adsbot-Google<br \/>\nAllow: \/<\/p><!--CusAds0-->\n<p>I know some people may ask &#8220;Why only allow Google?&#8221; The answer to this question is because that&#8217;s the only search engine that I&#8217;ve noticed that has the highest referral visitors to my websites. I don&#8217;t see the point of having the other crawlers use up CPU and resources which could potentially slow down the website and I&#8217;d rather keep the website nice and clean with minimal user traffic. Also, I&#8217;ve noticed other crawlers are used for competitor analysis which I don&#8217;t really use.<\/p>\n<p><a href=\"https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file.png?strip=all&w=2560\"><img decoding=\"async\" class=\"alignnone size-full wp-image-3288\" src=\"https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file.png?strip=all\" alt=\"robots text file\" width=\"344\" height=\"306\" srcset=\"https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file.png?strip=all 344w, https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file-300x267.png?strip=all 300w, https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file.png?strip=all&amp;w=68 68w, https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file.png?strip=all&amp;w=137 137w, https:\/\/cdn.businesslegions.com\/blog\/wp-content\/uploads\/2015\/03\/robots-text-file.png?strip=all&amp;w=206 206w\" sizes=\"(max-width: 344px) 100vw, 344px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<div class=\"lizatom-infobox image: tip\"><span class=\"close-infobox\"><\/span><div class=\"text\"><p><strong>Do you have any other tips?<\/strong> Is this the best way to do this? If you have any other tips please let me know.<\/p><\/div><\/div>\n<div class=\"lt-box\" style=\"border:1px solid #1d6a9e\"><div class=\"lt-box-title\" style=\"background-color:#2485C6;border-top:1px solid #a7cee8;text-shadow:1px 1px 0 #0b283b\">DO YOU LIKE WHAT YOU'VE READ?<\/div><div class=\"lt-box-content\">Join our subscription list and receive our content right in your mailbox. If you like to receive some Great deals our Freebies then subscribe now!\r\n\r\n<p><div class=\"tnp tnp-subscription \">\n<form method=\"post\" action=\"https:\/\/www.businesslegions.com\/blog\/wp-admin\/admin-ajax.php?action=tnp&amp;na=s\">\n<input type=\"hidden\" name=\"nlang\" value=\"\">\n<div class=\"tnp-field tnp-field-firstname\"><label for=\"tnp-1\">Name<\/label>\n<input class=\"tnp-name\" type=\"text\" name=\"nn\" id=\"tnp-1\" value=\"\" placeholder=\"\"><\/div>\n<div class=\"tnp-field tnp-field-email\"><label for=\"tnp-2\">Email<\/label>\n<input class=\"tnp-email\" type=\"email\" name=\"ne\" id=\"tnp-2\" value=\"\" placeholder=\"\" required><\/div>\n<div class=\"tnp-field tnp-field-button\" style=\"text-align: left\"><input class=\"tnp-submit\" type=\"submit\" value=\"Subscribe\" style=\"\">\n<\/div>\n<\/form>\n<\/div>\n<\/p>\r\n\r\n<\/div><\/div><!--CusAds0-->\n<div style=\"font-size: 0px; height: 0px; line-height: 0px; margin: 0; padding: 0; clear: both;\"><\/div>","protected":false},"excerpt":{"rendered":"<p>I recently discovered one of my VPS servers was running constantly at 1-2 CPU load and received notifications from the VPS provider that it was\u00a0using more than my fair share of CPU resources and temporarily suspended the VPS to prevent it from impacting my neighbouring customers &#8211; which was fair enough. Upon investigation there were [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":3288,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[1805,1804,97,1156,93],"class_list":["post-3282","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-bots","tag-crawlers","tag-google","tag-robots","tag-seo"],"_links":{"self":[{"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/posts\/3282","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/comments?post=3282"}],"version-history":[{"count":4,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/posts\/3282\/revisions"}],"predecessor-version":[{"id":3290,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/posts\/3282\/revisions\/3290"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/media\/3288"}],"wp:attachment":[{"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/media?parent=3282"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/categories?post=3282"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.businesslegions.com\/blog\/wp-json\/wp\/v2\/tags?post=3282"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}