http://www.labnol.org/forums/topic/too-many-crawler-errors-can-i-restrict-all-the-urls-in-robotstxt Help Center Topic: Too Many Crawler Errors - Can I restrict all the URL's In Robots.txt en-US Mon, 13 Feb 2012 04:21:35 +0000 http://bbpress.org/?v=1.0.1 q http://www.labnol.org/forums/search.php http://www.labnol.org/forums/topic/too-many-crawler-errors-can-i-restrict-all-the-urls-in-robotstxt#post-6583 Tue, 22 Sep 2009 06:43:05 +0000 Anonymous 6583@http://www.labnol.org/forums/ <p>To exclude all the robots from your server....edit your robots.txt file like this</p> <p>To exclude all robots from the server:<br /> User-agent: *<br /> Disallow: / </p> http://www.labnol.org/forums/topic/too-many-crawler-errors-can-i-restrict-all-the-urls-in-robotstxt#post-6576 Sat, 19 Sep 2009 17:11:44 +0000 Anonymous 6576@http://www.labnol.org/forums/ <p>I have too many errors in my blog say 85 and the blog is under penalty. I don know how to resolve this. I just want to restrict all the URL's using robots.txt . Will it solve the issue or make things worse. </p> <p>Somebody please let me know. </p>