http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers Help Center Topic: Blogger Robots.txt Blocks Search Engine Crawlers en-US Mon, 13 Feb 2012 02:02:07 +0000 http://bbpress.org/?v=1.0.1 q http://www.labnol.org/forums/search.php http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-275 Fri, 10 Aug 2007 20:23:19 +0000 Anurag 275@http://www.labnol.org/forums/ <p>Abhi - your robots.txt is absolutely fine and you are not marked as spam by Google.</p> <p>User-agent: *<br /> Disallow: /search<br /> Sitemap: blogspot.com/feeds/posts/..</p> <p>Can you check your Adsense reports and see if there are any error messages. </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-274 Fri, 10 Aug 2007 18:46:38 +0000 Anonymous 274@http://www.labnol.org/forums/ <p>hi all, </p> <p>i am expecting a similar problem. in my case, although my blog is not marked as spam, but its having problem with adsense. this seems to be beacuse of the configuration of blogger.com's robots.txt , .. but then adsense is working fine on many of the blogs on blogspot.com . Is anyone else also facing similar issues ??</p> <p>thanks,<br /> abhi </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-233 Fri, 03 Aug 2007 07:01:26 +0000 Anonymous 233@http://www.labnol.org/forums/ <p>It has been rectified as I got a mail from Blogger team as under:-</p> <p>Hello,</p> <p>Your blog has been reviewed, verified, and cleared for regular use so that<br /> it will no longer appear as potential spam. If you sign out of Blogger and<br /> sign back in again, you should be able to post as normal. Thanks for your<br /> patience, and we apologize for any inconvenience this has caused.</p> <p>Sincerely,<br /> The Blogger Team </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-231 Thu, 02 Aug 2007 11:34:48 +0000 Anurag 231@http://www.labnol.org/forums/ <p>So I think the guess was correct. YOur blog is flagged as spam by the Blogger staff. </p> <p>Once that ban is lifted, the robots.txt will auto-correct. </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-229 Thu, 02 Aug 2007 01:09:04 +0000 Anonymous 229@http://www.labnol.org/forums/ <p>when I tried to edit a previous post, it showed a warning message:-<br /> <a href="http://rupeshmandal.googlepages.com/robots-spam.JPG" rel="nofollow">http://rupeshmandal.googlepages.com/robots-spam.JPG</a></p> <p>I have requested to unblock it. and now have nothing to do except wait and watch keeping fingers crossed. </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-227 Wed, 01 Aug 2007 22:22:33 +0000 Anurag 227@http://www.labnol.org/forums/ <p>When you submit a ticket to Blogger support, you will immediately get an auto-reply.</p> <p>YOU HAVE TO CONFIRM AND REPLY TO THAT MAIL FOR BLOGGER STAFF TO TAKE ACTION.</p> <p>If you have not received an acknowledgement mail, I suggest that you submit the request again.</p> <p>I know how you may be feeling at the moment but don't worry too much as this should resolve in a day or two. Such things happen. </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-226 Wed, 01 Aug 2007 19:27:17 +0000 Anonymous 226@http://www.labnol.org/forums/ <p>one more thing, I have tried your suggestion i.e. set "Add your Blog to our listings?" to No, save settings, then again set to Yes and click save settings; but in vain.</p> <p>changed the blog's template; switched back to classic template and then to new templates again, but no use.</p> <p>My blog is not listed in search engines anymore. You can check "recent came from" in blog's statistics. there is no entry from Google search.</p> <p>even, incoming traffic has decreased and adsense revenue has also decreased.</p> <p>I am lost, totally lost.<br /> This has happened with me; which can happen with anyone else too. </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-225 Wed, 01 Aug 2007 19:14:24 +0000 Anonymous 225@http://www.labnol.org/forums/ <p>hi,</p> <p>I have already reported to Blogger help and official google group.</p> <p><a href="http://groups.google.com/group/blogger-help-troubleshoot/browse_thread/thread/2ec6aca46f2158f/c05f75132815939d?lnk=gst&#38;q=fundubytes&#38;rnum=2#c05f75132815939d" rel="nofollow">http://groups.google.com/group/blogger-help-troubleshoot/browse_thread/thread/2ec6aca46f2158f/c05f75132815939d?lnk=gst&#38;q=fundubytes&#38;rnum=2#c05f75132815939d</a></p> <p><a href="http://www.blogger.com/problem.g" rel="nofollow">http://www.blogger.com/problem.g</a></p> <p>But, have not got any reply yet. Even I have not got any acknowledgement mail containing ticket no for the problem reported to Blogger team.</p> <p>How can I know whether my blog has been flagged/spammed or not? Navbar is hidden but can be accessed via <a href="http://www2.blogger.com/navbar.g?blogID=2423522895927610496" rel="nofollow">http://www2.blogger.com/navbar.g?blogID=2423522895927610496</a></p> <p>But, the question why my blog will be flagged or spammed? There is no objectionable content.</p> <p>I am currently out of Bangalore, at my native place at Jharkhand. Due to electricity and dial-up problems I cannot do much now. I do not have patience and time to shift to any other address or wordpress. So, it seems its end of blogging by me.</p> <p>If you can help me anyway, please help me to save my blog.</p> <p>thanks and regards, </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-224 Wed, 01 Aug 2007 15:15:43 +0000 Anurag 224@http://www.labnol.org/forums/ <p>Another question - was your blog anytime flagged by the blogger team for some reason ? </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-222 Wed, 01 Aug 2007 13:55:22 +0000 Anurag 222@http://www.labnol.org/forums/ <p>If you have set the "Add Your Blog to Our Listings" as Yes, this surely looks like a serious issue or a bug in blogger.</p> <p>Since your robots.text disallows the root, Google and other search engines will soon kick out your site out of their index.</p> <p>User-agent: *<br /> Disallow: /search<br /> Disallow: /</p> <p>Try this - set that to No, save settings, then again set to Yes and click save settings.</p> <p>If that doesn't help, try these URLs:</p> <p>help.blogger.com/?page=help - The secret page to contact blogger when you have a problem with your blogspot blog. People are not using this much but believe me, the blogger team at Google is very responsive and do listen to your complaints.</p> <p>google.com/group/blogger-help - This google group is monitored by employees of google working on Blogger. This is one place you should visit very frequently to get an update on new blogger template ideas, suggestions, issues, etc. </p> http://www.labnol.org/forums/topic/robotstxt-disallowing-web-crawlers#post-221 Wed, 01 Aug 2007 08:17:43 +0000 Anonymous 221@http://www.labnol.org/forums/ <p>hi,</p> <p>check out fundubytes.blogspot.com/robots.txt which is blocking web crawlers to crawl even though "Add your Blog to our listings?"</p> <p>under basic settings is set to yes and blog template contains &lt;META<br /> content='index,follow' name='robots'/&gt;</p> <p>even, Sitemap: fundubytes.blogspot.com/feeds/posts/default?orderby=updated<br /> is missing in robots.txt.</p> <p>Google webmaster tools is showing errors. Take a look at the screenshots:-<br /> <a href="http://rupeshmandal.googlepages.com/robots1.JPG" rel="nofollow">http://rupeshmandal.googlepages.com/robots1.JPG</a><br /> <a href="http://rupeshmandal.googlepages.com/robots2.JPG" rel="nofollow">http://rupeshmandal.googlepages.com/robots2.JPG</a><br /> <a href="http://rupeshmandal.googlepages.com/robots3.JPG" rel="nofollow">http://rupeshmandal.googlepages.com/robots3.JPG</a><br /> <a href="http://rupeshmandal.googlepages.com/robots4.JPG" rel="nofollow">http://rupeshmandal.googlepages.com/robots4.JPG</a></p> <p>Blog's pagerank has also dropped.</p> <p>I have reported this issue to Blogger team but have not got any reply.</p> <p>Please help me. </p>