Hi Guys,
Just wondering if you have any advice on what folders etc to add to a robots.txt file for a Grav install…or do you have an example of one that we could use as a standard as part of the install?
I don’t think a robots.txt file will be very useful for Grav. Our .htaccess file is already blocking all undesired requests, and everything else is routed through index.php.
A robots.txt file would just allow everything which is the same as having no file at all.
Ok…that makes sense. The only reason I mentioned it is because when I listed the website chefsknivesonline.com.au with Google Webmaster Tools…they sent this email 2 days later
“Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn’t crawl any pages listed in that file, we postponed our crawl. Your site’s overall robots.txt error rate is 100.0%.”
This might be an issue moving forward for people wanting to use the Grav system for landing pages, blogs or stores that are designed to get traction with organic SEO.
The site has been indexed by Google so that’s no issue…it’s just that crawl rate for updates etc might be affected with no robots.txt.
Hmm… That’s strange. You could try something like this (assuming you have the sitemap plugin installed):
User-Agent: *
Disallow:
Allow: /
Sitemap: http://yoursite.com/sitemap.xml
FYI, getgrav.org has no robots.txt and it is indexed just fine in google:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=modern+flat+cms
Cool…will try that. Should hopefully make Google happy.
Hasn’t stopped the site being indexed tho 
I will look into this more and document the sitemap stuff better in the coming weeks!