Post by matthew on Jun 24, 2011 6:29:18 GMT 1
I think many of you have heard about how the Google bot (or any other spider bot) can delete your whole site if you are using a poorly designed CMS. It will follow all of the delete links. Well there is something new to worry about now . . .
The bot ia_archiver can follow javascript links! I purposefully designed my CMS to use javascript link so that it couldn't be spidered, but this bot will follow onclick="location.href='url'". ugh.
Thankfully I protect all of my admin sections with .htaccess, but not sites I am currently developing. I don't even know how it found the admin section in the first place as there are no external links. An entry in the robots.txt file could also be helpful, however this could make the path to your admin section publicly available to anyone who looks at the robots.txt, which mayor may not be a bad thing, depending on how you protect it. I also usually only delete items from a form submission so this only affects a small portion of the CMS.
=================
Web Design Glasgow|Graphic Design Glasgow
The bot ia_archiver can follow javascript links! I purposefully designed my CMS to use javascript link so that it couldn't be spidered, but this bot will follow onclick="location.href='url'". ugh.
Thankfully I protect all of my admin sections with .htaccess, but not sites I am currently developing. I don't even know how it found the admin section in the first place as there are no external links. An entry in the robots.txt file could also be helpful, however this could make the path to your admin section publicly available to anyone who looks at the robots.txt, which mayor may not be a bad thing, depending on how you protect it. I also usually only delete items from a form submission so this only affects a small portion of the CMS.
=================
Web Design Glasgow|Graphic Design Glasgow