Dr.Rob wrote:What ? There is a robot spider crawling around inside my computer! aaaaaaaaaaaaaaaaaaaaaaaaaaa! Darth your kidding right?
no i'm not kidding and they aren't inside your computer, they only go towards web servers with registered domain names really and they obey the commands that web servers give them as well.
so if you don't want a section of your site to be crawled and indexed, then you can tell the bots to stay out of the folder you don't want indexed.
This is done with the bot.txt file in the root of any webserver.
Your home systems aren't indexed because they are not registered domains.
Coconuts. Bananas. Mangos. Rice. Beans. Water. It's good.