Thanks for the replys. I've opted for the first version at the moment. Off course no one knows how exactly crawlers out there are implemented or if they obey robots.txt at all. Atleast i can serve a valid robots.txt now. cheers Ren?