robots.txt

Moderators: Moderator, Global Moderator

Post Reply
Tami
Administrator
Administrator
Posts: 10892
Joined: Sun Apr 25, 2004 1:05 pm

robots.txt

Post by Tami »

A suggestion from balamm to slow down the index bot parties

[quote]I had that problem with MSN for a while but they suggested adding "crawl delay=600" to the robots.txt to slow down the spidering a bit and take a load off the server.

Seems to have worked.[/quote]
Last edited by Tami on Fri Jan 28, 2005 7:24 am, edited 1 time in total.
Image

[color=\"#41211C\"]It takes years to build up trust and only seconds to destroy it

[/color]
Top
Post Reply

Return to “Uber Coffee Room”