SciPost Code Repository

Skip to content
Snippets Groups Projects
Commit db4a48bb authored by Jean-Sébastien Caux's avatar Jean-Sébastien Caux
Browse files

Add crawl-delay to ahrefs bot

parent 2e12f8be
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 3 additions and 0 deletions
robots.txt
+ 3
− 0
View file @ db4a48bb
User-agent: AhrefsBot
Crawl-Delay: 10
User-agent: BLEXBot User-agent: BLEXBot
Disallow: / Disallow: /
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

SciPost Code Repository