SciPost Code Repository

Skip to content
Snippets Groups Projects
Normal view History Permalink
robots.txt 104 B
Newer Older
Jean-Sébastien Caux's avatar
Add crawl-delay to ahrefs bot  
Jean-Sébastien Caux committed
1 2 3 
User-agent: AhrefsBot
Crawl-Delay: 10
Jean-Sébastien Caux's avatar
Add robots.txt
Jean-Sébastien Caux committed
4 5 
User-agent: BLEXBot
Disallow: /
Jean-Sébastien Caux's avatar
Tweak robots.txt  
Jean-Sébastien Caux committed
6 7 8 

User-Agent: *
Disallow: /search

SciPost Code Repository