SciPost Code Repository

Skip to content
Snippets Groups Projects
Normal view History Permalink
robots.txt 202 B
Newer Older
Jean-Sébastien Caux's avatar
Add crawl-delay to ahrefs bot  
Jean-Sébastien Caux committed
1 2 3 
User-agent: AhrefsBot
Crawl-Delay: 10
Jean-Sébastien Caux's avatar
Add robots.txt
Jean-Sébastien Caux committed
4 5 
User-agent: BLEXBot
Disallow: /
Jean-Sébastien Caux's avatar
Tweak robots.txt  
Jean-Sébastien Caux committed
6
George Katsikas's avatar
chore: 🔧 block Apple's LLM training bot  
George Katsikas committed
7 8 9 10 
# Block Apple from training LLM models on our content
User-agent: Applebot-Extended 
Disallow: /
Jean-Sébastien Caux's avatar
Tweak robots.txt  
Jean-Sébastien Caux committed
11 12 
User-Agent: *
Disallow: /search

SciPost Code Repository