referatele.com Review

 # Aggressive crawlers & scrapers
User-agent: FriendlyCrawler
User-agent: Orbbot
User-agent: PHPCrawl
User-agent: Nutch
User-agent: magpie-crawler
User-agent: Heritrix
User-agent: Indy Library
User-agent: Go-http-client
Disallow: /

# SEO & data mining bots
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: MJ12bot
User-agent: BLEXBot
User-agent: DotBot
User-agent: ZoominfoBot
User-agent: MaxPointCrawler
User-agent: DataForSeoBot
User-agent: seoscanners.net
Disallow: /

# AI / LLM crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Bytespider
User-agent: PetalBot
User-agent: Barkrowler
Disallow: /

# Regional / foreign search engines
User-agent: YandexBot
User-agent: SeznamBot
User-agent: Mail.Ru
User-agent: coccocbot-image
Disallow: /

# Social / marketing crawlers
User-agent: Amazonbot
User-agent: ADmantX
User-agent: Buzzbot
User-agent: trendictionbot
User-agent: rogerbot
User-agent: ZumBot
User-agent: AlphaBot
User-agent: AspiegelBot
Disallow: /

# Privacy search engines
User-agent: DuckDuckBot
Disallow: /

HTTP/1.1 302 Found
Date: Thu, 29 Jan 2026 07:19:16 GMT
Server: Apache
Location: https://www.referatele.com/
Content-Length: 251
Keep-Alive: timeout=2, max=350
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Date: Thu, 29 Jan 2026 07:19:18 GMT
Server: Apache
X-Powered-By: PHP/7.0.33
Upgrade: h2
Connection: Upgrade, Keep-Alive
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 18321
Keep-Alive: timeout=2, max=350
Content-Type: text/html; charset=iso-8859-1