NewsHuntBot is the news crawler of NewsHunt, a news search and reading service. It reads the latest articles from news websites so that NewsHunt can show the headline, a short summary and a link back to the publisher.
NewsHuntBot/0.1 (+https://newshunt.io/bot)Signature-Agent: "https://newshunt.io". Our public key is published at /.well-known/http-message-signatures-directory. Crawl-delay, that delay is used instead, however long it is.NewsHuntBot (case does not matter), only that group is used. Otherwise the * group is used.Allow and Disallow: the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.* matches any characters and $ marks the end of the URL.Crawl-delay (in seconds) is read from the group that applies, as described above.Retry-After time if it is longer.Retry-After (seconds or an HTTP date) for every request to that host. Without it we wait at least 60 seconds after a 429.Add this to your robots.txt:
User-agent: NewsHuntBot Disallow: /
Questions or problems: please contact us.
Newshunt utilizza cookie essenziali per mantenerti connesso e ricordare la tua lingua e il tuo paese, così il sito funziona come previsto. Con il tuo consenso, vorremmo inoltre utilizzare cookie analitici per capire come viene utilizzato Newshunt e migliorarlo nel tempo.
Il consenso riguarda solo i cookie analitici. Per saperne di più, consulta la nostra Informativa sulla privacy or Termini e condizioni.