NewsHuntBot is the news crawler of NewsHunt, a news search and reading service. It reads the latest articles from news websites so that NewsHunt can show the headline, a short summary and a link back to the publisher.
NewsHuntBot/0.1 (+https://newshunt.io/bot)Signature-Agent: "https://newshunt.io". Our public key is published at /.well-known/http-message-signatures-directory. Crawl-delay, that delay is used instead, however long it is.NewsHuntBot (case does not matter), only that group is used. Otherwise the * group is used.Allow and Disallow: the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.* matches any characters and $ marks the end of the URL.Crawl-delay (in seconds) is read from the group that applies, as described above.Retry-After time if it is longer.Retry-After (seconds or an HTTP date) for every request to that host. Without it we wait at least 60 seconds after a 429.Add this to your robots.txt:
User-agent: NewsHuntBot Disallow: /
Questions or problems: please contact us.
Newshunt utilise des cookies essentiels pour vous garder connecté et mémoriser votre langue et votre pays, afin que le site fonctionne comme prévu. Avec votre permission, nous aimerions également utiliser des cookies analytiques pour comprendre comment les utilisateurs utilisent Newshunt et l'améliorer au fil du temps.
Accepter n'affecte que les analyses. Pour en savoir plus, consultez notre Politique de confidentialité or Conditions générales.