NewsHuntBot is the news crawler of NewsHunt, a news search and reading service. It reads the latest articles from news websites so that NewsHunt can show the headline, a short summary and a link back to the publisher.
NewsHuntBot/0.1 (+https://newshunt.io/bot)Signature-Agent: "https://newshunt.io". Our public key is published at /.well-known/http-message-signatures-directory. Crawl-delay, that delay is used instead, however long it is.NewsHuntBot (case does not matter), only that group is used. Otherwise the * group is used.Allow and Disallow: the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.* matches any characters and $ marks the end of the URL.Crawl-delay (in seconds) is read from the group that applies, as described above.Retry-After time if it is longer.Retry-After (seconds or an HTTP date) for every request to that host. Without it we wait at least 60 seconds after a 429.Add this to your robots.txt:
User-agent: NewsHuntBot Disallow: /
Questions or problems: please contact us.
يستخدم نيوزهنت ملفات تعريف ارتباط ضرورية لإبقائك مسجلاً للدخول ولتذكر لغتك ودولتك، حتى يعمل الموقع كما تتوقع. بإذنك، نود أيضاً استخدام ملفات تعريف ارتباط تحليلية لفهم كيفية استخدام الأشخاص لنيوزهنت وتحسينه بمرور الوقت.
الموافقة تؤثر فقط على التحليلات. لمعرفة المزيد، اطّلع على سياسة الخصوصية or الشروط والأحكام.