NewsHuntBot is the news crawler of NewsHunt, a news search and reading service. It reads the latest articles from news websites so that NewsHunt can show the headline, a short summary and a link back to the publisher.
NewsHuntBot/0.1 (+https://newshunt.io/bot)Signature-Agent: "https://newshunt.io". Our public key is published at /.well-known/http-message-signatures-directory. Crawl-delay, that delay is used instead, however long it is.NewsHuntBot (case does not matter), only that group is used. Otherwise the * group is used.Allow and Disallow: the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.* matches any characters and $ marks the end of the URL.Crawl-delay (in seconds) is read from the group that applies, as described above.Retry-After time if it is longer.Retry-After (seconds or an HTTP date) for every request to that host. Without it we wait at least 60 seconds after a 429.Add this to your robots.txt:
User-agent: NewsHuntBot Disallow: /
Questions or problems: please contact us.
نیوشانت از کوکیهای ضروری برای حفظ ورود شما و به خاطر سپردن زبان و کشور انتخابیتان استفاده میکند تا سایت مطابق انتظار شما کار کند. با اجازه شما، همچنین میخواهیم از کوکیهای تحلیلی برای درک نحوه استفاده افراد از نیوشانت و بهبود آن استفاده کنیم.
پذیرش فقط بر کوکیهای تحلیلی تأثیر میگذارد. برای اطلاعات بیشتر، این مورد را ببینید: سیاست حفظ حریم خصوصی or شرایط و ضوابط.