NewsHuntBot

NewsHuntBot is the news crawler of NewsHunt, a news search and reading service. It reads the latest articles from news websites so that NewsHunt can show the headline, a short summary and a link back to the publisher.

How to recognise it

  • User agent: NewsHuntBot/0.1 (+https://newshunt.io/bot)
  • Every request is signed with Web Bot Auth (HTTP Message Signatures) and sends Signature-Agent: "https://newshunt.io". Our public key is published at /.well-known/http-message-signatures-directory.
  • It reads public RSS and Atom feeds, sitemaps, section pages and article pages only. It does not log in and does not run scripts.

Request rate

  • At most one request is started every 2 seconds per host (30 requests a minute at most).
  • If robots.txt sets a longer Crawl-delay, that delay is used instead, however long it is.
  • Each site's feeds or sitemaps are checked every 2 to 15 minutes (busy sites more often), and then only the new articles listed there are fetched.

robots.txt

  • robots.txt is read before the first request to each origin (scheme, host and port), including feed hosts and redirect targets. Every redirect step is checked against the robots.txt of its own origin.
  • Groups: if a group names NewsHuntBot (case does not matter), only that group is used. Otherwise the * group is used.
  • Allow and Disallow: the longest matching rule wins. When an Allow and a Disallow rule are the same length, Allow wins.
  • Wildcards: * matches any characters and $ marks the end of the URL.
  • Crawl-delay (in seconds) is read from the group that applies, as described above.
  • robots.txt is cached for up to 24 hours per origin.
  • 404 or 410: we treat the site as having no rules.
  • 401 or 403, a bot check page, an HTML page instead of robots.txt, a 5xx error, a timeout or a network error: we do not crawl that origin at all. We ask for robots.txt again later: after 1 hour for 401, 403 or a bot check page, otherwise after 10 minutes or the Retry-After time if it is longer.

Errors and refusals

  • 401, 402, 403 or 451, or a bot check page: that page is not asked again, and the whole site is left alone for 6 hours.
  • 429 and 503: we honour Retry-After (seconds or an HTTP date) for every request to that host. Without it we wait at least 60 seconds after a 429.
  • Other 5xx errors and timeouts: at most one quick retry after 2 seconds, otherwise the page waits for a later turn.
  • Refused or challenged requests are never retried from another IP address, network, proxy, browser or user agent. We do not use proxies, rotate IP addresses or user agents, or solve CAPTCHAs.

How to block it

Add this to your robots.txt:

User-agent: NewsHuntBot
Disallow: /

Questions or problems: please contact us.

कुकीहरू बारे जानकारी

Newshunt ले तपाईंलाई साइन इन राख्न र तपाईंको भाषा र देश सम्झन आवश्यक कुकीहरू प्रयोग गर्दछ, ताकि साइट तपाईंले अपेक्षा गरेअनुसार काम गरोस्। तपाईंको अनुमतिमा, हामी Newshunt लाई मानिसहरूले कसरी प्रयोग गर्छन् भन्ने बुझ्न र समयसँगै सुधार्न एनालिटिक्स कुकीहरू पनि प्रयोग गर्न चाहन्छौं।

स्वीकार गर्नाले एनालिटिक्समा मात्र असर पार्छ। थप जान्नका लागि, हाम्रो हेर्नुहोस् गोपनीयता नीति or नियम र शर्तहरू.