Crawler Info

At Team NorthArchive we utilize automated web crawlers for regularly archiving webpages.

Our crawler can be identified by its User-Agent which contains a link to this page.

Fair Limits

All crawl-tasks are by default configured to respect fair limits. This includes limiting request count and bandwidth on a per-domain basis. These limits are adapted for the task at hand.

Software

Our crawler tasks are run on the Heritrix 3 engine developed by the Internet Archive.

Contact

The administrator for the crawler is Kevin Kandlbinder. He can be contacted via email at [email protected].